* fix: Add JSON repair for truncated LLM responses across all providers and Gemini thinking budget support
LengthFinishReasonError from OpenAI-compatible providers (custom, openai, groq) was crashing the deriver
with 14k+ occurrences in production. The vLLM path already had repair logic but it was gated on
provider=="vllm", unreachable when routing through litellm as a custom provider.
- Extract shared _repair_response_model_json() helper for all providers
- Catch LengthFinishReasonError in OpenAI/custom parse() path and repair truncated JSON
- Add repair fallback to Anthropic and Gemini response_model paths
- Add repair fallback to Groq response_model path
- Pass thinking_budget_tokens to Gemini 2.5 models via thinking_config
- Add 14 tests covering repair paths for all providers and Gemini thinking budget
Fixes HONCHO-YC
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: live llm integration tests
* feat: Consistent Model Config Protocol
* fix: migrate the remaining app callers off the legacy llm_settings path
* fix: Docs and regression tests
* fix: refactor llm runtime path to model-config-only API
* fix: refactor config to nested model-config source of truth
* fix: refactor llm streaming and tool dispatch through backends
* fix: cut over llm config to nested model_config only
* fix: collapse vllm and custom into openai_compatible transport
* feat: refactor llm config to explicit transports and bare model ids
* feat: (embed) Add configurability for embedding model
* fix: tests for embedding provider
* fix: Address Review Comments
* fix: (llm) remove Groq backend and per-vendor base URLs
* chore: move llm tests
* fix: (llm) address review findings — config regressions, backend bugs, dead code
* fix: address backend end silly errors
* chore: (docs) update configuration and self-hosting guides
* chore: fix tests
* fix: address code rabbit comments
* fix: add validation to the dream settings
* fix: further address code rabbit comments
* fix: Address Code Rabbit Comments
* fix: Another round of code rabbit
* fix: Address Code Rabbit Nits
* fix: tests
* refactor: rename thinking validator to reflect transport scope
_validate_anthropic_thinking_minimum only enforces the >=1024 rule for
Anthropic and no-ops for other transports, so the name was misleading
now that it's shared across ConfiguredModelSettings, FallbackModelSettings,
and ModelConfig. Renamed to _validate_thinking_constraints with a docstring
clarifying per-transport behavior. No logic change.
* fix(config): drop transport-specific thinking params when env override changes transport
_fill_defaults_for_nested_field previously preserved the default MODEL_CONFIG's
thinking_budget_tokens/thinking_effort across a transport override. This leaked
Gemini-family defaults (e.g. thinking_budget_tokens=1024) into OpenAI-transport
overrides, and the OpenAI backend then correctly rejected the unsupported param
at call time (OpenAI uses reasoning.effort, not a token budget).
The helper now strips thinking_budget_tokens and thinking_effort from the
default dict when the env override supplies a transport different from the
default's. Explicit thinking params in the override are preserved.
* fix(config): apply thinking-param strip to dialectic level merge too
DialecticSettings._merge_level_defaults does its own inline MODEL_CONFIG
merge (parallel to _fill_defaults_for_nested_field), so the previous fix
missed dialectic-level overrides. E.g. flipping
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__TRANSPORT from gemini (default)
to openai still leaked the default thinking_budget_tokens=0 into the
openai config, which the OpenAI backend then rejected at call time.
The level-merge path now applies the same 'strip transport-specific
thinking params when transport changes' rule as the generic helper.
Added a regression test exercising the merge validator directly.
* refactor(llm): wire ModelConfig knobs through, prune clients.py migration leftovers
Three connected fixes to finish carving the LLM stack out of src/utils/clients.py
and into src/llm/:
1. Propagate ModelConfig tuning knobs into backend calls.
honcho_llm_call_inner built extra_params from only {json_mode, verbosity},
silently dropping top_p, top_k, frequency_penalty, presence_penalty, seed,
and operator-supplied provider_params from any ModelConfig. Thread the
selected config through ProviderSelection and merge
build_config_extra_params(selected_config) into extra_params; per-call
kwargs still win over provider_params defaults. Makes
_build_config_extra_params public as build_config_extra_params so
clients.py and request_builder.py share one translation. Adds
TestModelConfigExtraParamsPropagation covering OpenAI/Anthropic knob
propagation, provider_params passthrough, and per-call override
precedence.
2. Drop dead extract_openai_* duplicates in clients.py.
extract_openai_reasoning_content, extract_openai_reasoning_details, and
extract_openai_cache_tokens had no callers outside their own definitions
— the live implementations live in src/llm/backends/openai.py. -103
lines from clients.py.
3. Unify on ModelTransport, delete SupportedProviders.
The "google" vs "gemini" split forced a _provider_for_model_config
translation shim in two places. Replace all SupportedProviders usages
with ModelTransport, rename CLIENTS["google"] → CLIENTS["gemini"],
update provider branches + LLMError labels + reasoning-trace entries
accordingly. Trace JSONL now writes "provider": "gemini" instead of
"google" — consistent with the broader env-var rename cutover.
Also tidies up pre-existing basedpyright findings in tests/llm/test_model_config.py
(pydantic before-validator dict inputs + descriptor-proxy call).
ruff: clean. basedpyright: 0 errors, 0 warnings. Tests: 153/153 pass across
tests/utils/test_clients.py, tests/utils/test_length_finish_reason.py,
tests/llm/, tests/dialectic/, tests/deriver/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): finish the src/utils/clients.py → src/llm/ migration
honcho_llm_call_inner now delegates to request_builder.execute_completion
and execute_stream instead of re-implementing backend call scaffolding
inline. The new _effective_config_for_call helper carries per-call kwargs
(temperature, stop_seqs, thinking_budget_tokens, reasoning_effort) onto
the selected ModelConfig — or synthesizes a minimal config for the
test-only callers that pass provider+model directly. max_output_tokens
is zeroed on the effective config to preserve the current
"per-call max_tokens wins" semantic; honoring ModelConfig.max_output_tokens
is a separable correctness concern.
Side effect of routing through the new path: ConfiguredModelSettings'
thinking_budget_tokens validator now fires on synthesized configs.
test_anthropic_thinking_budget was asserting that a sub-1024 budget
propagated to Anthropic — bumped to 1024 to match what Anthropic actually
accepts.
Unified client construction. Promoted the cached client factories in
src/llm/__init__.py (get_anthropic_client, get_openai_client,
get_gemini_client, get_{anthropic,openai,gemini}_override_client) to
public API and added them to __all__. Promoted
credentials._default_transport_api_key → default_transport_api_key.
Deleted the duplicate _build_client and _default_credentials_for_provider
from clients.py; _client_for_model_config now falls through to the
public factories. CLIENTS dict and _get_backend_for_provider stay as the
mockable seam for the ~50 patch.dict(CLIENTS, {...}) test call sites.
Wired operator-configurable Gemini cached-content reuse end-to-end.
PromptCachePolicy moved from src/llm/caching.py into src/config.py so
ModelConfig can reference it as a field without a circular import;
caching.py re-exports the name for existing imports. Added
cache_policy: PromptCachePolicy | None on ConfiguredModelSettings,
FallbackModelSettings, ResolvedFallbackConfig, and ModelConfig.
resolve_model_config, _resolve_fallback_config, and
_select_model_config_for_attempt copy the field through.
honcho_llm_call_inner passes effective_config.cache_policy into
execute_completion / execute_stream, so operators opt in via
e.g. DERIVER_MODEL_CONFIG__CACHE_POLICY__MODE=gemini_cached_content
and the selection actually fires instead of sitting on a dead path.
New regression test test_cache_policy_reaches_gemini_backend asserts the
PromptCachePolicy object reaches the Gemini backend's extra_params.
ruff + basedpyright: clean. Tests: 154/154 pass across
tests/utils/test_clients.py, tests/utils/test_length_finish_reason.py,
tests/llm/, tests/dialectic/, tests/deriver/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): move all LLM orchestration into src/llm/ and delete clients.py
The 1624-line src/utils/clients.py has been carved up into focused modules
under src/llm/ and deleted. There is now one golden path for LLM
orchestration and no dual entrypoint.
New module layout:
src/llm/
__init__.py thin stable re-export surface
api.py public honcho_llm_call with retry + fallback + tool
loop delegation
executor.py honcho_llm_call_inner (single-call executor); bridges
to request_builder.execute_completion / execute_stream
tool_loop.py execute_tool_loop + stream_final_response, plus
assistant-tool-message and tool-result formatting
runtime.py AttemptPlan dataclass (replaces the loose
ProviderSelection NamedTuple), effective_config_for_call,
plan_attempt, per-retry temperature bump, attempt
ContextVar
registry.py single owner of CLIENTS dict + cached default and
override SDK-client factories + backend/history-adapter
selection + high-level get_backend(config)
conversation.py count_message_tokens, tool-aware message grouping,
truncate_messages_to_fit
types.py HonchoLLMCallResponse, HonchoLLMCallStreamChunk,
StreamingResponseWithMetadata, IterationData,
IterationCallback, ReasoningEffortType, VerbosityType,
ProviderClient
request_builder.py low-level request assembly (ModelConfig → backend
complete/stream); no longer owns credential resolution
credentials.py default_transport_api_key, resolve_credentials
caching.py gemini_cache_store; re-exports PromptCachePolicy
from src.config
backend.py Protocol + normalized result types
history_adapters.py provider-specific assistant/tool message shapes
structured_output.py
backends/ AnthropicBackend, OpenAIBackend, GeminiBackend
handle_streaming_response had no production callers; it is deleted. The
three tests that used it now drive honcho_llm_call_inner(stream=True,
client_override=...) directly, which exercises the same code path the
public API uses.
Dead credential passthrough removed. The ProviderBackend Protocol and
all three concrete backends no longer accept api_key / api_base — those
are baked into the underlying SDK client at registry construction time
and were being del'd everywhere they appeared. request_builder also
stops resolving and forwarding them.
Client construction is unified. The cached default-client factories
(get_anthropic_client, get_openai_client, get_gemini_client) and override
factories (get_*_override_client) are promoted to public API; the
module-level CLIENTS dict populates from them and remains the
patch.dict(CLIENTS, {...}) mocking seam tests rely on. Old duplicate
helpers (_build_client, _default_credentials_for_provider) are gone.
default_transport_api_key is promoted to public.
Application imports now come from src.llm (dreamer, dialectic, deriver,
summarizer, telemetry-adjacent tests). No code imports from
src.utils.clients anywhere in the repo.
ruff: clean. basedpyright: 0 errors, 0 warnings. Tests: 1013/1013 pass
across the entire non-infra test suite (excluding tests/unified,
tests/bench, tests/live_llm, tests/alembic).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): sanitize tool schemas for Gemini's function_declarations validator
Gemini's native-transport function-declarations validator accepts a narrow
subset of JSON-Schema / OpenAPI: type, format, description, nullable, enum,
properties, required, items, minItems, maxItems, minimum, maximum, title.
Anything else — additionalProperties, allOf, if/then/else, $ref, anyOf,
oneOf, $defs, patternProperties — triggers an INVALID_ARGUMENT 400 at call
time.
Our agent tool schemas in src/utils/agent_tools.py use several of those
(additionalProperties: false, allOf + if/then conditionals) because they
were authored for OpenAI strict-mode + Anthropic, which need the richer
vocabulary. GeminiBackend._convert_tools was passing them straight through.
Add _sanitize_schema(): walks the parameters tree and drops unsupported
keywords while preserving semantics for the keywords that hold user data
(properties maps field-name → sub-schema; required / enum are lists of
literals; items is a single sub-schema). Other backends are untouched and
continue to receive the full strict schemas.
Regression tests:
- test_gemini_sanitize_schema_strips_unsupported_keywords: confirms
additionalProperties, allOf + if/then, and $defs are stripped at nested
levels while legitimate fields survive.
- test_gemini_convert_tools_sanitizes_parameters_schema: end-to-end
_convert_tools output has no forbidden keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: fix tool calling syntax for gemini
* refactor(llm): normalize defaults, widen OpenAI reasoning-model routing
* chore: fix test
* fix(llm): address post-migration review feedback
* fix(llm): gemini robustness + dreamer specialist ergonomics
* chore: addres review comments
* chore: (docs) unrelease changelog addition
* chore: (docs) merge commit changes
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Erosika <eri@plasticlabs.ai>
* fix: further remove extraneous transactions
* fix: (search) use 2 phase function to reduce un-needed transaction
* fix: refactor agent search to perform external operations before making a transaction
* fix: reduce scope of queue manager transaction
* fix: (bench) add concurrency to test bench
* fix: address review findings for search dedup, webhook idempotency, and bench throttling
* Fix Leakage in non-session-scoped chat call (#526)
* fix: (search) reduce scope for peer based searches
* fix: tests
* fix: (test) address coderabbit comment
* fix: drop db param from deliver_webhook
---------
Co-authored-by: Rajat Ahuja <rahuja445@gmail.com>
* fix: dialectic held connection
* fix: (agent) pre-compute embeddings for agent tools
* fix: (tests) refactor tests to use smaller test db connections
* fix: Embedding client to branch depending on vector store
* fix: reflect dedup-skipped observations in created counts and isolate DB sessions in extract_preferences
* fix: (tests) update tests to match changes
* fix: expunge docs + don't pass in db to query_documents
---------
Co-authored-by: Rajat Ahuja <rahuja445@gmail.com>
* fix: adding strict checking and updating readme
* chore: changelog and version
* fix: Add strict validation to all Python and TypeScript classes
* fix: Address Code Rabit Comments
* fix: duplicate searchQuery param in typescript session.context()
* feat: add created_at, is_active fields, and get_message method
* feat: Add pagintion params to sdk
* fix: Remove lazy initalization behavior from sdks
* fix: Address File Upload Validation, add compatibility shims, address review comments
* chore: Docs updates
* fix: Convert session config from API format in Peer.sessions()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Pass all args to Session constructor in Peer.sessions()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Preserve createdAt in Peer.refresh(), pass all data in session.peers()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: changelog for ts
* fix: Review Comments
* fix: Add createAt and to peers call
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RepresentationManager._query_documents_recent() and
._query_documents_most_derived() do not filter soft-deleted documents,
unlike every other document query function in the codebase. This causes
the deriver's working representation to include documents that are being
garbage-collected.
Refs #444
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Add bounds to gemini client
* fix: Prevent empty summaries from being saved to DB (HONCHO-M7)
Raise LLMError on blocked Gemini responses (SAFETY, RECITATION, etc.)
so retry/backup-provider logic triggers. Treat empty LLM responses in
the summarizer as fallback instead of persisting empty strings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: Summary Eval via Locomo
* fix: Code Rabbit Comments
* fix: Code Rabbit Comments
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: implement async workspace deletion with active session checks
- Updated the DELETE /workspaces/:id endpoint to return 202 Accepted, indicating that the deletion request is processed in the background.
- Added a check for active sessions before allowing workspace deletion, raising a ConflictException if any exist.
- Updated related tests to ensure proper handling of active sessions during workspace deletion.
* fix: Address review issues
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* chore: parallelize tests for speed
* fix: truncate all tables after each test
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* refactor: enhance specialist exploration and observation processes
- Updated the orchestration logic to allow specialists to explore freely with optional hints from high-surprisal observations.
- Removed predefined probing questions, enabling a more dynamic approach to observation gathering.
- Adjusted the deduction and induction specialists to utilize the current peer card context and exploration hints in their prompts.
- Improved documentation within the code to clarify the roles and responsibilities of specialists in the observation process.
* feat: add thorough dream test
* refactor: standardize hint terminology and dedup peer card context
* feat: make session_name nullable for documents and update related SDKs
- Introduced a migration to make the `session_name` column in the documents table nullable, allowing for sessionless dreams.
- Updated Python and TypeScript SDKs to reflect the optional nature of `session_id` in conclusion creation and related methods.
- Enhanced tests to cover scenarios for creating conclusions without a session ID, ensuring proper handling of sessionless conclusions.
- Adjusted documentation and type definitions to clarify the optional session context in various components.
* chore: add migration test
* fix: ensure orphaned sessions exist during downgrade for nullable session_name migration
* refactor: update semantic search parameter from `last_user_message` to `search_query` across documentation and SDKs
- Changed references in documentation and code to use `search_query` instead of `last_user_message` for fetching semantically relevant observations and conclusions.
- Updated related function signatures and descriptions in Python and TypeScript SDKs to reflect this change.
- Adjusted tests to ensure compatibility with the new parameter naming.
* chore: openapi v3 formatted how we like it
* fix: reorder docs, update examples in README, update skills
* fix: message type option in sdk reference
* chore: update remaining getcontext and representation language
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* chore: 3.0 honcho and 2.0 sdks changelog
fix: use PeerContextResponse in peer.ts
* chore: move docs to /v3/, build SDKs
* chore: code review
* feat: [WIP] migrate away from stainless in typescript sdk
* chore: move api from /v2/ to /v3/
* feat: no-stainless typescript with real tests
* feat: migrate python sdk off of stainless
* feat: clean typescript sdk
* chore: add tests for ts http client
* fix: rewrite entire python sdk in new format, update typescript sdk to use `configuration` not `config` for consistency with API
* fix: clean up SDKs, synchronize
* chore: update sdk examples
* chore: update OpenAPI documentation and SDK examples to reflect changes
* fix: better test
* fix: install deps in test runner, improve robustness of streaming in sdk, coderabbit nits
* fix: standardize around camelCase in TS SDK
* refactor: update configuration handling in SDKs to use typed models for workspace, session, and peer configurations
* docs: clarify queue status usage and remove polling methods from SDKs
add claude skills for migrations
* chore: fix links in docs
* feat: add deriver flush mode to bypass batch token threshold
- Introduced `is_deriver_flush_enabled` function to check if flush mode is active.
- Updated `QueueManager` to conditionally apply batch token thresholds based on flush mode.
- Enhanced `UnifiedTestExecutor` to enable flush mode via Redis.
- Added `flush` parameter to test cases to facilitate testing of flush mode behavior.
- Updated various test cases to utilize the new flush functionality.
* feat: implement schedule_dream functionality in SDKs, use in unified test runner
- Added `schedule_dream` method to both Python and TypeScript SDKs for scheduling dream tasks.
- Updated HTTP routes to include endpoint for scheduling dreams.
- Enhanced test runner to utilize the new `schedule_dream` method for scheduling actions.
- Updated TypeScript client to support the new scheduling functionality with appropriate parameters.
* feat: update single deriver task to support multiple observers
- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.
* refactor: update enqueue tests to support deduplication of queue items with multiple observers
- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.
* fix: add backwards compatibility for representation work unit keys and payload observers
* feat: update dialectic configuration and introduce cost calculator
- Adjusted LLM and dialectic settings in `.env.template`, `config.toml.example`, and `src/config.py` to reduce maximum tool output characters and session history tokens for cost efficiency.
- Implemented a new `dialectic_cost_calculator.py` script to estimate costs based on reasoning levels and model pricing.
- Enhanced `DialecticAgent` to utilize minimal tools and adjusted output token settings based on reasoning level to optimize performance and reduce costs.
* feat: add reasoning level to chat input in unified test runner
- Enhanced the `UnifiedTestExecutor` to include a `reasoning_level` parameter in the chat method call.
- Updated the `QueryAction` model to support the new `reasoning_level` attribute, allowing for more nuanced chat interactions.
* feat: run deriver once for multiple observers (#335)
* feat: update single deriver task to support multiple observers
- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.
* refactor: update enqueue tests to support deduplication of queue items with multiple observers
- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.
* fix: add backwards compatibility for representation work unit keys and payload observers
* feat: refactor benchmark runners to share common functionality
- Introduced a new `runner_common.py` module containing shared utilities for benchmark test runners, including common argument parsing, client creation, and queue management.
- Updated `BEAMRunner`, `LoCoMoRunner`, and `LongMemEvalRunner` to inherit from `RunnerMixin`, leveraging shared functionality for metrics collection and logging.
- Added `reasoning_level` and `redis_url` parameters to runner constructors for enhanced configuration.
- Streamlined argument parsing by utilizing `add_common_arguments` for shared command-line options across all runners.
* fix: update last_user_message handling to use message content instead of ID
* fix: standardize config vs configuration
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* feat: implement forced batching for representation tasks and adjust max tokens
- Updated `REPRESENTATION_BATCH_MAX_TOKENS` to 1024 in `config.py`.
- Enhanced `get_and_claim_work_units` in `QueueManager` to enforce batching based on token thresholds.
- Added tests to ensure representation work units are only claimed when token counts meet or exceed the threshold.
- Introduced a synthesis prompt for tool execution to improve final response generation.
* chore: update queue-status docs to match new behavior
* fix: make representation work query efficient
* fix: Align alembic models and force dreams on for tests
* fix: clean queue between tests
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* fix: swap to reasoning config in unified test files, handle new representation output format
* fix: resolve bugs in dream scheduling -- message from a peer should cancel all dreams observing them, and document counts should be checked at moment of dream, not in enqueue. clean up and test
* feat: run unified tests in CI
* fix: attempt use aws secrets manager
* fix: temp add verification workflow
* fix: CodeRabbit comments
* fix: remove debugging step
* fix: only run on main
* fix: add UNIFIED_TEST_LOG_LEVEL env var; default to WARNING
Rajat Ahuja
qxxaa
Vineeth Voruganti
ajspig
Hart
LRRuan
3un01a
doria
Benjamin McCormick