* Structured outputs for dialectic
* cleanup
* rename json_schema_to_pydantic to clarify it's not a general schema converter
* clean up schema DoS guards
* simplification and cleanup of schema conversion
* chore: ruff and pyproject toml
* chore: basedpyright cleanup in test
* fix: some needed unrelated test failures
* test(schema_conversion-and-anthropic-backend): expand test coverage
include table tests
* fix(llm): support combined tool calling and structured output across backends
- OpenAI: parse() 500s on non-strict function tools; route tool-carrying
structured requests through create() with an explicit json_schema
response_format (mirrors the streaming path)
- Anthropic: skip the '{' JSON prefill when tools are present so tool_use
blocks stay reachable; make the schema instruction conditional and rely
on parse + repair
- Gemini: native response_schema + function calling is rejected before
Gemini 3; with tools present, inject a schema instruction into the final
turn instead and rely on parse + repair
- All backends: tool-call turns carry no consumable content, so skip
structured-output parsing on them
Extracted from the dialectic structured-output branch (DEV-1652) so the
transport layer can land independently.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): exercise combined tools + structured output per provider
Two-turn live flow per backend: a forced tool-call turn (structured
parsing must be skipped) followed by a replay turn that must return a
schema-conforming answer with tools still attached. Asserts the
provider-specific request shaping: no parse() for OpenAI (500s on
non-strict tools), no '{' prefill for Anthropic, no native
response_schema for Gemini.
Verified against live OpenAI (gpt-4.1, gpt-5, gpt-5.4, gpt-5.4-mini)
and Gemini (gemini-2.5-flash).
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(unified): dialectic chat with response_format schema under tool use
Adds response_format pass-through to the unified runner's chat query and
a test case that forces the dialectic tool loop (reasoning off + global
enumeration question) while requiring a schema-conforming JSON answer —
end-to-end coverage of the combined tools + structured output transport
path on whichever provider each level is configured with.
Verified locally against a full harness run (json_match assertions pass;
the llm_judge assertion additionally runs in CI where the Anthropic key
is available).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: some needed unrelated test failures
* ci: add label-triggered live LLM test workflow
Adding the run-live-llm label to a PR (or workflow_dispatch) runs
tests/live_llm/ against real provider APIs — the only place the
--live-llm suite runs in CI. Reuses the unified-tests environment and
its Secrets Manager staging-dotenv resolution for provider keys; runs
on ubuntu-latest (no Fly runner, no Docker — the suite only touches the
LLM backends). Pins LIVE_LLM_ANTHROPIC_45_PLUS_MODELS=claude-sonnet-4-5
since the Anthropic family has no default models and would otherwise
silently collect empty.
Opt-in by design: live model behavior is variable, so this is a signal,
not a required check.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: run live LLM tests on main pushes touching the transport
Mirrors unified-tests' push trigger, scoped to paths that can affect
the live suite (src/llm/, config, the tests, deps, and the workflow
itself) so provider API calls aren't spent on unrelated changes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: disable auth in live LLM test environment
The staging dotenv sets AUTH_USE_AUTH=true without a usable JWT secret,
and src/config.py validates the pair at import time — the same reason
unified-tests overrides it. This suite never runs the API server.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): fix gpt-5.4 reasoning_effort and gemini replay-turn flake
- test_live_openai: gpt-5.4 dropped 'minimal' from the reasoning_effort
vocabulary, so the gpt5 caching test 400'd — and the OpenAI backend's
BadRequestError terminal swallowed it into an empty CompletionResult.
Pick the effort per model generation.
- test_live_tools_structured_output: use tool_choice='auto' on the
replay turn, matching the production dialectic loop (which never
forces 'none') — NONE mode is what provoked gemini-2.5-flash's empty
candidates. Drop the temperature pin so retries actually resample,
and treat a repeat tool call as a retryable attempt.
Verified live: full suite green, gemini 4/4 consecutive passes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: fail live LLM run when no staging secret was loaded
If the latest-tag fetch fails and no second tag exists, the fallback
step is skipped rather than failed, and the job would proceed without
provider keys — every test then skips via require_provider_key and the
run goes green. Guard on both fetch outcomes so that path fails loudly.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(live-llm-tests-GHA): remove extra comments
* feat(structured-output): enable non-recursive schema references
* docs(structured-outputs): clean up new doc
* test(structured-output): fix caching refs memory leak, add tests
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(conclusions): expose reasoning level + allow filtering by level
The `level` of a conclusion (explicit / deductive / inductive /
contradiction) was filterable server-side but stripped from the
`Conclusion` response and not surfaced in either SDK. This adds it
end-to-end so callers can list explicit-only ("not dreamed on")
conclusions without dropping to raw HTTP.
- api: add `level` to the Conclusion response schema
- python sdk: `ConclusionLevel` type, `level` on Conclusion/response,
`level=` kwarg on ConclusionScope.list() and the async variant
- ts sdk: `ConclusionLevel` type, `level` on Conclusion/response,
`level` option on list()
- tests: assert level is exposed; add level-filter list test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(conclusions): use generic filters= on list() instead of level= kwarg
Match the documented SDK convention (peers/sessions/messages all take a
generic `filters` dict passed through to the same dynamic server-side
filter logic) instead of a one-off `level=` kwarg. `level` filtering now
works as `list(filters={"level": "explicit"})` alongside any other
supported filter/operator.
The `level` field on the Conclusion response (added in the previous
commit) is kept — it's still not otherwise returned by the API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(conclusions): allow filtering by level on query() in py + ts SDKs
The branch's level-filter work exposed `filters=` on `list()` but left
`query()` (semantic search) hardcoding `{observer, observed}`, so callers
could filter the list endpoint by reasoning level but not semantic search —
asymmetric in both SDKs.
- Python: add keyword-only `filters` to `ConclusionScope.query` and
`ConclusionScopeAio.query`, merged over the scope's observer/observed.
- TypeScript: add optional `filters` arg to `ConclusionScope.query`,
mirroring the existing `list()` change.
The server `/conclusions/query` endpoint already honors filters in the body
(verified against production), so this is purely SDK surface parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(filters): document filtering conclusions by reasoning level
The using-filters page covered workspaces/peers/sessions/messages but not
conclusions. Add a "Filtering Conclusions" section showing level-based
filtering on both list() and query(), including the common "explicit only"
(exclude dream-derived) case and the in[deductive,inductive] inverse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(conclusions): simplify filter merge to a single dict spread
Replace the merged_filters + if-block pattern in list()/query() (py sync,
aio, ts) with a single dict spread that layers the caller's filters over the
scope's observer/observed (and session). No behavior change — same merge
order (caller wins) — just less code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(conclusions): reject scope-managed keys in SDK conclusion filters
The generic filters= argument on ConclusionScope.list()/query() spread
user-supplied filters last, so a stray observer/observed/session key
silently overrode the scope and returned data from a different peer
pair. Add a fail-loud guard in both the Python and TypeScript SDKs that
rejects scope-managed filter keys with a clear error, directing callers
to peer.conclusions / conclusions_of(target) and the session= parameter.
session_id remains a valid filter on query() (which has no dedicated
session parameter).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* fix: adding strict checking and updating readme
* chore: changelog and version
* fix: Add strict validation to all Python and TypeScript classes
* fix: Address Code Rabit Comments
* fix: duplicate searchQuery param in typescript session.context()
* feat: add created_at, is_active fields, and get_message method
* feat: Add pagintion params to sdk
* fix: Remove lazy initalization behavior from sdks
* fix: Address File Upload Validation, add compatibility shims, address review comments
* chore: Docs updates
* fix: Convert session config from API format in Peer.sessions()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Pass all args to Session constructor in Peer.sessions()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Preserve createdAt in Peer.refresh(), pass all data in session.peers()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: changelog for ts
* fix: Review Comments
* fix: Add createAt and to peers call
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: make session_name nullable for documents and update related SDKs
- Introduced a migration to make the `session_name` column in the documents table nullable, allowing for sessionless dreams.
- Updated Python and TypeScript SDKs to reflect the optional nature of `session_id` in conclusion creation and related methods.
- Enhanced tests to cover scenarios for creating conclusions without a session ID, ensuring proper handling of sessionless conclusions.
- Adjusted documentation and type definitions to clarify the optional session context in various components.
* chore: add migration test
* fix: ensure orphaned sessions exist during downgrade for nullable session_name migration
* chore: 3.0 honcho and 2.0 sdks changelog
fix: use PeerContextResponse in peer.ts
* chore: move docs to /v3/, build SDKs
* chore: code review
* feat: [WIP] migrate away from stainless in typescript sdk
* chore: move api from /v2/ to /v3/
* feat: no-stainless typescript with real tests
* feat: migrate python sdk off of stainless
* feat: clean typescript sdk
* chore: add tests for ts http client
* fix: rewrite entire python sdk in new format, update typescript sdk to use `configuration` not `config` for consistency with API
* fix: clean up SDKs, synchronize
* chore: update sdk examples
* chore: update OpenAPI documentation and SDK examples to reflect changes
* fix: better test
* fix: install deps in test runner, improve robustness of streaming in sdk, coderabbit nits
* fix: standardize around camelCase in TS SDK
* refactor: update configuration handling in SDKs to use typed models for workspace, session, and peer configurations
* docs: clarify queue status usage and remove polling methods from SDKs
add claude skills for migrations
* chore: fix links in docs
* feat: add deriver flush mode to bypass batch token threshold
- Introduced `is_deriver_flush_enabled` function to check if flush mode is active.
- Updated `QueueManager` to conditionally apply batch token thresholds based on flush mode.
- Enhanced `UnifiedTestExecutor` to enable flush mode via Redis.
- Added `flush` parameter to test cases to facilitate testing of flush mode behavior.
- Updated various test cases to utilize the new flush functionality.
* feat: implement schedule_dream functionality in SDKs, use in unified test runner
- Added `schedule_dream` method to both Python and TypeScript SDKs for scheduling dream tasks.
- Updated HTTP routes to include endpoint for scheduling dreams.
- Enhanced test runner to utilize the new `schedule_dream` method for scheduling actions.
- Updated TypeScript client to support the new scheduling functionality with appropriate parameters.
* feat: update single deriver task to support multiple observers
- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.
* refactor: update enqueue tests to support deduplication of queue items with multiple observers
- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.
* fix: add backwards compatibility for representation work unit keys and payload observers
* feat: update dialectic configuration and introduce cost calculator
- Adjusted LLM and dialectic settings in `.env.template`, `config.toml.example`, and `src/config.py` to reduce maximum tool output characters and session history tokens for cost efficiency.
- Implemented a new `dialectic_cost_calculator.py` script to estimate costs based on reasoning levels and model pricing.
- Enhanced `DialecticAgent` to utilize minimal tools and adjusted output token settings based on reasoning level to optimize performance and reduce costs.
* feat: add reasoning level to chat input in unified test runner
- Enhanced the `UnifiedTestExecutor` to include a `reasoning_level` parameter in the chat method call.
- Updated the `QueryAction` model to support the new `reasoning_level` attribute, allowing for more nuanced chat interactions.
* feat: run deriver once for multiple observers (#335)
* feat: update single deriver task to support multiple observers
- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.
* refactor: update enqueue tests to support deduplication of queue items with multiple observers
- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.
* fix: add backwards compatibility for representation work unit keys and payload observers
* feat: refactor benchmark runners to share common functionality
- Introduced a new `runner_common.py` module containing shared utilities for benchmark test runners, including common argument parsing, client creation, and queue management.
- Updated `BEAMRunner`, `LoCoMoRunner`, and `LongMemEvalRunner` to inherit from `RunnerMixin`, leveraging shared functionality for metrics collection and logging.
- Added `reasoning_level` and `redis_url` parameters to runner constructors for enhanced configuration.
- Streamlined argument parsing by utilizing `add_common_arguments` for shared command-line options across all runners.
* fix: update last_user_message handling to use message content instead of ID
* fix: standardize config vs configuration
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
Eugene Eisenstein
ajspig
Vineeth Voruganti
doria
Benjamin McCormick
Rajat Ahuja