- Conclusion class: sourceIds + timesDerived fields (appended to the
constructor signature with defaults, so existing positional callers
are unaffected)
- ConclusionScope: get(), getMany() (batch fetch via the list endpoint
with an id-in filter, chunked at the 100 page-size cap), and derived()
(paginated reverse traversal via GET /conclusions/{id}/derived)
- ConclusionResponse type: optional source_ids / times_derived
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add GET /conclusions/{id}/derived to walk the reasoning tree upward
(conclusions that list the given one in their source_ids), paginated,
wired to the previously-unused get_child_observations crud (now returns
a Select for pagination; the get_reasoning_chain agent tool updated to
execute the statement itself)
- Support source_ids in the filter DSL: JSONB containment (@>) instead of
invalid ILIKE SQL, with explicit JSONB binds for scalars; also fixes
'contains' on metadata/configuration JSONB columns
- Make id, level, source_ids, times_derived explicit entries in the
documents filter allowlist
- Python SDK: derived() and get_many() (batch fetch premises via the list
endpoint with an id-in filter), sync + async
- Tests for the new endpoint, filters, SDK methods, and previously
uncovered get_reasoning_chain tool handler
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(llm): support per-request provider timeouts
* fix(llm): convert Gemini timeout to milliseconds
* fix(llm): validate Gemini HTTP options
* test(llm): type Anthropic stream context args
* test(llm): live per-request timeout coverage for all providers
Two live checks per provider: a generous timeout asserted at the SDK
call boundary, and a tight timeout that must abort well under the 600s
client default. Gemini's async transport can be aiohttp, so its tight
timeout surfaces as asyncio.TimeoutError rather than httpx.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style(tests): drop extra blank line in anthropic backend test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(llm): validate provider_params.timeout at config load
Move the timeout coercion into src.config as coerce_provider_timeout and
run it from a field validator on ModelOverrideSettings.provider_params, so
a bad value in config.toml/env fails at startup with the exact config path
instead of surfacing per-request as a retried 500. Good values normalize
to float seconds at load. The per-request guard in src.llm.backend now
delegates to the same coercion (wrapping ValueError in ValidationException)
and continues to cover extra_params passed programmatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: document provider_params.timeout load-time validation and gotchas
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(llm): address review nits on timeout plumbing
Apply eisene's review feedback:
- Rename PROVIDER_TIMEOUT_ERROR → PROVIDER_TIMEOUT_ERROR_TEXT
- Move request_timeout_from_extra_params from backend.py (pure
dataclasses) to request_builder.py (request assembly)
- Add comment explaining Gemini's ms timeout conversion
- Generalize _normalize_extra_params with _strip_none_params helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Aakash Kattelu <aakash@plasticlabs.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: guard against top_k=0 reaching the vector store
Turbopuffer rejects top_k=0 with a 400 ('top_k must be between 1 and
10000'). The fix returns [] for a non-positive top_k, before the embedding call), and
floor the semantic budget at 1 so an explicitly requested search isn't
silently allocated zero.
* fix: comments
* fix: apply session scoping to all working-representation query paths
session_name was only applied to the recent-documents query in
RepresentationManager; the semantic and most-derived paths ignored it,
so limit_to_session leaked cross-session conclusions into perspectives.
- Thread a session allowlist (session_names) uniformly through all
three query paths; pushed down to pgvector and external vector stores
- Accept a list so the upcoming session-allowlist API reuses this path
- Fail closed on an empty allowlist (downstream stores drop empty IN
clauses, which would silently widen scope)
Fixes DEV-1994
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: bare-list membership sugar in the filter DSL
{"session_id": ["s1", "s2"]} is now shorthand for
{"session_id": {"in": [...]}} on regular columns, generically
(peer_id, etc.). JSONB metadata columns are excluded — a bare list
there keeps JSONB containment semantics, unchanged.
Previously a bare list on a regular column compiled to a type-mismatched
equality that matched nothing, so this is strictly additive.
Also translates the same shape in the turbopuffer/lancedb filter
builders, and fixes lancedb dropping empty IN clauses (fail-open) —
an empty membership list now emits an always-false condition.
Groundwork for DEV-1995 (session allowlist via the existing filters
DSL, no new API params)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: session allowlist on dialectic and representation via filters
Adds a constrained 'filters' body to peer.chat and /representation —
the same DSL search and conclusions already accept, supporting only the
session_id key (a session id, a bare list, or {"in": [...]}).
Unsupported keys and shapes are rejected with 422, never silently
ignored. Composes with session_id (must be included in the allowlist
when both are given). Capped at 1,000 sessions per request.
Enforcement is uniform at every recall chokepoint, fail-closed:
- dialectic prefetch + search_memory: conclusion recall restricted to
the allowlist; dream docs (session_name IS NULL) excluded
- message tools (search/grep/date-range/temporal/context/history):
strict intersection of allowlist and observer session membership
- get_reasoning_chain: unavailable under an allowlist (chains traverse
provenance across sessions and cannot be scoped without leaking)
- empty allowlist short-circuits to empty results everywhere
Auth: workspace keys pass the allowlist as-given; peer-scoped JWTs must
be a member of every allowlisted session (403 otherwise), mirroring the
existing single-session check.
Fixes DEV-1995
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: fail closed on empty session allowlist across all filter builders
Empty session allowlists relied solely on the early-return guard in
_get_working_representation_internal. The layers below it were
inconsistent, so a future direct caller (the DEV-1995 allowlist API)
would silently widen scope instead of failing closed:
- _build_filter_conditions used a truthiness check; an empty list was
treated like None and dropped the filter. Now uses `is not None`,
matching the recent/most-derived SQL paths.
- turbopuffer emitted a bare `In []` with undocumented (possibly
fail-open) semantics. Now emits an explicit always-false predicate,
mirroring lancedb's `1 = 0`.
Also extract the duplicated JSONB column tuple in filter.py to a
JSONB_COLUMNS constant.
Tests exercise each fail-closed guarantee at the layer it lives, rather
than masking it behind the early-return guard.
* fix: address tests
* fix(crud): fail closed when session_name is outside the allowlist
search/grep/history helpers scoped to a single session_name ignored the
session_names allowlist entirely — a caller could read a session the
allowlist forbids. The API routes guarded this with a 422, but the
dialectic tools call these CRUD functions directly and bypassed it.
Enforce it at the boundary: return [] when session_name is set and not in
the allowlist, across _semantic_search_messages (covers search_messages +
search_messages_temporal), grep_messages, get_messages_by_date_range,
get_recent_history, and get_observation_context.
Also rename the public param allowed_sessions -> session_names for
consistency with representation.py / chat.py / peers.py; the resolved
intersection keeps its distinct name allowed_session_names.
* fix: test
* fix(scopes): tighten and consolidate session allowlist per review
Addresses review feedback on the session allowlist (DEV-1995).
Behavior changes:
- Auth gate on peers.chat now uses active membership (left_at IS NULL)
via get_peer_session_names(active_only=True), matching the adjacent
is_peer_in_session check on options.session_id. Previously a peer that
had left a session was denied when naming it directly but permitted
when naming it in filters.session_id.
- Scoped conclusion recall is restricted to level == "explicit"
(ALLOWLIST_SAFE_LEVELS). Dream-derived conclusions are stamped with a
single session_name but synthesized across all sessions, so that stamp
can't be scoped on. Applied at all four recall paths. Unscoped recall
is unchanged. Follow-up to give conclusions an authoritative
source-session set is tracked in DEV-2201.
- The allowlist gate checks `is not None` rather than truthiness, so
filters={"session_id": []} reaches it instead of being skipped.
Refactors:
- New crud.message.resolve_session_scope replaces four near-identical
copies of the allowlist-membership intersection. Returns
(allowlist, deny) and never returns an empty list, so the None vs []
distinction that external stores fail open on lives in one tested
place. Takes db=None and opens its own short-lived session only when
distinction that external stores fail open on lives in one tested
place. Takes db=None and opens its own short-lived session only when
an observer lookup is needed, preserving external-lookup-first
ordering on the vector-store path.
- extract_session_allowlist takes must_include, collapsing the
session_id-in-allowlist check duplicated across both peer routes.
- DialecticAgent._select_tools dedupes the two toolset-selection blocks
and drops get_reasoning_chain under an allowlist, rather than paying
for the schema plus a wasted turn to return a refusal.
- Rename session_names -> session_allowlist across crud, agent tools,
dialectic and routes, to remove the one-character ambiguity with
session_name. Internal only; the public filters.session_id surface is
unchanged.
Docs:
- session_allowlist documented across all message and recall entry
points, including the None / [] / populated contract.
- session_name marked deprecated for scoping. Not removed and not
aliased: it also pins the query to one session, bypasses observer
scoping, and drives session-history injection into the dialectic
prompt, so it has no drop-in replacement.
- Note at the Document branch in utils/filter.py that the raw-key
fallback is load-bearing for session scoping.
Tests: 20 -> 39 in tests/test_session_allowlist.py, covering the
peer-scoped JWT gate (member, non-member, left-session, workspace-key
bypass, empty allowlist), the resolve_session_scope tri-state including
the no-DB-checkout path, must_include, and the level narrowing.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: redact Redis password from cache connection logs
The cache client logged the full Redis URL — including the password —
at INFO and WARNING levels on every connection attempt and failure.
This exposed the live Redis credential in stdout/container logs and any
downstream log aggregation.
Add _redact_cache_url() to mask the password component before logging.
URLs without a password are returned unchanged.
Closes#866
* fix: handle malformed URLs and IPv6 in _redact_cache_url
Address review feedback from VVoruganti and CodeRabbit:
- Wrap urlparse/urlunparse in try/except so malformed URLs (e.g. invalid
port) don't raise ValueError inside except blocks, which would crash
startup instead of degrading gracefully
- Preserve IPv6 brackets (e.g. [::1]) in reconstructed URLs
- Add Google-style Args/Returns docstring sections
- Add unit tests for password masking, no-password URLs, IPv6,
malformed inputs, and the invalid-port regression
* test: use real secrets in redaction test fixtures
Three fixtures were weakened by copy-paste mangling: literal '***'
placeholders instead of real passwords (assertions trivially true),
an unescaped '#' that truncated netloc parsing via the URL fragment,
and a no-password case that actually contained userinfo. Restore
inputs that genuinely exercise the masking paths.
* fix: never leak password through malformed-URL fallback
The catch-all fallback returned the original URL when parsing failed,
so a Redis URL with a password and an invalid port (typo, out-of-range)
was logged in clear text - the exact leak #866 exists to fix. Narrow
the handling: .port access gets its own try/except (invalid port is
omitted from the output; userinfo/hostname masking never raises), and
the outer fallback now returns a generic placeholder instead of the
raw input. Tightened the invalid-port test to assert the password is
absent and added out-of-range-port and unparseable-URL cases.
* fix: redact secrets in query params and scheme-less URLs
_redact_cache_url only masked userinfo, but a credential can reach the
URL through two other real configuration paths: redis-py accepts
?password= (all querystring options become client kwargs) and cashews
accepts ?secret= (its HMAC signing key) - and honcho's own default
CACHE.URL already uses a query param (?suppress=true), so this is the
expected configuration style. Separately, a URL missing its scheme
(':pass@host:6379/0') parses with an empty netloc, making the password
invisible to .password and echoing it back verbatim.
Mask sensitive query values in place on the raw query string (no
decode/re-encode, so non-secret params are preserved byte-for-byte)
and return the generic placeholder for @-carrying strings with no
parseable authority. Verified with a 20k-case randomized fuzz run in
addition to the unit tests: no functional credential reaches the
output.
Add a PEP 508 marker so lancedb is not installed on darwin/x86_64, wrap the
LanceDB vector store import in try/except for a clear config error, and
regenerate uv.lock.
Branch rebased onto upstream/main; prior src/utils/clients.py CI tweak is
obsolete because LLM wiring moved under src/llm/.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: apply session scoping to all working-representation query paths
session_name was only applied to the recent-documents query in
RepresentationManager; the semantic and most-derived paths ignored it,
so limit_to_session leaked cross-session conclusions into perspectives.
- Thread a session allowlist (session_names) uniformly through all
three query paths; pushed down to pgvector and external vector stores
- Accept a list so the upcoming session-allowlist API reuses this path
- Fail closed on an empty allowlist (downstream stores drop empty IN
clauses, which would silently widen scope)
Fixes DEV-1994
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: bare-list membership sugar in the filter DSL
{"session_id": ["s1", "s2"]} is now shorthand for
{"session_id": {"in": [...]}} on regular columns, generically
(peer_id, etc.). JSONB metadata columns are excluded — a bare list
there keeps JSONB containment semantics, unchanged.
Previously a bare list on a regular column compiled to a type-mismatched
equality that matched nothing, so this is strictly additive.
Also translates the same shape in the turbopuffer/lancedb filter
builders, and fixes lancedb dropping empty IN clauses (fail-open) —
an empty membership list now emits an always-false condition.
Groundwork for DEV-1995 (session allowlist via the existing filters
DSL, no new API params)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: fail closed on empty session allowlist across all filter builders
Empty session allowlists relied solely on the early-return guard in
_get_working_representation_internal. The layers below it were
inconsistent, so a future direct caller (the DEV-1995 allowlist API)
would silently widen scope instead of failing closed:
- _build_filter_conditions used a truthiness check; an empty list was
treated like None and dropped the filter. Now uses `is not None`,
matching the recent/most-derived SQL paths.
- turbopuffer emitted a bare `In []` with undocumented (possibly
fail-open) semantics. Now emits an explicit always-false predicate,
mirroring lancedb's `1 = 0`.
Also extract the duplicated JSONB column tuple in filter.py to a
JSONB_COLUMNS constant.
Tests exercise each fail-closed guarantee at the layer it lives, rather
than masking it behind the early-return guard.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: enforce explicit-document session purity in dedup/merge paths
Audit for DEV-2000 (Scopes RFC prerequisite): explicit-level documents must
stay session-pure so scope memory can be built by copying explicit documents
between collections. Two classes of violation were possible:
- Exact-content and semantic dedup in crud/document.py matched candidates
with no level or session scoping, so an explicit document could be
reinforced by — or soft-deleted in favor of — a same-content document from
a different session or a different level (silently merging cross-session
derivations into one row).
- The generic create_observations tool handler accepted level='explicit'
from agents with no message context (dreamer/dialectic), which would mint
session-less explicit documents.
Enforcement (refuse, never rewrite):
- create_documents refuses explicit documents with a null session_name
- exact dedup keys on (content, level, session-for-explicit); derived levels
keep cross-session consolidation
- is_rejected_duplicate scopes candidate search to the same level, and the
same session for explicit documents
- the create_observations tool rejects explicit-level input outside message
ingestion (deriver) context
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: add card_refresh dream type for event-driven peer-card updates
Adds a lightweight dream variant (DEV-2000, Scopes RFC prerequisite) that
runs ONLY the peer-card update — for event-driven refreshes such as scope
membership changes and cold starts:
- DreamType.CARD_REFRESH alongside OMNI; dispatched by process_dream to a
new run_card_refresh_dream orchestration
- CardRefreshSpecialist: restricted to get_recent_observations,
search_memory, and update_peer_card (no observation-mutating tools), with
a low tool-iteration cap of min(6, DREAM.MAX_TOOL_ITERATIONS)
- rebuild=True mode carried in the dream payload: the existing card is NOT
injected into the prompt and the specialist rebuilds it solely from
observations present in the collection (for use after removals)
- enqueue-able via the manual enqueue_dream path (bypasses volume gates);
the work-unit key already embeds the dream type so a card refresh never
collides with a pending omni dream. POST /v3/workspaces/{id}/schedule_dream
accepts dream_type=card_refresh plus the rebuild flag
- card refreshes never advance the omni dream guard pair
(last_dream_at / last_dream_document_count)
- shared PEER CARD prompt section extracted (verbatim) from
DeductionSpecialist for reuse; CallPurpose gains dream.card_refresh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: fix tests
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Structured outputs for dialectic
* cleanup
* rename json_schema_to_pydantic to clarify it's not a general schema converter
* clean up schema DoS guards
* simplification and cleanup of schema conversion
* chore: ruff and pyproject toml
* chore: basedpyright cleanup in test
* fix: some needed unrelated test failures
* test(schema_conversion-and-anthropic-backend): expand test coverage
include table tests
* fix(llm): support combined tool calling and structured output across backends
- OpenAI: parse() 500s on non-strict function tools; route tool-carrying
structured requests through create() with an explicit json_schema
response_format (mirrors the streaming path)
- Anthropic: skip the '{' JSON prefill when tools are present so tool_use
blocks stay reachable; make the schema instruction conditional and rely
on parse + repair
- Gemini: native response_schema + function calling is rejected before
Gemini 3; with tools present, inject a schema instruction into the final
turn instead and rely on parse + repair
- All backends: tool-call turns carry no consumable content, so skip
structured-output parsing on them
Extracted from the dialectic structured-output branch (DEV-1652) so the
transport layer can land independently.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): exercise combined tools + structured output per provider
Two-turn live flow per backend: a forced tool-call turn (structured
parsing must be skipped) followed by a replay turn that must return a
schema-conforming answer with tools still attached. Asserts the
provider-specific request shaping: no parse() for OpenAI (500s on
non-strict tools), no '{' prefill for Anthropic, no native
response_schema for Gemini.
Verified against live OpenAI (gpt-4.1, gpt-5, gpt-5.4, gpt-5.4-mini)
and Gemini (gemini-2.5-flash).
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(unified): dialectic chat with response_format schema under tool use
Adds response_format pass-through to the unified runner's chat query and
a test case that forces the dialectic tool loop (reasoning off + global
enumeration question) while requiring a schema-conforming JSON answer —
end-to-end coverage of the combined tools + structured output transport
path on whichever provider each level is configured with.
Verified locally against a full harness run (json_match assertions pass;
the llm_judge assertion additionally runs in CI where the Anthropic key
is available).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: some needed unrelated test failures
* ci: add label-triggered live LLM test workflow
Adding the run-live-llm label to a PR (or workflow_dispatch) runs
tests/live_llm/ against real provider APIs — the only place the
--live-llm suite runs in CI. Reuses the unified-tests environment and
its Secrets Manager staging-dotenv resolution for provider keys; runs
on ubuntu-latest (no Fly runner, no Docker — the suite only touches the
LLM backends). Pins LIVE_LLM_ANTHROPIC_45_PLUS_MODELS=claude-sonnet-4-5
since the Anthropic family has no default models and would otherwise
silently collect empty.
Opt-in by design: live model behavior is variable, so this is a signal,
not a required check.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: run live LLM tests on main pushes touching the transport
Mirrors unified-tests' push trigger, scoped to paths that can affect
the live suite (src/llm/, config, the tests, deps, and the workflow
itself) so provider API calls aren't spent on unrelated changes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: disable auth in live LLM test environment
The staging dotenv sets AUTH_USE_AUTH=true without a usable JWT secret,
and src/config.py validates the pair at import time — the same reason
unified-tests overrides it. This suite never runs the API server.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): fix gpt-5.4 reasoning_effort and gemini replay-turn flake
- test_live_openai: gpt-5.4 dropped 'minimal' from the reasoning_effort
vocabulary, so the gpt5 caching test 400'd — and the OpenAI backend's
BadRequestError terminal swallowed it into an empty CompletionResult.
Pick the effort per model generation.
- test_live_tools_structured_output: use tool_choice='auto' on the
replay turn, matching the production dialectic loop (which never
forces 'none') — NONE mode is what provoked gemini-2.5-flash's empty
candidates. Drop the temperature pin so retries actually resample,
and treat a repeat tool call as a retryable attempt.
Verified live: full suite green, gemini 4/4 consecutive passes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: fail live LLM run when no staging secret was loaded
If the latest-tag fetch fails and no second tag exists, the fallback
step is skipped rather than failed, and the job would proceed without
provider keys — every test then skips via require_provider_key and the
run goes green. Guard on both fetch outcomes so that path fails loudly.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(live-llm-tests-GHA): remove extra comments
* feat(structured-output): enable non-recursive schema references
* docs(structured-outputs): clean up new doc
* test(structured-output): fix caching refs memory leak, add tests
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): add device code oauth login for honcho servers
* fix(cli): harden device-auth input validation and transport errors
Reject zero/negative auth-method choices instead of letting Python
negative indexing wrap to the tail of the options list, and wrap httpx
transport failures in the OAuth POST helpers as OAuthFlowError so
connection errors surface through existing caller handling rather than
escaping as an uncaught traceback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: add HONCHO_CONFIG_DIR env var for cli
* refactor(cli): address review nits on device-auth PR
- split `init` into manual-key and interactive helpers
- document that access_valid checks persisted expiry, not the token
- rename single-letter local in redacted()
- cover 500 alongside 404 in supports_device_login metadata probe
- add config edge-case tests: stale apiKey drop, garbage/string
accessExpiresAt, empty-env-var popping, refresh-rotation fallback,
missing-token access_valid
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cli): stop deleting shared apiKey on device login; bind oauth grant to its host
apiKey is shared with sibling tools that read it from the same config
file, so device login must not remove it. Precedence flips to a live
OAuth token over apiKey; a dead grant now degrades to the saved key
with a warning instead of aborting. The oauth block records the host
it was minted against and is ignored (no use, no refresh) when
base_url points elsewhere, so a staging grant is never sent to prod.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* CI/CD Workflow file Added
* Updated Tag & SA Key
* update input Tag
* fix: pass workflow inputs to shell via env to prevent command injection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Rajat Ahuja <rahuja445@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(llm): support combined tool calling and structured output across backends
- OpenAI: parse() 500s on non-strict function tools; route tool-carrying
structured requests through create() with an explicit json_schema
response_format (mirrors the streaming path)
- Anthropic: skip the '{' JSON prefill when tools are present so tool_use
blocks stay reachable; make the schema instruction conditional and rely
on parse + repair
- Gemini: native response_schema + function calling is rejected before
Gemini 3; with tools present, inject a schema instruction into the final
turn instead and rely on parse + repair
- All backends: tool-call turns carry no consumable content, so skip
structured-output parsing on them
Extracted from the dialectic structured-output branch (DEV-1652) so the
transport layer can land independently.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): exercise combined tools + structured output per provider
Two-turn live flow per backend: a forced tool-call turn (structured
parsing must be skipped) followed by a replay turn that must return a
schema-conforming answer with tools still attached. Asserts the
provider-specific request shaping: no parse() for OpenAI (500s on
non-strict tools), no '{' prefill for Anthropic, no native
response_schema for Gemini.
Verified against live OpenAI (gpt-4.1, gpt-5, gpt-5.4, gpt-5.4-mini)
and Gemini (gemini-2.5-flash).
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: some needed unrelated test failures
* ci: add label-triggered live LLM test workflow
Adding the run-live-llm label to a PR (or workflow_dispatch) runs
tests/live_llm/ against real provider APIs — the only place the
--live-llm suite runs in CI. Reuses the unified-tests environment and
its Secrets Manager staging-dotenv resolution for provider keys; runs
on ubuntu-latest (no Fly runner, no Docker — the suite only touches the
LLM backends). Pins LIVE_LLM_ANTHROPIC_45_PLUS_MODELS=claude-sonnet-4-5
since the Anthropic family has no default models and would otherwise
silently collect empty.
Opt-in by design: live model behavior is variable, so this is a signal,
not a required check.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: run live LLM tests on main pushes touching the transport
Mirrors unified-tests' push trigger, scoped to paths that can affect
the live suite (src/llm/, config, the tests, deps, and the workflow
itself) so provider API calls aren't spent on unrelated changes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: disable auth in live LLM test environment
The staging dotenv sets AUTH_USE_AUTH=true without a usable JWT secret,
and src/config.py validates the pair at import time — the same reason
unified-tests overrides it. This suite never runs the API server.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): fix gpt-5.4 reasoning_effort and gemini replay-turn flake
- test_live_openai: gpt-5.4 dropped 'minimal' from the reasoning_effort
vocabulary, so the gpt5 caching test 400'd — and the OpenAI backend's
BadRequestError terminal swallowed it into an empty CompletionResult.
Pick the effort per model generation.
- test_live_tools_structured_output: use tool_choice='auto' on the
replay turn, matching the production dialectic loop (which never
forces 'none') — NONE mode is what provoked gemini-2.5-flash's empty
candidates. Drop the temperature pin so retries actually resample,
and treat a repeat tool call as a retryable attempt.
Verified live: full suite green, gemini 4/4 consecutive passes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: fail live LLM run when no staging secret was loaded
If the latest-tag fetch fails and no second tag exists, the fallback
step is skipped rather than failed, and the job would proceed without
provider keys — every test then skips via require_provider_key and the
run goes green. Guard on both fetch outcomes so that path fails loudly.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(live-llm-tests-GHA): remove extra comments
* ci(CODEOWNERS): introduce CODEOWNERS and gate GHA heavy test runs behind being a CODEOWNER
* ci(GHA-live-LLM-tests): consolidate common GHA steps
* test(test_live_openai): fix reasoning level adjustment for gpt-5
* test(live-llm-tests): temporary removal of gate to test the workflow
* test(live-llm-tests): revert removal of gate
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Split `REPRESENTATION_BATCH_MAX_TOKENS` into a "minimum work unit" setting on the producer side and a "maximum LLM tokens" setting on the consumer side
* feat(telemetry): CloudEvents + Langfuse tracing as projections over a captured LLM stream
Capture each LLM call once (CapturedLLMCall) and fan it out to multiple
exporters -- "one data model, two projections": a CloudEvents trace stream
(llm.call.traced / trace.content) and a Langfuse projection, both reconstructing
trace -> run -> step -> generation from the same source of truth.
- Capture seam (src/llm/capture.py): one canonicalization + content-addressed
hashing point, with an O(N) per-span memo so repeated context isn't re-hashed.
- Session correlation threaded telemetry -> captured call -> exporters,
namespaced only at the Langfuse export boundary.
- Span identity consolidated onto LLMTelemetryContext; dropped TRACE_ENDPOINT.
- Canonical generation/step names; dreamer branches nest under one dream trace;
tool calls become spans under their step.
- LANGFUSE_EXPORTER_MODE toggle ("exporter" default; "inline" kept one release
for side-by-side validation), centralized into computed settings predicates.
- Per-run/per-trace dedup registries (trace_session, langfuse_session) bounded
by an LRU so dedup and span grouping survive long-running workers.
- Embedding-call tracing; deterministic high-volume event sampling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(telemetry): address trace-review findings (span/step_seq collisions, test, logging)
- Dreamer specialists mint a distinct span_id per execution (trace_id stays the
shared dream run_id), so their CloudEvents trace resource ids no longer collide
between deduction and induction.
- Tool-loop no-tool early-return streams the tail with the next ordinal
(iteration+2) instead of reusing the in-loop call's step_seq, avoiding a
colliding trace resource id; mirrors the synthesis path.
- Tighten test_clips_oversized_string to assert output stays within TRACE_MAX_BYTES.
- emit_trace logs the swallowed exception with exc_info for debuggability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(telemetry): silence exporter-mode Langfuse warning + drop summarizer run_id placeholder
Two CloudEvents/Langfuse correctness fixes, independent of the trace viewer.
Langfuse exporter-mode gating: annotate_current_generation_io (and its two
executor.py call-site guards) were gated on LANGFUSE_PUBLIC_KEY instead of
langfuse_inline_enabled. In the default `exporter` mode they called
get_client().update_current_generation() with no active @observe span, logging
"No active span in current context" (~14 per dialectic run) and building
throwaway model_dump payloads on every LLM call. The LangfuseExporter projects
I/O from the captured stream, so these helpers must no-op in exporter mode.
Gated all three on langfuse_inline_enabled; added a regression test; fixed a
stale conditional_observe docstring.
Summarizer run_id placeholder: AgentToolSummaryCreatedEvent hardcoded
run_id="deriver"/iteration=0 because summarization is a single LLM call, not an
agentic run. That placeholder pollutes run_id grouping in the CloudEvents stream
(any consumer that groups by run_id sees a phantom "deriver" run). Made
run_id/iteration optional (None) and re-keyed get_resource_id on
message_id:summary_type (the real per-summary identity; run_id/iteration can no
longer identify it); bumped schema_version 2->3. Xatu ingestion stores only the
CloudEvent envelope, so the field/resource_id/version changes are transparent to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: update docstrings to be less verbose
* fix(telemetry): address PR review on captured-stream tracing
- embedding traces get a fresh span_id under parent_span_id=run_id, so
sibling embeddings in one run no longer share a span/idempotency key
- capture the provider finish_reason from stream chunks instead of
hardcoding "stop" on a successful drain
- gate the Langfuse exporter behind TELEMETRY.ENABLED (master switch) so
disabling telemetry sends no traces at all
- rename _emit_derived_content -> _emit_hashed_content
- inline the _emit_trace wrapper; drop unused trace_session.end_run
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: rename TELEMETRY_TRACE_PAYLOADS to TELEMETRY_TRACE_PAYLOADS_ENABLED
* fix(telemetry): capture provider tool calls in trace stream
The captured trace stream dropped assistant tool calls for openai/gemini:
build_captured_messages only read {role, content, tool_call_id}, but those
providers keep tool calls outside content (openai's tool_calls, gemini's
parts), so replayed tool-call turns landed as empty content and gemini lost
its text and tool results entirely. Anthropic (tool_use in content) was fine.
Normalize each input message per provider into a unified tool_calls
[{id, name, input}] field on CapturedMessage/TraceContentEvent, recovering
gemini text/results along the way, and fold tool_calls into
compute_content_hash so empty-content openai turns no longer collide in the
dedup store. langfuse_exporter._input now surfaces the calls.
Also fix a silent serialization drop: gemini thought_signature is bytes, so
model_dump(mode="json") on the traced event raised UnicodeDecodeError and
emit_trace swallowed it -- dropping the whole tool-calling iteration from the
trace stream (billing and Langfuse were unaffected). base64-encode the
signature on the telemetry path; replay keeps the raw bytes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(telemetry): type replay tool-call dict for bytes signature
thought_signature widened to str | bytes | None, but
_tool_call_result_to_dict's literal was inferred as
dict[str, str | dict[str, Any]], so the bytes assignment failed project-wide
basedpyright (the per-file pre-commit hook didn't catch it). Annotate the
dict as dict[str, Any]; the replay path keeps the raw bytes unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: remove 3 tests
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(conclusions): expose reasoning level + allow filtering by level
The `level` of a conclusion (explicit / deductive / inductive /
contradiction) was filterable server-side but stripped from the
`Conclusion` response and not surfaced in either SDK. This adds it
end-to-end so callers can list explicit-only ("not dreamed on")
conclusions without dropping to raw HTTP.
- api: add `level` to the Conclusion response schema
- python sdk: `ConclusionLevel` type, `level` on Conclusion/response,
`level=` kwarg on ConclusionScope.list() and the async variant
- ts sdk: `ConclusionLevel` type, `level` on Conclusion/response,
`level` option on list()
- tests: assert level is exposed; add level-filter list test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(conclusions): use generic filters= on list() instead of level= kwarg
Match the documented SDK convention (peers/sessions/messages all take a
generic `filters` dict passed through to the same dynamic server-side
filter logic) instead of a one-off `level=` kwarg. `level` filtering now
works as `list(filters={"level": "explicit"})` alongside any other
supported filter/operator.
The `level` field on the Conclusion response (added in the previous
commit) is kept — it's still not otherwise returned by the API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(conclusions): allow filtering by level on query() in py + ts SDKs
The branch's level-filter work exposed `filters=` on `list()` but left
`query()` (semantic search) hardcoding `{observer, observed}`, so callers
could filter the list endpoint by reasoning level but not semantic search —
asymmetric in both SDKs.
- Python: add keyword-only `filters` to `ConclusionScope.query` and
`ConclusionScopeAio.query`, merged over the scope's observer/observed.
- TypeScript: add optional `filters` arg to `ConclusionScope.query`,
mirroring the existing `list()` change.
The server `/conclusions/query` endpoint already honors filters in the body
(verified against production), so this is purely SDK surface parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(filters): document filtering conclusions by reasoning level
The using-filters page covered workspaces/peers/sessions/messages but not
conclusions. Add a "Filtering Conclusions" section showing level-based
filtering on both list() and query(), including the common "explicit only"
(exclude dream-derived) case and the in[deductive,inductive] inverse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(conclusions): simplify filter merge to a single dict spread
Replace the merged_filters + if-block pattern in list()/query() (py sync,
aio, ts) with a single dict spread that layers the caller's filters over the
scope's observer/observed (and session). No behavior change — same merge
order (caller wins) — just less code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(conclusions): reject scope-managed keys in SDK conclusion filters
The generic filters= argument on ConclusionScope.list()/query() spread
user-supplied filters last, so a stray observer/observed/session key
silently overrode the scope and returned data from a different peer
pair. Add a fail-loud guard in both the Python and TypeScript SDKs that
rejects scope-managed filter keys with a clear error, directing callers
to peer.conclusions / conclusions_of(target) and the session= parameter.
session_id remains a valid filter on query() (which has no dedicated
session parameter).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* feat: add exact content deduplication in document creation
* feat: add comment for index
* fix: harden times_derived logic across all callers to use max of inputs and existing + 1
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
The OpenAI backend passed tool_choice through raw while the Anthropic and
Gemini backends translate Honcho's canonical vocabulary to their native
form. On a mixed-provider fallback chain (e.g. Gemini primary -> OpenAI
backup), a canonical "any" reached OpenAI unchanged and was rejected as an
invalid param, since OpenAI only accepts none/auto/required.
Add a _convert_tool_choice to the OpenAI backend mirroring the others so a
single TOOL_CHOICE value resolves correctly regardless of which provider a
fallback lands on. "any"/"required" -> "required", auto/none pass through,
a tool-name string or {"name": ...} dict -> a function selection.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(llm): stop capturing live LLM clients in Langfuse generation spans
honcho_llm_call_inner is the @observe generation boundary, and default
auto-capture serialized every argument into the span input -- including
client_override (a live AsyncOpenAI/genai client) and selected_config
(which carries api_key). Auto-capture deep-copies the client into a
half-constructed object whose teardown raises:
- AsyncHttpxClientWrapper ... no attribute '_state' (OpenAI, stderr flood)
- BaseApiClient ... no attribute '_http_options' (Gemini, HONCHO-4HA)
and it leaked ModelConfig.api_key into traces.
Switch from auto-capture (denylist) to explicit annotation (allowlist):
disable capture_input/capture_output on the decorator and stamp curated,
serializable input (messages) and output (HonchoLLMCallResponse) via the
new annotate_current_generation_io helper. Full trace fidelity is
preserved; no client object or secret can reach a trace.
Fixes HONCHO-4HA
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(llm): track call tuning knobs as Langfuse model_parameters
Restore full trace fidelity after disabling @observe auto-capture: surface
every tuning knob (temperature, max_tokens, tools, reasoning effort, ...) on
the generation via model_parameters, sourced from the resolved effective
config instead of the raw function args.
Use a deny-list, not an allow-list: dump the whole ModelConfig and exclude
only secret-bearing fields (api_key, base_url, fallback, provider_params), so
new config knobs are traced automatically without keeping a hand-written list
in sync. The live client is never passed -- there is no useful trace
representation of it and serializing it is what triggered HONCHO-4HA.
Adds a deny-list test proving secrets never leak even when the config carries
a real api_key/base_url/provider_params (the production override-client path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(llm): duplicate token usage to Langfuse + skip payload build when disabled
Mirror per-call token usage (input, output, prompt-cache read/creation) onto
the Langfuse generation via usage_details, so Langfuse renders native tokens
and cost in addition to the CloudEvents accounting.
Also guard both generation-annotation blocks behind settings.LANGFUSE_PUBLIC_KEY
so the model_dump-backed model_parameters payload (and the usage dict) are only
built when Langfuse is actually configured (addresses CodeRabbit: the annotate
helper no-ops when disabled, but the payload was still being constructed every
call).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: use git tags to fetch secrets for unified test
* fix: test failure
* fix: override AUTH_USE_AUTH and SENTRY_ENABLED
* fix: upload traces
* chore: rm run on PR
* fix: rm bucket from logs
ajspig
Alexei Vedernikov
Aakash Kattelu
Rajat Ahuja
Vineeth Voruganti
Leonardo Baray
papesy384
ajspig
Eugene Eisenstein
Ulysse Pence
Ulysse Pence
Talha Abdur Rahman
Chris Caldwell
Aru Sharma