From b0f0295fd1d6a46e460ed924d8a72dc9ae3cb16a Mon Sep 17 00:00:00 2001 From: Xiangzhe <32761048+xz-dev@users.noreply.github.com> Date: Tue, 19 May 2026 22:29:08 +0800 Subject: [PATCH 1/7] fix(docker): gate deriver startup on api healthcheck (#689) --- docker-compose.yml.example | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docker-compose.yml.example b/docker-compose.yml.example index c52699f0..51b279e7 100644 --- a/docker-compose.yml.example +++ b/docker-compose.yml.example @@ -21,6 +21,18 @@ services: condition: service_healthy ports: - "127.0.0.1:8000:8000" + healthcheck: + test: + [ + "CMD", + "/app/.venv/bin/python", + "-c", + "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).read()", + ] + interval: 5s + timeout: 5s + retries: 5 + start_period: 10s # -- Development: mount source for live reload -- # volumes: # - .:/app @@ -40,6 +52,8 @@ services: dockerfile: Dockerfile entrypoint: ["/app/.venv/bin/python", "-m", "src.deriver"] depends_on: + api: + condition: service_healthy database: condition: service_healthy redis: From b5f24a6ac55cb500f3c3e803feed338c795e07f7 Mon Sep 17 00:00:00 2001 From: Rajat Ahuja Date: Wed, 20 May 2026 18:25:30 -0400 Subject: [PATCH 2/7] feat: add new cloudevents for api routes (#637) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add new cloudevents for api routes * fix: add total input tokens to RepresentationCompletedEvent * feat(telemetry): inject honcho_version + emitter health metrics * feat(telemetry): per-LLM-call event with try/finally emission + sampler Adds LLMCallCompletedEvent (llm.call.completed) — fires once per provider hit with full cost-attribution context: transport/provider_label, model, token counts with cache breakdown, finish_reason, outcome (success or error), is_final_attempt flag, retry/fallback state, duration, tool-call shape, streaming flag, and agent correlation (run_id + iteration). - src/telemetry/events/llm.py: new event class + CallPurpose closed enum (deriver.representation, dialectic.answer, dream.deduction|induction, summary.short|long). Resource id includes attempt so multi-attempt retries in one iteration get distinct deterministic ids. - src/telemetry/events/base.py: BaseEvent._volume_class ClassVar (default "ground_truth"); the new event opts into "high_volume". - src/config.py: TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE (default 1.0). - src/telemetry/emitter.py: deterministic sampler keyed on run_id (so an entire agent trace is kept or dropped together). Aggregate envelopes bypass the sampler. Sampled-out events increment the dedicated counter separate from buffer_full/send_failed drops. - src/llm/runtime.py: AttemptPlan gains attempt/retry_attempts/is_fallback so the executor reads retry state without re-deriving it. - src/llm/types.py: LLMTelemetryContext dataclass carrying workspace, call_purpose, run_id, iteration, peer fields. Iteration is mutable so the tool loop can set it per inner call. - src/llm/executor.py: honcho_llm_call_inner wraps the backend call in try/finally — emits on success AND on exception, with is_final_attempt computed from AttemptPlan. Stream path emits a was_stream=True placeholder (token totals deferred until streaming completion is wired through). Telemetry failures swallowed. - src/llm/api.py: threads telemetry kwarg through all 4 signatures into both honcho_llm_call_inner and execute_tool_loop. - src/llm/tool_loop.py: _telemetry_for_iteration helper copies the caller context with iteration set per call — covers both the normal iteration loop AND the max-iteration synthesis call (iteration N+1). Tests cover success/error emission, sampler trace-coherence (same run_id → same decision), volume_class enforcement, unknown call_purpose tolerance, provider_label inference, and telemetry failure isolation. 378/378 pass. * feat(telemetry): emit agent.iteration on every LLM response + synthesis AgentIterationEvent was defined but never emitted on this branch. Phase 2 wires it up in execute_tool_loop so every LLM call inside an agentic loop produces one event — including the no-tool terminating iteration and the max-iteration synthesis call — and threads LLMTelemetryContext from dialectic and dreamer specialists down through honcho_llm_call. - src/telemetry/events/agent.py: AgentIterationEvent opts into _volume_class="high_volume" so the Phase 1 sampler throttles it. - src/llm/tool_loop.py: _emit_agent_iteration() helper fires once per honcho_llm_call_inner response, BEFORE the no-tool early return so the terminating iteration is counted. A second emission fires for the max-iteration synthesis call BEFORE final_response is mutated with cumulative totals (otherwise the per-iteration counts would double-count). Emission is defensively skipped when telemetry context lacks run_id / agent_type / parent_category / workspace_name; emit failures are swallowed. - src/dreamer/specialists.py: BaseSpecialist.run passes LLMTelemetryContext with parent_category="dream", agent_type=self.name, observer/observed, call_purpose=f"dream.{self.name}". - src/dialectic/core.py: _telemetry_context() builds a shared context for both answer() and answer_stream(), using self._run_id (always set) + workspace + observed peer. Tests cover fresh-copy semantics, per-iteration vs terminating emission, defensive skip cases, telemetry-failure isolation, and volume_class. 408/408 pass across telemetry + llm + utils + dreamer + dialectic. * feat(telemetry): agent.tool.call.completed event + ToolResult metadata Adds the missing generic per-tool-call event so read-only tools (search_*, get_recent_history, get_observation_context, etc.) and the four existing state-change tools all produce a telemetry record. Built on a new internal ToolResult(content, metadata) contract so handlers can surface search-specific fields (top_k/used_embedding/query_tokens/results_count) to Phase 3 and create/delete counts to Phase 5's specialist rollups. - src/telemetry/events/agent.py: AgentToolCallCompletedEvent at v1 with _volume_class="high_volume". Resource id = {run_id}:{iteration}:{tool_call_seq} so two calls to the same tool in one iteration don't collide deterministic ids and get dedup-dropped downstream. - src/utils/types.py: ToolResult dataclass; two new ContextVars (_current_tool_call_seq + _last_tool_metadata) so tool_loop and the execute_tool closure can communicate per-call telemetry without changing the public Callable[[str, dict], Any] signature. - src/utils/agent_tools.py: execute_tool times handlers, unwraps ToolResult, publishes metadata, emits the event. Handlers updated to ToolResult where useful: create/delete observations, update_peer_card, search_memory, search_messages. Other handlers continue to return str. - src/llm/tool_loop.py: set_current_tool_call_seq before each executor call; read get_last_tool_metadata after and stash on all_tool_calls[i] for Phase 5 rollups. Tests cover ToolResult str-likeness, ContextVar round-trip, full-context emission with search metadata, resource-id disambiguation, defensive skip cases, telemetry isolation, truncation metadata, volume_class. 420/420 pass. * feat(telemetry): RepresentationCompletedEvent v2 token breakdown + tool-less truncation Bulks out the deriver's per-batch telemetry without bumping the event schema version. New additive fields capture the full token breakdown (queued vs. extra-context vs. scaffold), the cap configuration (batch_max_tokens, max_input_tokens, was_flush_enabled), real cap-hit flags, and observer fanout. `input_tokens` stays unchanged as the queued-message-tokens billing key Xatu's Stripe meter reads. The big enabler: src/llm/api.py now actually enforces max_input_tokens on the tool-less LLM path. Before this, the deriver passed the kwarg but the path silently dropped it — so the configured cap was advisory and hit_input_token_cap couldn't be measured. Phase 4 wires truncation through the same truncate_messages_to_fit helper the tool loop uses and surfaces input_was_truncated on HonchoLLMCallResponse. - src/telemetry/events/representation.py: 12 additive fields, schema_version stays at 2. - src/llm/types.py: input_was_truncated on HonchoLLMCallResponse. - src/llm/api.py: tool-less path truncates messages before dispatch, flips input_was_truncated on the response when clamping occurs. Split into Literal[True]/Literal[False] branches for typecheck. - src/deriver/queue_manager.py: QueueBatchResult dataclass replaces the 3-tuple return from get_queue_item_batch; carries hit_batch_token_cap (computed from cumulative token sum vs cap), was_flush_enabled snapshot, and batch_max_tokens. Worker loop unpacks + forwards. - src/deriver/consumer.py: process_representation_batch gains the three flag kwargs and forwards. - src/deriver/deriver.py: derives the breakdown fields locally, populates the new fields on emit, sources hit_input_token_cap from response.input_was_truncated. Tests cover schema stability, defaultable fields, input_tokens semantic preservation, cap-hit flag round-trip, model_dump completeness, and HonchoLLMCallResponse.input_was_truncated mutability. Existing test_queue_processing.py tests updated for QueueBatchResult and mock process_representation_batch signature. 479/479 pass. * feat(telemetry): DreamRunEvent v2 scheduler reasons + DreamSpecialistEvent v2 rollups Bumps both dream events to v2 with additive fields. DreamRunEvent gains scheduler context (threshold_reason / delay_reason / documents_since_last_dream_at_schedule / document_threshold / dream_type / enabled_types_count) threaded through the dream queue payload — the two scheduler gates stay as separate fields rather than collapsing into one trigger_reason, preserving the WHY-vs-WHEN semantics. DreamSpecialistEvent gains denormalized rollups (created_observation_count / deleted_observation_count / peer_card_updated / search_tool_calls_count) sourced from Phase 3's ToolResult.metadata so the counts reflect observation truth, not call truth. - src/telemetry/events/dream.py: schema_version → 2 for both events; new fields all defaultable so older producers still construct valid events. - src/utils/queue_payload.py: DreamPayload + create_dream_payload accept threshold_reason / delay_reason / documents_since_last_dream_at_schedule / document_threshold. - src/dreamer/dream_scheduler.py: check_and_schedule_dream computes the two reasons at decision time and threads them through schedule_dream → _delayed_dream → execute_dream → enqueue_dream. - src/deriver/enqueue.py: create_dream_record / enqueue_dream gain the kwargs and persist on the queue payload. - src/dreamer/orchestrator.py: process_dream unpacks the payload; run_dream accepts the kwargs and stamps them on DreamRunEvent. - src/dreamer/specialists.py: BaseSpecialist.run walks response.tool_calls_made and sums ToolResult.metadata.created_count / .deleted_count, sets peer_card_updated, counts search-tool calls by name. Tests cover schema_version bumps, defaultable Phase 5 fields, threshold-vs-delay semantics, observation-vs-call-count rollup distinction, and DreamPayload round-trip. Existing tests updated for the schema bump and the new enqueue_dream kwargs. 488/488 pass. * feat(telemetry): AgentToolSummaryCreatedEvent v2 token breakdown Bumps schema_version to 2 and adds three additive breakdown fields so analytics can answer "how much of a summary call's cost was the previous- summary rollup vs. the new messages vs. the scaffold instructions". - src/telemetry/events/agent.py: previous_summary_tokens, message_tokens, prompt_scaffold_tokens added with sensible 0 defaults. input_tokens retains its current semantic (provider-side LLM tokens) — the plan's proposed `provider_input_tokens` was omitted because input_tokens already serves that purpose and a duplicate would fork queries. - src/utils/summarizer.py: emit now populates the three new fields from values already in scope (messages_tokens, previous_summary_tokens, prompt_tokens). Hoisted prompt_tokens calculation out of the is_fallback conditional so both the save-summary path and the emit share one binding — basedpyright couldn't prove the sibling-scope binding was safe, and the compute is cheap + idempotent. Tests cover schema bump, defaultable fields, input_tokens semantic preservation, first-summary edge case, and breakdown round-trip. 493/493 pass. * feat(telemetry): embedding.call.completed event + call-purpose ContextVar Adds the final piece of cost-attribution telemetry: per-embedding-call events covering every provider hit (single + batch + retry attempts). Embedding calls are real provider spend that was invisible before this phase; search-heavy paths (dialectic agentic) can produce more embedding calls than LLM calls, so the new event participates in the shared HIGH_VOLUME_SAMPLE_RATE. - src/telemetry/events/llm.py: EmbeddingCallCompletedEvent at v1 with _volume_class="high_volume". EmbeddingCallPurpose closed enum (search_memory / search_messages / create_observations / vector_sync / summary / message_create). Resource id = run:purpose:provider:model:input_count so per-iteration calls in one agentic run don't collide. - src/utils/types.py: _embedding_call_purpose ContextVar plus @contextmanager wrapper. Nesting-safe via ContextVar.reset(token). Callers wrap embedding-driving operations in `with embedding_call_purpose("search_memory"): ...` — no changes to the embedding client signature. - src/embedding_client.py: _emit_embedding_call wraps each provider hit with try/finally so success AND error paths emit. Errors propagate unchanged. Each retry attempt of _process_batch emits its own event. Unknown call_purpose slugs drop to None (validation against the enum happens at emit time, not at context-manager-set time). - src/utils/agent_tools.py: search_memory / search_messages / search_messages_temporal / create_observations (batch + fallback) all tag their embedding calls. - src/crud/representation.py: save_representation tags with CREATE_OBSERVATIONS; get_working_representation precompute tags with SEARCH_MEMORY. - src/crud/message.py: create_messages batch embed tags with MESSAGE_CREATE; search_messages/temporal fallback tags with SEARCH_MESSAGES. Tests cover event shape, enum closure, ContextVar nesting/exception cleanup, wrapper success+error emission, unknown-purpose graceful fallback, telemetry-failure isolation. 550/550 pass across the full telemetry+llm+utils+dreamer+dialectic+deriver+crud test set. * chore: fix tests * fix(telemetry): address review findings on stream events, context propagation, and cap detection Five findings from a post-Phase-7 review (one resolved by the merge from main, four addressed here): - src/llm/executor.py: stream-path LLMCallCompletedEvent now fires AFTER the stream is set up and drained (or on exception), with real duration and accurate outcome. Previously the event was emitted before execute_stream() ran and was always recorded as outcome="success" with duration_ms=0, which silently masked stream-setup and stream-drain failures. Wrapping the async generator in try/finally surfaces the real outcome; token counts stay 0 because we still don't have them at stream end (aggregate envelopes carry totals). - src/deriver/deriver.py + src/utils/summarizer.py: deriver and summarizer LLM calls now thread LLMTelemetryContext into honcho_llm_call. Before this, the closed CallPurpose enum had DERIVER_REPRESENTATION / SUMMARY_SHORT / SUMMARY_LONG slugs but those production call sites didn't actually pass `telemetry=`, so their LLMCallCompletedEvents lost workspace_name, parent_category, and call_purpose. summarizer threads workspace_name through _create_and_save_summary → _create_summary → create_short_summary / create_long_summary. - src/utils/types.py + src/embedding_client.py: embedding_call_purpose ctx manager now accepts workspace_name and run_id kwargs, backed by two new ContextVars. EmbeddingCallCompletedEvent's publisher reads both via get_embedding_workspace_name / get_embedding_run_id so embedding events carry workspace and run correlation. All call sites updated: search_memory / search_messages / search_messages_temporal / _handle_create_observations_impl pass ctx.workspace_name + ctx.run_id; create_observations standalone and create_messages pass workspace_name; RepresentationManager.save_representation and get_working_representation pass self.workspace_name. - src/deriver/queue_manager.py: hit_batch_token_cap detection rewritten. Previously summed kept-rows' token_count and checked against batch_max_tokens, but the SQL filter `cumulative_token_count <= cap` guarantees kept rows stay under the cap, so the flag almost never fired. Now uses two follow-up queries: total token_count across the included id range + EXISTS check for any session message past the last-kept id. Both true → cap was actually binding. (The fifth finding — deriver scaffold-token computation needing estimate_deriver_prompt_tokens(custom_instructions) — was resolved by the merge from main; the Phase 4 emit at src/deriver/deriver.py:283 already sources prompt_scaffold_tokens from the wrapped helper.) 567/567 telemetry+llm+utils+dreamer+dialectic+deriver+crud tests pass. ruff + basedpyright clean. * chore: ruff linting * chore: clean AI generated comments references specs * fix: address coderabbit changes * fix(telemetry): address remaining PR review findings Six findings from the PR 637 telemetry review batched into one commit. - src/llm/executor.py + src/embedding_client.py: asyncio.CancelledError now surfaces as outcome="cancelled" on both stream and sync paths, distinct from "error". Client disconnects mid-stream and server shutdowns are normal control flow and should not feed error-rate alerting. LLMCallCompletedEvent and EmbeddingCallCompletedEvent outcome Literal extended; docstrings + tests cover the new state. - src/utils/types.py + src/llm/tool_loop.py: new iteration_scope() context manager captures and resets the four per-tool-loop ContextVars (_current_iteration, _current_tool_call_seq, _current_provider_tool_call_id, _last_tool_metadata). Applied as a typed decorator to execute_tool_loop so back-to-back loops in the same asyncio Task (worker batches, tests) don't observe stale state. - src/telemetry/events/api.py + src/routers/messages.py: MessageCreatedEvent schema v1 → v2. Added required last_message_id (nanoid public_id of the trailing message); get_resource_id now keys on it instead of message_count, eliminating the collision case where two same-size batches in the same session+source produced identical event ids. message_count stays on the body for analytics. - src/deriver/queue_manager.py: hit_batch_token_cap now computed from the FINAL post-config-filter batch. Previously the flag used the pre-filter messages_context[-1].id, which produced false positives when _resolve_batch_configuration trimmed the trailing queue item — telemetry reported a cap-hit when the actual returned batch was short for unrelated reasons. Cap-detection block moved inside the async with after the filter; no extra DB connection. - src/config.py + src/telemetry/emitter.py: documented the HIGH_VOLUME_SAMPLE_RATE orphan trade-off (rate<1.0 keeps aggregates but drops children, so JOIN ON run_id queries see partial traces). Behavior unchanged — rate defaults to 1.0. - src/deriver/deriver.py: WARNING-level invariant logs when response.input_tokens < messages_tokens (provider tokenization drift) or prompt_scaffold_tokens <= 0 (estimator silent failure). Best-effort — telemetry never bleeds into the deriver path * fix(telemetry): stream retry, embed attempts, truncation, dedup Address remaining audit findings on the cloudevents PR: - Stream setup now runs inside the awaited honcho_llm_call_inner so tenacity's retry wrapper in stream_final_response catches transient setup failures (rate-limit, auth, network). Previously the returned generator deferred execute_stream until first iteration — outside the retry wrapper — crashing the request and bypassing telemetry. - Embedding _emit_embedding_call gains an is_final_attempt parameter; _process_batch threads the real retry index so dashboards stop conflating one-shot, mid-retry, and exhausted-retry calls. - _truncate_tool_output returns (text, original_chars, was_truncated) and a new _maybe_truncated_result helper wraps in ToolResult when truncation happens. Five handlers migrated. AgentToolCallCompletedEvent fields was_truncated and result_chars_before_truncation are now populated instead of always None/False. - execute_tool_loop tracks any_iteration_truncated and stamps input_was_truncated on the final response (both HonchoLLMCallResponse and StreamingResponseWithMetadata). Dialectic now reports hit_input_token_cap correctly. - GetContextEvent.get_resource_id uses empty-string sentinel instead of literal "none" so a peer named "none" can't collide with absent. - generate_event_id folds honcho_version into the deterministic id so same logical event from different deploys produces distinct ids. * fix(telemetry): address audit findings across LLM/embed/event paths Three rounds of telemetry audit findings, grouped by area: Retry correctness - Stream LLM setup now runs inside the awaited honcho_llm_call_inner so tenacity's outer retry catches setup failures (Fix 1). Previously the inner generator deferred execute_stream past the retry wrapper. - stream_final_response bumps the per-retry attempt index via dataclasses.replace so emitted events show [1, 2, 3] instead of [1, 1, 1] (Fix 13). - Embedding _emit_embedding_call takes is_final_attempt; _process_batch threads the real retry index (Fix 2). Token + cost reporting - HonchoLLMCallResponse.hit_input_token_cap (renamed from input_was_truncated) uses a token-based rule so single-message over-cap inputs are correctly flagged — the deriver's prompt-only path used to silently fly through. Propagated through tool_loop's per-iteration check (Fix 4) and into RepresentationCompletedEvent. - DialecticCompletedEvent gains hit_input_token_cap; output_tokens now folds in the final-stream's cumulative usage via StreamingResponseWithMetadata.__aiter__ (Fix 7). Event emission completeness - AgentToolCallCompletedEvent's was_truncated / result_chars_before_truncation populated by _truncate_tool_output via a new _maybe_truncated_result wrapper; 5 handlers migrated (Fix 3). - DreamSpecialistEvent emits on failure with success=False + new error_class field, via try/finally (Fix 11). - DeletionCompletedEvent emits on failure paths via try/finally (Fix 12). - CleanupStaleItemsCompletedEvent.queue_items_cleaned populated from deleted_count (Fix 8). Embedding call attribution (Fix 9) - embedding_call_purpose context manager accepts parent_category. - 4 new EmbeddingCallPurpose enum values: DIALECTIC_PREFETCH, SESSION_CONTEXT_SEARCH, PREFERENCE_EXTRACTION, GENERIC_DOCUMENT_SEARCH. - Wrapped previously-unattributed sites: dialectic prefetch, session context search, preference extraction, conclusions search, vector sync (×2). Deterministic event ID + dedup - generate_event_id folds honcho_version into the hash so cross-deploy events don't silently collide on ID (Fix 6). - GetContextEvent resource_id uses empty-string sentinel instead of "none" so a peer literally named "none" can't collide (Fix 5). Queue batch cap detection (P2.1) - hit_batch_token_cap keys on the pre-config-filter SQL boundary so the "kept=900 of 1000 cap, next=300 excluded by cap" case reports True while still avoiding the config-filter false positive. Tool result metadata - search_messages_temporal returns ToolResult with the same search_meta shape as search_memory / search_messages (P2.3) — top_k, used_embedding, embedding_query_count, query_tokens, results_count. Tests: stream-setup retry, stream-retry attempt sequence, post-stream output_tokens write-back, is_final_attempt matrix, truncation E2E, tool-loop hit_input_token_cap propagation, honcho_version in event id, GetContextEvent disambiguation, queue_items_cleaned round-trip. * fix(telemetry): address audit findings across LLM/embed/event paths Four rounds of telemetry audit findings (initial + 3 follow-ups), grouped by area: Retry correctness - Stream LLM setup now runs inside the awaited honcho_llm_call_inner so tenacity's outer retry catches setup failures (Fix 1). The inner generator previously deferred execute_stream past the retry wrapper. - stream_final_response bumps the per-retry attempt index via dataclasses.replace so emitted events show [1, 2, 3] instead of [1, 1, 1] (Fix 13). - Embedding _emit_embedding_call takes is_final_attempt; _process_batch threads the real retry index (Fix 2). Token + cost reporting - HonchoLLMCallResponse.hit_input_token_cap (renamed from input_was_truncated) uses a token-based rule so single-message over-cap inputs are correctly flagged — the deriver's prompt-only path used to silently fly through. Propagated through tool_loop's per-iteration check (Fix 4) and into RepresentationCompletedEvent. - DialecticCompletedEvent gains hit_input_token_cap; output_tokens now folds in the final-stream's cumulative usage via StreamingResponseWithMetadata.__aiter__ (Fix 7). Queue batch cap detection - hit_batch_token_cap previously required total_in_range >= cap, which produced false negatives whenever the kept range didn't fully exhaust the budget. Replaced with a pre-config-filter SQL boundary check (P2.1), then further refined to a queue-item boundary comparison (Fix 14) so trailing-context trimming doesn't false-negative either. Event emission completeness - AgentToolCallCompletedEvent's was_truncated / result_chars_before_truncation now populated by _truncate_tool_output via _maybe_truncated_result; 5 handlers migrated (Fix 3). - DreamSpecialistEvent emits on failure with success=False + new error_class field, via try/finally (Fix 11). except BaseException catches cancellations too (Fix 16). - DeletionCompletedEvent emits on failure paths via try/finally (Fix 12), and uses ValidationException for unsupported types per project guideline (Fix 17). - CleanupStaleItemsCompletedEvent.queue_items_cleaned populated from deleted_count (Fix 8). Embedding call attribution (Fix 9) - embedding_call_purpose accepts parent_category. - 4 new EmbeddingCallPurpose values: DIALECTIC_PREFETCH, SESSION_CONTEXT_SEARCH, PREFERENCE_EXTRACTION, GENERIC_DOCUMENT_SEARCH. - Wrapped previously-unattributed sites: dialectic prefetch, session context search, preference extraction, conclusions search, vector sync (×2). Reconciler no longer holds DB session during embedding (Fix 15) - _sync_documents and _sync_message_embeddings refactored into three phases per CLAUDE.md guideline: fetch+detach in a small DB scope, external embedding call without DB locks, writes in a fresh short-lived DB scope. New _apply_*_sync helpers; orchestrators expunge ORM objects before invoking. Vector store upsert + sync_state updates stay in the apply phase together. Deterministic event ID + dedup - generate_event_id folds honcho_version into the hash so cross-deploy events don't silently collide on ID (Fix 6). - GetContextEvent resource_id uses empty-string sentinel instead of "none" so a peer literally named "none" can't collide (Fix 5). Tool result metadata - search_messages_temporal returns ToolResult with the same search_meta shape as search_memory / search_messages (P2.3). - Dialectic.prefetched_conclusion_count uses Representation.len() so inductive + contradiction observations count too (Fix 10). * fix(telemetry): orchestrator emit + review feedback Three more rounds of audit findings + inline PR review, grouped: Orchestration / emit reliability - run_dream wrapped in try/finally so DreamRunEvent always emits, even on unexpected exceptions including CancelledError (`finally` still runs while cancellation propagates). Specialist except clauses broadened from SpecialistExecutionError (never raised in src/) to Exception so provider/DB/tool failures are recorded with deduction_success=False / induction_success=False instead of crashing past the emit. - BaseSpecialist.run() telemetry state initialization + try/finally hoisted above the preflight phase (peer lookup, peer-card preload, create_tool_executor, get_model_config, prompt construction) so preflight failures emit DreamSpecialistEvent(success=False) instead of being dropped on the floor. - Reverted the Round-4 _sync_documents / _sync_message_embeddings phase split. The split introduced a race: rows were released from FOR UPDATE SKIP LOCKED before the embed call, allowing two workers to claim and clobber the same batch. Long-held DB transaction restored (pre-existing CLAUDE.md violation accepted as a deliberate trade-off; proper fix requires a claim/in_flight migration tracked separately). Schema + naming (PR-internal — none of these have shipped) - threshold_reason → trigger_reason on DreamRunEvent, DreamPayload, and every emit/scheduler/router/test call site (~45 src + 21 test lines). Name now accurately reflects the field's role across "manual", "surprisal", and "document_threshold" values. - MessageCreatedEvent reset to schema v1 (was internally bumped to v2 for last_message_id but never shipped at v1 — downstream sees it for the first time at merge). - DreamSpecialistEvent gains created_counts_by_level / deleted_counts_by_level: dict[str, int] keyed on the closed level taxonomy. Per-tool-call events use list[str] (≤10 items), but specialist runs aggregate 20+ — dict keeps emissions compact. - QueueBatchResult marked frozen=True. Per-call embedding attribution - Agent tool embedding_call_purpose wraps for search_memory, search_messages, search_messages_temporal, create_observations now driven embedding cost rolls up under the right workflow. - create_observations() signature gains parent_category kwarg (mirrors existing run_id pattern). Manual dream scheduling - Manual /schedule_dream route now passes trigger_reason="manual" and delay_reason="immediate". Previously both arrived as null in DreamRunEvent, breaking analytics joins. Queue-batch SQL perf - next_exists_check folded into the main CTE query via bool_or(cumulative_token_count > batch_max_tokens) OVER () in a nested subquery. Cap detection is now one roundtrip per batch instead of two. Code/doc cleanup - representation.py docstring uses generic "downstream metering key" language (was "Xatu's Stripe meter"). bench runner --base-url help uses a generic example host (was "groudon.fly.dev"). Public-facing code/docs shouldn't reference internal service names. Tests added for: orchestrator failure-path DreamRunEvent emission, specialists preflight try/finally coverage, manual-dream trigger_reason/delay_reason round-trip, dict-rollup accumulation across multiple tool calls in a specialist run, CTE-fold one-roundtrip behavior. Full Python suite passes (1236). * fix(telemetry): correctness + attribution + emitter robustness - Dreamer iteration count: read response.iterations directly so one-shot runs no longer report iterations=0 and tool-using runs include the terminal/synthesis LLM call. - RepresentationCompletedEvent.observer_count counts successful saves, not attempts. - search_memory empty-memory fallback reports the snippet count when message context is returned (was always 0). - Wire parent_category through every embedding emit path: message create (api), save_representation (representation), per-observation fallback (caller-supplied), and the peer/session context routes (api). get_working_representation accepts parent_category and embedding_purpose so the internal fallback embed lands in the same analytics bucket as the route-level precompute even when the precompute is suppressed. - BatchItem carries token_count so _process_batch reuses chunk-prep counts instead of re-encoding every chunk for the telemetry proxy. - Drop vestigial EmbeddingCallCompletedEvent.batch_size (always == input_count). - Emitter: release the lock during HTTP send so a failing endpoint's retry+backoff (~36s worst case) doesn't block other flushers; edge-trigger the 80%-capacity warning so sustained backpressure doesn't flood logs; defer event_id generation past the high-volume sampler for events with run_id so sampled-out children don't pay the sha256; harden emit() against sync callers with no running loop; track threshold-flush tasks so shutdown() drains in-flight sends before closing the HTTP client. * fix(telemetry): tool cancellation emit, nanoid run_ids, version unification - execute_tool: wrap post-work in finally so AgentToolCallCompletedEvent fires on CancelledError; explicit handler sets is_error/result_str before re-raising. - run_id: replace str(uuid.uuid4())[:8] with generate_nanoid() across dialectic/dreamer/specialists; matches project-wide nanoid convention. - Bump _schema_version on events touched by run_id widening: DialecticCompletedEvent v1→v2 (also covers hit_input_token_cap field), AgentIterationEvent v1→v2, AgentToolConclusionsCreatedEvent v1→v2, AgentToolConclusionsDeletedEvent v2→v3, AgentToolPeerCardUpdatedEvent v1→v2. - Unify honcho_version: single HONCHO_VERSION constant in src/_version.py read from pyproject.toml (importlib.metadata fallback). Drop TELEMETRY.HONCHO_VERSION setting. Use the constant for the FastAPI app version (no more hardcoded "3.0.6") and for emitter body injection. - Delete 17 tautological per-event test_schema_version methods; the parametrized contract test still enforces version >= 1 across all events. --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> --- src/_version.py | 33 + src/config.py | 15 + src/crud/message.py | 46 +- src/crud/representation.py | 45 +- src/deriver/consumer.py | 191 ++--- src/deriver/deriver.py | 59 +- src/deriver/enqueue.py | 20 + src/deriver/prompts.py | 8 +- src/deriver/queue_manager.py | 183 ++++- src/dialectic/core.py | 53 +- src/dreamer/dream_scheduler.py | 46 +- src/dreamer/orchestrator.py | 220 +++--- src/dreamer/specialists.py | 388 ++++++---- src/embedding_client.py | 264 +++++-- src/llm/api.py | 112 ++- src/llm/executor.py | 236 +++++- src/llm/runtime.py | 6 + src/llm/tool_loop.py | 195 ++++- src/llm/types.py | 74 +- src/main.py | 3 +- src/reconciler/sync_vectors.py | 14 +- src/routers/conclusions.py | 25 +- src/routers/messages.py | 41 ++ src/routers/peers.py | 53 +- src/routers/sessions.py | 62 +- src/routers/workspaces.py | 7 + src/telemetry/emitter.py | 212 +++++- src/telemetry/events/__init__.py | 27 + src/telemetry/events/agent.py | 146 +++- src/telemetry/events/api.py | 167 +++++ src/telemetry/events/base.py | 32 +- src/telemetry/events/dialectic.py | 14 +- src/telemetry/events/dream.py | 91 ++- src/telemetry/events/llm.py | 264 +++++++ src/telemetry/events/representation.py | 74 +- src/telemetry/prometheus/metrics.py | 58 ++ src/utils/agent_tools.py | 482 ++++++++++-- src/utils/queue_payload.py | 17 + src/utils/search.py | 9 +- src/utils/summarizer.py | 49 +- src/utils/types.py | 203 +++++- src/vector_store/turbopuffer.py | 25 +- tests/bench/molecular.py | 18 +- tests/bench/oolong_common.py | 20 +- tests/bench/runner_common.py | 2 +- tests/crud/test_representation_manager.py | 36 +- tests/deriver/test_deriver_processing.py | 106 +++ tests/deriver/test_queue_processing.py | 235 +++++- tests/dreamer/test_dream_v2_rollups.py | 183 +++++ tests/dreamer/test_dreamer_integration.py | 1 + tests/integration/test_telemetry.py | 1 + tests/llm/test_telemetry_agent_iteration.py | 295 ++++++++ tests/llm/test_telemetry_agent_tool_call.py | 355 +++++++++ tests/llm/test_telemetry_llm_call.py | 686 ++++++++++++++++++ tests/llm/test_tool_loop_truncation.py | 197 +++++ tests/scripts/test_configure_embeddings.py | 2 +- tests/startup/test_embedding_validator.py | 6 +- tests/telemetry/conftest.py | 126 +++- tests/telemetry/test_embedding_call_event.py | 333 +++++++++ tests/telemetry/test_emit_function.py | 1 + tests/telemetry/test_emitter.py | 57 ++ tests/telemetry/test_events.py | 371 ++++++++-- .../test_representation_v2_fields.py | 194 +++++ tests/telemetry/test_summary_v2_fields.py | 122 ++++ tests/test_models_vector_dim.py | 2 +- tests/utils/test_agent_tools.py | 25 +- 66 files changed, 6938 insertions(+), 705 deletions(-) create mode 100644 src/_version.py create mode 100644 src/telemetry/events/api.py create mode 100644 src/telemetry/events/llm.py create mode 100644 tests/dreamer/test_dream_v2_rollups.py create mode 100644 tests/llm/test_telemetry_agent_iteration.py create mode 100644 tests/llm/test_telemetry_agent_tool_call.py create mode 100644 tests/llm/test_telemetry_llm_call.py create mode 100644 tests/llm/test_tool_loop_truncation.py create mode 100644 tests/telemetry/test_embedding_call_event.py create mode 100644 tests/telemetry/test_representation_v2_fields.py create mode 100644 tests/telemetry/test_summary_v2_fields.py diff --git a/src/_version.py b/src/_version.py new file mode 100644 index 00000000..5cd84569 --- /dev/null +++ b/src/_version.py @@ -0,0 +1,33 @@ +"""Single source of truth for the Honcho service version. + +Reads pyproject.toml directly so the value never drifts from the authoritative +source. Falls back to installed package metadata for wheel-only deploys where +pyproject.toml may not be shipped. + +Used by: +- src/main.py for the FastAPI app `version=...` +- src/telemetry/emitter.py and src/telemetry/events/base.py for event tagging +""" + +from __future__ import annotations + +from importlib.metadata import PackageNotFoundError +from importlib.metadata import version as _pkg_version +from pathlib import Path + +import tomllib + + +def _read_version() -> str: + pyproject = Path(__file__).resolve().parent.parent / "pyproject.toml" + try: + with pyproject.open("rb") as f: + return tomllib.load(f)["project"]["version"] + except (OSError, KeyError, tomllib.TOMLDecodeError): + try: + return _pkg_version("honcho") + except PackageNotFoundError: + return "unknown" + + +HONCHO_VERSION: str = _read_version() diff --git a/src/config.py b/src/config.py index 189676dc..5ae93fa4 100644 --- a/src/config.py +++ b/src/config.py @@ -1071,6 +1071,21 @@ class TelemetrySettings(HonchoSettings): # Namespace for instance identification (propagated from top-level NAMESPACE if not set) NAMESPACE: str | None = None + # Sample rate for high-volume events: llm.call.completed, embedding.call.completed, + # agent.iteration, agent.tool.call.completed. Deterministic on run_id so traces + # remain coherent end-to-end. Aggregate envelopes (RepresentationCompleted, + # DialecticCompleted, DreamRun, etc.) are NEVER sampled — they're calibration + # ground truth. + # + # Design trade-off: at rate < 1.0, aggregate events still emit but their + # high-volume children get dropped. Downstream `JOIN ... ON run_id` queries + # will see parents without complete children — this is intentional (the + # aggregates carry totals; detail events are best-effort), but consumers + # MUST NOT rebuild per-call analytics from the sampled children alone or + # they'll undercount. If you tune this below 1.0, audit dashboards/queries + # that join high-volume events to aggregate envelopes first. + HIGH_VOLUME_SAMPLE_RATE: Annotated[float, Field(default=1.0, ge=0.0, le=1.0)] = 1.0 + class CacheSettings(HonchoSettings): model_config = SettingsConfigDict(env_prefix="CACHE_", extra="ignore") # pyright: ignore diff --git a/src/crud/message.py b/src/crud/message.py index da81fdb3..ec673bb0 100644 --- a/src/crud/message.py +++ b/src/crud/message.py @@ -12,8 +12,10 @@ from src.config import settings from src.dependencies import tracked_db from src.embedding_client import embedding_client from src.exceptions import VectorStoreError +from src.telemetry.events import EmbeddingCallPurpose from src.utils.filter import apply_filter from src.utils.formatting import ILIKE_ESCAPE_CHAR, escape_ilike_pattern +from src.utils.types import embedding_call_purpose from src.vector_store import VectorRecord, get_external_vector_store from .session import get_or_create_session @@ -283,11 +285,17 @@ async def create_messages( for message in message_objects if message.content and message.content.strip() } - embedding_dict = ( - await embedding_client.batch_embed(id_resource_dict) - if id_resource_dict - else {} - ) + if id_resource_dict: + with embedding_call_purpose( + EmbeddingCallPurpose.MESSAGE_CREATE.value, + workspace_name=workspace_name, + parent_category="api", + ): + embedding_dict = await embedding_client.batch_embed( + id_resource_dict + ) + else: + embedding_dict = {} external_vector_store = get_external_vector_store() @@ -891,9 +899,17 @@ async def search_messages( Each snippet may contain multiple matches if they were close together. Context messages are ordered chronologically and include the matched messages. """ - query_embedding = ( - embedding if embedding is not None else await embedding_client.embed(query) - ) + if embedding is not None: + query_embedding = embedding + else: + # Caller didn't precompute; tag this fallback path as search_messages. + # Callers that have a more specific intent should set their own + # context manager before calling and pass the precomputed embedding. + with embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MESSAGES.value, + workspace_name=workspace_name, + ): + query_embedding = await embedding_client.embed(query) return await _semantic_search_messages( workspace_name, session_name, @@ -1083,9 +1099,17 @@ async def search_messages_temporal( List of tuples: (matched_messages, context_messages) Each snippet may contain multiple matches if they were close together. """ - query_embedding = ( - embedding if embedding is not None else await embedding_client.embed(query) - ) + if embedding is not None: + query_embedding = embedding + else: + # Caller didn't precompute; tag this fallback path as search_messages. + # Callers that have a more specific intent should set their own + # context manager before calling and pass the precomputed embedding. + with embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MESSAGES.value, + workspace_name=workspace_name, + ): + query_embedding = await embedding_client.embed(query) return await _semantic_search_messages( workspace_name, session_name, diff --git a/src/crud/representation.py b/src/crud/representation.py index 9b689d06..bb969120 100644 --- a/src/crud/representation.py +++ b/src/crud/representation.py @@ -15,6 +15,7 @@ from src.dependencies import tracked_db from src.dreamer.dream_scheduler import check_and_schedule_dream from src.embedding_client import embedding_client from src.schemas import ResolvedConfiguration +from src.telemetry.events import EmbeddingCallPurpose from src.telemetry.logging import accumulate_metric from src.utils.formatting import format_datetime_utc from src.utils.representation import ( @@ -22,6 +23,7 @@ from src.utils.representation import ( ExplicitObservation, Representation, ) +from src.utils.types import embedding_call_purpose logger = logging.getLogger(__name__) @@ -96,7 +98,14 @@ class RepresentationManager: observation_texts = [_observation_text(obs) for obs in all_observations] try: - embeddings = await embedding_client.simple_batch_embed(observation_texts) + with embedding_call_purpose( + EmbeddingCallPurpose.CREATE_OBSERVATIONS.value, + workspace_name=self.workspace_name, + parent_category="representation", + ): + embeddings = await embedding_client.simple_batch_embed( + observation_texts + ) except ValueError as e: raise exceptions.ValidationException( "Observation content exceeds maximum token limit of " @@ -210,6 +219,8 @@ class RepresentationManager: semantic_search_max_distance: float | None = None, include_most_derived: bool = False, max_observations: int = settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS, + parent_category: str | None = None, + embedding_purpose: EmbeddingCallPurpose = EmbeddingCallPurpose.SEARCH_MEMORY, ) -> Representation: """ Get working representation with flexible query options. @@ -224,13 +235,32 @@ class RepresentationManager: semantic_search_max_distance: Maximum distance for semantic search include_most_derived: Include most derived observations max_observations: Maximum total observations to return + parent_category: Optional workflow attribution forwarded to the + fallback embedding call when the caller didn't pre-compute + an embedding (or pre-compute failed). + embedding_purpose: Embedding call_purpose tag to use on the + fallback embed when no pre-computed embedding was supplied. + Defaults to SEARCH_MEMORY; callers whose route-level + precompute uses a more specific purpose (e.g. + SESSION_CONTEXT_SEARCH) should pass that here so the + fallback path lands in the same analytics bucket. Returns: Representation combining various query strategies """ if include_semantic_query and embedding is None: - with suppress(Exception): - # Best-effort precompute + # Best-effort precompute when caller didn't supply one (or their + # precompute was suppressed). The purpose is parameterized so + # this fallback shows up in the same telemetry bucket as the + # successful path — see embedding_purpose docstring above. + with ( + suppress(Exception), + embedding_call_purpose( + embedding_purpose.value, + workspace_name=self.workspace_name, + parent_category=parent_category, + ), + ): embedding = await embedding_client.embed(include_semantic_query) if db is not None: @@ -496,6 +526,8 @@ async def get_working_representation( semantic_search_max_distance: float | None = None, include_most_derived: bool = False, max_observations: int = settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS, + parent_category: str | None = None, + embedding_purpose: EmbeddingCallPurpose = EmbeddingCallPurpose.SEARCH_MEMORY, ) -> Representation: """ Get raw working representation data from the relevant document collection. @@ -507,6 +539,11 @@ async def get_working_representation( db: Optional database session. If provided, uses it directly; otherwise creates a new session via tracked_db. embedding: Pre-computed embedding for the semantic query. + parent_category: Workflow attribution forwarded to the fallback + embedding call when no pre-computed embedding was supplied. + embedding_purpose: Embedding call_purpose for the fallback embed; + callers should match it to whatever purpose their route-level + precompute used so failure/retry paths stay in the same bucket. """ manager = RepresentationManager( workspace_name=workspace_name, @@ -522,4 +559,6 @@ async def get_working_representation( semantic_search_max_distance=semantic_search_max_distance, include_most_derived=include_most_derived, max_observations=max_observations, + parent_category=parent_category, + embedding_purpose=embedding_purpose, ) diff --git a/src/deriver/consumer.py b/src/deriver/consumer.py index d4fd2a04..8bbd9359 100644 --- a/src/deriver/consumer.py +++ b/src/deriver/consumer.py @@ -9,7 +9,7 @@ from src import crud, models from src.dependencies import tracked_db from src.deriver.deriver import process_representation_tasks_batch from src.dreamer import process_dream -from src.exceptions import ResourceNotFoundException +from src.exceptions import ResourceNotFoundException, ValidationException from src.models import Message from src.reconciler.queue_cleanup import cleanup_queue_items from src.reconciler.sync_vectors import run_vector_reconciliation_cycle @@ -158,6 +158,9 @@ async def process_representation_batch( observers: list[str] | None, observed: str | None, queue_item_message_ids: list[int], + hit_batch_token_cap: bool = False, + was_flush_enabled: bool = False, + batch_max_tokens: int = 0, ) -> None: """ Prepares and processes a batch of messages for representation tasks. @@ -168,6 +171,9 @@ async def process_representation_batch( observers: List of observers for the messages observed: The observed of the messages queue_item_message_ids: Message IDs from queue items + hit_batch_token_cap: whether the queue batcher clamped this batch to fit + was_flush_enabled: snapshot of DERIVER.FLUSH_ENABLED at fetch time + batch_max_tokens: DERIVER.REPRESENTATION_BATCH_MAX_TOKENS snapshot """ if not messages or not messages[0]: logger.debug("process_representation_batch received no messages") @@ -182,6 +188,9 @@ async def process_representation_batch( observers=observers, observed=observed, queue_item_message_ids=queue_item_message_ids, + hit_batch_token_cap=hit_batch_token_cap, + was_flush_enabled=was_flush_enabled, + batch_max_tokens=batch_max_tokens, ) @@ -218,92 +227,107 @@ async def process_deletion( workspace_name, ) - async with tracked_db("process_deletion") as db: - if deletion_type == "session": - try: - result = await crud.delete_session( - db, workspace_name=workspace_name, session_name=resource_id - ) - messages_deleted = result.messages_deleted - conclusions_deleted = result.conclusions_deleted - logger.info( - "Successfully deleted session %s in workspace %s " - + "(messages=%d, conclusions=%d)", - resource_id, - workspace_name, - messages_deleted, - conclusions_deleted, - ) - except ResourceNotFoundException as e: - # Session not found - may have already been deleted, treat as success - logger.warning( - "Session %s not found during deletion (may already be deleted): %s", - resource_id, - str(e), - ) + # try/except/finally so the event ALWAYS fires — both success and + # failure (unsupported type, unexpected CRUD error). Previously the + # unsupported-type branch raised before the emit ran, and unexpected + # CRUD errors bubbled up without telemetry. + try: + async with tracked_db("process_deletion") as db: + if deletion_type == "session": + try: + result = await crud.delete_session( + db, workspace_name=workspace_name, session_name=resource_id + ) + messages_deleted = result.messages_deleted + conclusions_deleted = result.conclusions_deleted + logger.info( + "Successfully deleted session %s in workspace %s " + + "(messages=%d, conclusions=%d)", + resource_id, + workspace_name, + messages_deleted, + conclusions_deleted, + ) + except ResourceNotFoundException as e: + # Session not found - may have already been deleted, treat as success + logger.warning( + "Session %s not found during deletion (may already be deleted): %s", + resource_id, + str(e), + ) - elif deletion_type == "observation": - try: - await crud.delete_document_by_id( - db, workspace_name=workspace_name, document_id=resource_id - ) - conclusions_deleted = 1 # Single observation deleted - logger.info( - "Successfully deleted observation %s in workspace %s", - resource_id, - workspace_name, - ) - except ResourceNotFoundException as e: - # Document not found - may have already been deleted, treat as success - logger.warning( - "Observation %s not found during deletion (may already be deleted): %s", - resource_id, - str(e), - ) + elif deletion_type == "observation": + try: + await crud.delete_document_by_id( + db, workspace_name=workspace_name, document_id=resource_id + ) + conclusions_deleted = 1 # Single observation deleted + logger.info( + "Successfully deleted observation %s in workspace %s", + resource_id, + workspace_name, + ) + except ResourceNotFoundException as e: + # Document not found - may have already been deleted, treat as success + logger.warning( + "Observation %s not found during deletion (may already be deleted): %s", + resource_id, + str(e), + ) - elif deletion_type == "workspace": - try: - result = await crud.delete_workspace(db, workspace_name=workspace_name) - peers_deleted = result.peers_deleted - sessions_deleted = result.sessions_deleted - messages_deleted = result.messages_deleted - conclusions_deleted = result.conclusions_deleted - logger.info( - "Successfully deleted workspace %s " - + "(peers=%d, sessions=%d, messages=%d, conclusions=%d)", - workspace_name, - peers_deleted, - sessions_deleted, - messages_deleted, - conclusions_deleted, - ) - except ResourceNotFoundException as e: - # Workspace not found - may have already been deleted, treat as success - logger.warning( - "Workspace %s not found during deletion (may already be deleted): %s", - workspace_name, - str(e), - ) + elif deletion_type == "workspace": + try: + result = await crud.delete_workspace( + db, workspace_name=workspace_name + ) + peers_deleted = result.peers_deleted + sessions_deleted = result.sessions_deleted + messages_deleted = result.messages_deleted + conclusions_deleted = result.conclusions_deleted + logger.info( + "Successfully deleted workspace %s " + + "(peers=%d, sessions=%d, messages=%d, conclusions=%d)", + workspace_name, + peers_deleted, + sessions_deleted, + messages_deleted, + conclusions_deleted, + ) + except ResourceNotFoundException as e: + # Workspace not found - may have already been deleted, treat as success + logger.warning( + "Workspace %s not found during deletion (may already be deleted): %s", + workspace_name, + str(e), + ) - else: - success = False - error_message = f"Unsupported deletion type: {deletion_type}" - raise ValueError(error_message) - - # Emit telemetry event - emit( - DeletionCompletedEvent( - workspace_name=workspace_name, - deletion_type=deletion_type, - resource_id=resource_id, - success=success, - peers_deleted=peers_deleted, - sessions_deleted=sessions_deleted, - messages_deleted=messages_deleted, - conclusions_deleted=conclusions_deleted, - error_message=error_message, + else: + success = False + error_message = f"Unsupported deletion type: {deletion_type}" + raise ValidationException(error_message) + except Exception as e: + # Catch anything that survived the per-branch `ResourceNotFoundException` + # handling above (incl. the ValueError from the unsupported-type branch). + # Record telemetry, then re-raise so the queue worker still surfaces + # the failure to its caller. + success = False + if error_message is None: + error_message = f"{type(e).__name__}: {e}" + raise + finally: + emit( + DeletionCompletedEvent( + workspace_name=workspace_name, + deletion_type=deletion_type, + resource_id=resource_id, + success=success, + peers_deleted=peers_deleted, + sessions_deleted=sessions_deleted, + messages_deleted=messages_deleted, + conclusions_deleted=conclusions_deleted, + error_message=error_message, + ) ) - ) async def process_reconciler(payload: ReconcilerPayload) -> None: @@ -363,6 +387,7 @@ async def process_reconciler(payload: ReconcilerPayload) -> None: # Emit telemetry event for cleanup stale items emit( CleanupStaleItemsCompletedEvent( + queue_items_cleaned=deleted_count, total_duration_ms=duration_ms, ) ) diff --git a/src/deriver/deriver.py b/src/deriver/deriver.py index 72cac3d1..303a6ec2 100644 --- a/src/deriver/deriver.py +++ b/src/deriver/deriver.py @@ -6,10 +6,12 @@ from src.config import ConfiguredModelSettings, settings from src.crud.representation import RepresentationManager from src.dependencies import tracked_db from src.llm import honcho_llm_call +from src.llm.types import LLMTelemetryContext from src.models import Message from src.schemas import ResolvedConfiguration from src.telemetry import prometheus_metrics from src.telemetry.events import RepresentationCompletedEvent, emit +from src.telemetry.events.llm import CallPurpose from src.telemetry.logging import accumulate_metric, log_performance_metrics from src.telemetry.prometheus.metrics import ( DeriverComponents, @@ -39,6 +41,9 @@ async def process_representation_tasks_batch( observers: list[str], observed: str, queue_item_message_ids: list[int], + hit_batch_token_cap: bool = False, + was_flush_enabled: bool = False, + batch_max_tokens: int = 0, ) -> None: """ Process messages with minimal overhead - single LLM call, save to multiple collections. @@ -49,6 +54,9 @@ async def process_representation_tasks_batch( observers: List of observer peer IDs (collections to save to). observed: The observed peer ID. queue_item_message_ids: Message IDs from queue items being processed + hit_batch_token_cap: queue batcher clamped this batch to fit + was_flush_enabled: DERIVER.FLUSH_ENABLED snapshot at batch time + batch_max_tokens: DERIVER.REPRESENTATION_BATCH_MAX_TOKENS snapshot """ if not messages: return @@ -146,6 +154,12 @@ async def process_representation_tasks_batch( enable_retry=True, retry_attempts=3, trace_name="minimal_deriver", + telemetry=LLMTelemetryContext( + workspace_name=latest_message.workspace_name, + call_purpose=CallPurpose.DERIVER_REPRESENTATION.value, + parent_category="representation", + observed=observed, + ), ) llm_duration = (time.perf_counter() - llm_start) * 1000 @@ -175,6 +189,7 @@ async def process_representation_tasks_batch( latest_message.created_at, ) + successful_observer_count = 0 if observations.is_empty() or not message_ids: logger.warning( "Deriver generated zero observations for messages %s:%s in %s/%s!", @@ -200,6 +215,7 @@ async def process_representation_tasks_batch( latest_message.created_at, message_level_configuration, ) + successful_observer_count += 1 except Exception as e: logger.error( "Failed to save representation for observer %s: %s", observer, e @@ -234,12 +250,39 @@ async def process_representation_tasks_batch( accumulate_metric( f"minimal_deriver_{latest_message.id}_{observed}", "explicit_observations", - "\n".join(f" • {obs}" for obs in observations.explicit), + "\n".join(f" • {obs}" for obs in observations.explicit), "blob", ) log_performance_metrics("minimal_deriver", f"{latest_message.id}_{observed}") + # token-breakdown fields derived from messages + cap snapshots. + queued_message_count = len(queue_item_message_ids) + prompt_message_count = len(messages) + prompt_message_tokens = sum(msg.token_count for msg in messages) + extra_context_message_count = max(prompt_message_count - queued_message_count, 0) + extra_context_tokens = max(prompt_message_tokens - messages_tokens, 0) + + # Data-quality invariants. Best-effort — telemetry never bleeds into the + # deriver path — but log loudly when violated so analytics alerting catches + # silent estimator failures (provider tokenization drift, scaffold helper + # returning 0) at the source instead of as drift in BigQuery later. + if response.input_tokens < messages_tokens: + logger.warning( + "token-breakdown invariant violated: response.input_tokens (%d) < messages_tokens (%d) for observed=%s, latest=%s — provider tokenization drift or wrong messages_tokens computation?", + response.input_tokens, + messages_tokens, + observed, + latest_message.public_id, + ) + if prompt_tokens <= 0: + logger.warning( + "prompt_scaffold_tokens estimated as %d for observed=%s, latest=%s — estimate_deriver_prompt_tokens may have failed silently", + prompt_tokens, + observed, + latest_message.public_id, + ) + # Emit telemetry event emit( RepresentationCompletedEvent( @@ -255,6 +298,20 @@ async def process_representation_tasks_batch( llm_call_ms=llm_duration, total_duration_ms=overall_duration, input_tokens=messages_tokens, + total_input_tokens=response.input_tokens, output_tokens=response.output_tokens, + # additive fields + queued_message_count=queued_message_count, + prompt_message_count=prompt_message_count, + prompt_message_tokens=prompt_message_tokens, + extra_context_message_count=extra_context_message_count, + extra_context_tokens=extra_context_tokens, + prompt_scaffold_tokens=prompt_tokens, + batch_max_tokens=batch_max_tokens, + max_input_tokens=settings.DERIVER.MAX_INPUT_TOKENS, + was_flush_enabled=was_flush_enabled, + hit_batch_token_cap=hit_batch_token_cap, + hit_input_token_cap=response.hit_input_token_cap, + observer_count=successful_observer_count, ) ) diff --git a/src/deriver/enqueue.py b/src/deriver/enqueue.py index cbd032c2..aaeda415 100644 --- a/src/deriver/enqueue.py +++ b/src/deriver/enqueue.py @@ -399,6 +399,10 @@ def create_dream_record( observed: str, dream_type: schemas.DreamType, session_name: str | None = None, + trigger_reason: str | None = None, + delay_reason: str | None = None, + documents_since_last_dream_at_schedule: int | None = None, + document_threshold: int | None = None, ) -> dict[str, Any]: """ Create a queue record for a dream task. @@ -409,6 +413,10 @@ def create_dream_record( observed: Name of the observed peer dream_type: Type of dream to execute session_name: Name of the session to scope the dream to if specified + trigger_reason: what tripped the schedule + delay_reason: what governed when it fires + documents_since_last_dream_at_schedule: count snapshot at schedule time + document_threshold: DOCUMENT_THRESHOLD snapshot at schedule time Returns: Queue record dictionary with workspace_name and other fields @@ -418,6 +426,10 @@ def create_dream_record( observer=observer, observed=observed, session_name=session_name, + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule, + document_threshold=document_threshold, ) return { @@ -436,6 +448,10 @@ async def enqueue_dream( observed: str, dream_type: schemas.DreamType, session_name: str | None = None, + trigger_reason: str | None = None, + delay_reason: str | None = None, + documents_since_last_dream_at_schedule: int | None = None, + document_threshold: int | None = None, ) -> None: """ Enqueue a dream task for immediate processing by the deriver. @@ -461,6 +477,10 @@ async def enqueue_dream( observed=observed, dream_type=dream_type, session_name=session_name, + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule, + document_threshold=document_threshold, ) work_unit_key = dream_record["work_unit_key"] diff --git a/src/deriver/prompts.py b/src/deriver/prompts.py index 4be35b95..683834bf 100644 --- a/src/deriver/prompts.py +++ b/src/deriver/prompts.py @@ -22,7 +22,9 @@ def _normalized_custom_instructions(custom_instructions: str | None) -> str | No def _custom_instructions_section(custom_instructions: str | None) -> str: """Render optional custom instructions for the deriver prompt.""" - normalized_custom_instructions = _normalized_custom_instructions(custom_instructions) + normalized_custom_instructions = _normalized_custom_instructions( + custom_instructions + ) if normalized_custom_instructions is None: return "" @@ -93,7 +95,9 @@ def estimate_minimal_deriver_prompt_tokens() -> int: def estimate_deriver_prompt_tokens(custom_instructions: str | None) -> int: """Estimate minimal deriver prompt tokens, including custom instructions if present.""" - normalized_custom_instructions = _normalized_custom_instructions(custom_instructions) + normalized_custom_instructions = _normalized_custom_instructions( + custom_instructions + ) if normalized_custom_instructions is None: return estimate_minimal_deriver_prompt_tokens() diff --git a/src/deriver/queue_manager.py b/src/deriver/queue_manager.py index cda6decd..8c2b5850 100644 --- a/src/deriver/queue_manager.py +++ b/src/deriver/queue_manager.py @@ -2,6 +2,7 @@ import asyncio import signal from asyncio import Task from collections.abc import Sequence +from dataclasses import dataclass, field from datetime import datetime, timedelta, timezone from logging import getLogger from typing import Any, NamedTuple, cast @@ -56,6 +57,25 @@ class WorkerOwnership(NamedTuple): aqs_id: str # The ID of the ActiveQueueSession that the worker is processing +@dataclass(frozen=True) +class QueueBatchResult: + """Result of `QueueManager.get_queue_item_batch`. + + telemetry needs to know two things in addition to the batch + contents: whether the cumulative-token cap clamped the batch, and what + the configured cap was. These flags feed `RepresentationCompletedEvent` + so analytics can detect "we under-batched because of a flush" vs + "we hit the cap and kept going". + """ + + messages_context: list[models.Message] = field(default_factory=list) + items_to_process: list["QueueItem"] = field(default_factory=list) + configuration: ResolvedConfiguration | None = None + hit_batch_token_cap: bool = False + was_flush_enabled: bool = False + batch_max_tokens: int = 0 + + def _detach_queue_batch_objects( db: AsyncSession, messages_context: list[models.Message], @@ -464,13 +484,12 @@ class QueueManager: break try: if work_unit.task_type == "representation": - ( - messages_context, - items_to_process, - message_level_configuration, - ) = await self.get_queue_item_batch( + batch_result = await self.get_queue_item_batch( work_unit.task_type, work_unit_key, ownership.aqs_id ) + messages_context = batch_result.messages_context + items_to_process = batch_result.items_to_process + message_level_configuration = batch_result.configuration logger.debug( f"Worker {worker_id} retrieved {len(messages_context)} messages and {len(items_to_process)} queue items for work unit {work_unit_key} (AQS ID: {ownership.aqs_id})" ) @@ -503,6 +522,9 @@ class QueueManager: observers=observers, observed=work_unit.observed, queue_item_message_ids=queue_item_message_ids, + hit_batch_token_cap=batch_result.hit_batch_token_cap, + was_flush_enabled=batch_result.was_flush_enabled, + batch_max_tokens=batch_result.batch_max_tokens, ) await self.mark_queue_items_as_processed( items_to_process, work_unit_key @@ -636,13 +658,17 @@ class QueueManager: task_type: str, work_unit_key: str, aqs_id: str, - ) -> tuple[list[models.Message], list[QueueItem], ResolvedConfiguration | None]: + ) -> "QueueBatchResult": """ Batch processing for representation and agent tasks. - Returns a tuple of (messages_context, items_to_process, configuration). + + Returns a `QueueBatchResult` carrying: - messages_context: unique Message rows (conversation turns) forming the context window - items_to_process: QueueItems for the current work_unit_key within that window - configuration: Resolved configuration for the batch + - hit_batch_token_cap: True when the cumulative-token window clamped the batch + - was_flush_enabled: snapshot of `settings.DERIVER.FLUSH_ENABLED` at fetch time + - batch_max_tokens: snapshot of the cap actually applied to this batch """ if task_type != "representation": raise ValueError( @@ -650,6 +676,7 @@ class QueueManager: ) batch_max_tokens = settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS + was_flush_enabled = settings.DERIVER.FLUSH_ENABLED parsed_key = parse_work_unit_key(work_unit_key) messages_context: list[models.Message] = [] items_to_process: list[QueueItem] = [] @@ -663,7 +690,10 @@ class QueueManager: .where(models.ActiveQueueSession.id == aqs_id) ) if not ownership_check.scalar_one_or_none(): - return [], [], None + return QueueBatchResult( + was_flush_enabled=was_flush_enabled, + batch_max_tokens=batch_max_tokens, + ) # Step 2: Build a single SQL query that: # 1. Finds the earliest unprocessed message for this work_unit_key @@ -712,9 +742,16 @@ class QueueManager: preceding_message_id_subq, min_unprocessed_message_id_subq ) - # Build CTE with ALL messages starting from effective_start_id - # This includes the preceding context message (if any) and interleaving messages - cte = ( + # Build CTE in two nested selects so we can layer a second window + # function on top of `cumulative_token_count`. Postgres doesn't + # allow nesting window functions in a single select; we compute + # `cumulative_token_count` in `inner_cte`, then `cap_exceeded` as + # `bool_or(cumulative > cap) OVER ()` in the outer CTE. The flag + # is identical across every row, so reading it from any returned + # row tells us whether the SQL cap would have excluded messages — + # eliminating the separate `SELECT EXISTS` roundtrip that used to + # run post-fetch. + inner_cte = ( select( models.Message.id.label("message_id"), models.Message.token_count.label("token_count"), @@ -726,7 +763,20 @@ class QueueManager: .where(models.Message.session_name == parsed_key.session_name) .where(models.Message.workspace_name == parsed_key.workspace_name) .where(models.Message.id >= effective_start_id) - .order_by(models.Message.id) + .subquery() + ) + + cte = ( + select( + inner_cte.c.message_id, + inner_cte.c.token_count, + inner_cte.c.peer_name, + inner_cte.c.cumulative_token_count, + func.bool_or(inner_cte.c.cumulative_token_count > batch_max_tokens) + .over() + .label("cap_exceeded"), + ) + .order_by(inner_cte.c.message_id) .cte() ) @@ -738,7 +788,11 @@ class QueueManager: ) query = ( - select(models.Message, models.QueueItem) + select( + models.Message, + models.QueueItem, + cte.c.cap_exceeded.label("cap_exceeded"), + ) .select_from(cte) .join(models.Message, models.Message.id == cte.c.message_id) .outerjoin( @@ -756,31 +810,108 @@ class QueueManager: result = await db.execute(query) rows = result.all() if not rows: - return [], [], None + return QueueBatchResult( + was_flush_enabled=was_flush_enabled, + batch_max_tokens=batch_max_tokens, + ) + + # cap_exceeded is window-aggregated over the CTE — same value on + # every row. Read once from the first row; default False if the + # cap is disabled (`batch_max_tokens == 0`). + cap_exceeded_from_query: bool = ( + bool(rows[0][2]) if rows and batch_max_tokens > 0 else False + ) seen_messages: set[int] = set() - for m, qi in rows: + for m, qi, _cap in rows: if m.id not in seen_messages: messages_context.append(m) seen_messages.add(m.id) if qi is not None: items_to_process.append(qi) + # Detach BEFORE config-filter — `_resolve_batch_configuration` is + # sync and doesn't need the session; `messages_context` is a plain + # Python list after detach and survives the rest of this block. _detach_queue_batch_objects(db, messages_context, items_to_process) - items_to_process, resolved_config = _resolve_batch_configuration( - items_to_process - ) - - if items_to_process: - max_queue_item_message_id = max( - qi.message_id for qi in items_to_process if qi.message_id is not None + # The QUEUE-ITEM boundary (not the messages_context tail) is + # what matters for cap detection. messages_context includes + # non-queue interleaving context messages — if SQL kept some + # trailing context past the last queued item, the config + # filter trims that context but doesn't touch the queue + # items. Using messages_context[-1].id as a "did config + # filter shrink the batch" signal produced false negatives + # for that case. + last_queued_id_before: int | None = ( + max( + qi.message_id + for qi in items_to_process + if qi.message_id is not None + ) + if items_to_process + else None ) - messages_context = [ - m for m in messages_context if m.id <= max_queue_item_message_id - ] - return messages_context, items_to_process, resolved_config + items_to_process, resolved_config = _resolve_batch_configuration( + items_to_process + ) + if items_to_process: + max_queue_item_message_id = max( + qi.message_id + for qi in items_to_process + if qi.message_id is not None + ) + messages_context = [ + m for m in messages_context if m.id <= max_queue_item_message_id + ] + + last_queued_id_after: int | None = ( + max( + qi.message_id + for qi in items_to_process + if qi.message_id is not None + ) + if items_to_process + else None + ) + + # detect if `batch_max_tokens` clamped this returned batch. + # + # `cap_exceeded_from_query` comes from the CTE's + # `bool_or(cumulative > cap) OVER ()` column — true iff the + # SQL would have excluded at least one message because of the + # cap. Combined with the queue-boundary guard below, this + # tells us the cap was binding on the returned batch: + # + # 1. Config filter didn't shrink the QUEUE-ITEM boundary + # (`last_queued_id_before == last_queued_id_after`) — + # i.e. SQL chose the trailing queue item, not config; AND + # 2. The CTE detected at least one message past the cap. + # + # Both conditions must hold; otherwise the cap wasn't the + # constraint on this specific returned batch. + # + # Previously we issued a separate `SELECT EXISTS` query for + # the second condition. Folding it into the CTE eliminates the + # roundtrip — every batch fetch is now one query, not two. + if ( + batch_max_tokens > 0 + and last_queued_id_before is not None + and last_queued_id_before == last_queued_id_after + ): + hit_batch_token_cap = cap_exceeded_from_query + else: + hit_batch_token_cap = False + + return QueueBatchResult( + messages_context=messages_context, + items_to_process=items_to_process, + configuration=resolved_config, + hit_batch_token_cap=hit_batch_token_cap, + was_flush_enabled=was_flush_enabled, + batch_max_tokens=batch_max_tokens, + ) async def mark_queue_items_as_processed( self, items: list[QueueItem], work_unit_key: str diff --git a/src/dialectic/core.py b/src/dialectic/core.py index f8f3b841..5a5f690b 100644 --- a/src/dialectic/core.py +++ b/src/dialectic/core.py @@ -7,10 +7,11 @@ and synthesize responses to queries about a peer. import logging import time -import uuid from collections.abc import AsyncIterator, Callable from typing import Any, cast +from nanoid import generate as generate_nanoid + from src import crud from src.config import ConfiguredModelSettings, ReasoningLevel, settings from src.dependencies import tracked_db @@ -21,8 +22,9 @@ from src.llm import ( StreamingResponseWithMetadata, honcho_llm_call, ) +from src.llm.types import LLMTelemetryContext from src.telemetry import prometheus_metrics -from src.telemetry.events import DialecticCompletedEvent, emit +from src.telemetry.events import DialecticCompletedEvent, EmbeddingCallPurpose, emit from src.telemetry.logging import ( accumulate_metric, log_performance_metrics, @@ -36,6 +38,7 @@ from src.utils.agent_tools import ( search_memory, ) from src.utils.formatting import format_new_turn_with_timestamp +from src.utils.types import embedding_call_purpose logger = logging.getLogger(__name__) @@ -99,9 +102,7 @@ class DialecticAgent: ] self._session_history_initialized: bool = False self._prefetched_conclusion_count: int = 0 - self._run_id: str = str(uuid.uuid4())[ - :8 - ] # Always generate for event correlation + self._run_id: str = generate_nanoid() # Always generate for event correlation async def _initialize_session_history(self) -> None: """Fetch and inject session history into the system prompt if configured.""" @@ -173,7 +174,13 @@ class DialecticAgent: try: # Pre-compute embedding once for both searches (no DB needed) - query_embedding = await embedding_client.embed(query) + with embedding_call_purpose( + EmbeddingCallPurpose.DIALECTIC_PREFETCH.value, + workspace_name=self.workspace_name, + run_id=self._run_id, + parent_category="dialectic", + ): + query_embedding = await embedding_client.embed(query) # search_memory manages its own short-lived DB sessions so no # connection is held during external vector-store calls. @@ -200,10 +207,11 @@ class DialecticAgent: if explicit_repr.is_empty() and derived_repr.is_empty(): return None - # Count prefetched conclusions for telemetry - explicit_count = len(explicit_repr.explicit) + len(explicit_repr.deductive) - derived_count = len(derived_repr.explicit) + len(derived_repr.deductive) - self._prefetched_conclusion_count = explicit_count + derived_count + # Count prefetched conclusions for telemetry. `Representation.len()` + # sums all four levels (explicit/deductive/inductive/contradiction); + # the previous hand-sum dropped inductive + contradiction even + # though prefetch explicitly requests them. + self._prefetched_conclusion_count = explicit_repr.len() + derived_repr.len() # Format as two separate sections parts: list[str] = [] @@ -242,7 +250,7 @@ class DialecticAgent: if self.metric_key: task_name = self.metric_key else: - run_id = str(uuid.uuid4())[:8] + run_id = generate_nanoid() task_name = f"dialectic_chat_{run_id}" start_time = time.perf_counter() @@ -293,6 +301,23 @@ class DialecticAgent: return tool_executor, task_name, run_id, start_time + def _telemetry_context(self) -> LLMTelemetryContext: + """Build the LLMTelemetryContext shared by answer() and answer_stream(). + + Carries the instance's `_run_id` (always set in __init__) + workspace + + peer identifiers so LLMCallCompletedEvent and 's + AgentIterationEvent can attribute every per-iteration LLM call back to + this dialectic invocation. + """ + return LLMTelemetryContext( + workspace_name=self.workspace_name, + call_purpose="dialectic.answer", + parent_category="dialectic", + agent_type="dialectic", + run_id=self._run_id, + peer_name=self.observed, + ) + def _log_response_metrics( self, task_name: str, @@ -306,6 +331,7 @@ class DialecticAgent: tool_calls_count: int, thinking_content: str | None, iterations: int, + hit_input_token_cap: bool = False, ) -> None: """ Log metrics common to both streaming and non-streaming responses. @@ -374,6 +400,7 @@ class DialecticAgent: output_tokens=output_tokens, cache_read_tokens=cache_read_input_tokens or 0, cache_creation_tokens=cache_creation_input_tokens or 0, + hit_input_token_cap=hit_input_token_cap, ) ) @@ -422,6 +449,7 @@ class DialecticAgent: track_name="Dialectic Agent", max_input_tokens=settings.DIALECTIC.MAX_INPUT_TOKENS, trace_name="dialectic_chat", + telemetry=self._telemetry_context(), ) self._log_response_metrics( @@ -436,6 +464,7 @@ class DialecticAgent: tool_calls_count=len(response.tool_calls_made), thinking_content=response.thinking_content, iterations=response.iterations, + hit_input_token_cap=response.hit_input_token_cap, ) return response.content @@ -489,6 +518,7 @@ class DialecticAgent: track_name="Dialectic Agent Stream", max_input_tokens=settings.DIALECTIC.MAX_INPUT_TOKENS, trace_name="dialectic_chat", + telemetry=self._telemetry_context(), ), ) @@ -510,4 +540,5 @@ class DialecticAgent: tool_calls_count=len(response.tool_calls_made), thinking_content=response.thinking_content, iterations=response.iterations, + hit_input_token_cap=response.hit_input_token_cap, ) diff --git a/src/dreamer/dream_scheduler.py b/src/dreamer/dream_scheduler.py index eb2e2177..bad850f4 100644 --- a/src/dreamer/dream_scheduler.py +++ b/src/dreamer/dream_scheduler.py @@ -60,8 +60,17 @@ class DreamScheduler: *, observer: str, observed: str, + trigger_reason: str | None = None, + delay_reason: str | None = None, + documents_since_last_dream_at_schedule: int | None = None, + document_threshold: int | None = None, ) -> None: - """Schedule a dream for a collection after a delay.""" + """Schedule a dream for a collection after a delay. + + telemetry kwargs are captured at schedule time and threaded + through the queue payload so DreamRunEvent can attribute the dream + back to its scheduling context. + """ if not settings.DREAM.ENABLED: return @@ -76,6 +85,10 @@ class DreamScheduler: dream_type, observer=observer, observed=observed, + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule, + document_threshold=document_threshold, ) ) self.pending_dreams[work_unit_key] = task @@ -133,6 +146,10 @@ class DreamScheduler: *, observer: str, observed: str, + trigger_reason: str | None = None, + delay_reason: str | None = None, + documents_since_last_dream_at_schedule: int | None = None, + document_threshold: int | None = None, ) -> None: try: await asyncio.sleep(delay_minutes * 60) @@ -142,6 +159,10 @@ class DreamScheduler: dream_type, observer=observer, observed=observed, + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule, + document_threshold=document_threshold, ) logger.info("Executed dream for %s", work_unit_key) @@ -159,6 +180,10 @@ class DreamScheduler: *, observer: str, observed: str, + trigger_reason: str | None = None, + delay_reason: str | None = None, + documents_since_last_dream_at_schedule: int | None = None, + document_threshold: int | None = None, ) -> None: """Execute the dream by enqueueing it.""" from src import crud @@ -204,6 +229,10 @@ class DreamScheduler: observed=observed, dream_type=dream_type, session_name=session_name, + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule, + document_threshold=document_threshold, ) async def shutdown(self) -> None: @@ -274,6 +303,14 @@ async def check_and_schedule_dream( ) if documents_since_last_dream >= settings.DREAM.DOCUMENT_THRESHOLD: + # capture *why* this schedule fired (threshold) and + # *how* it will fire (idle vs immediate). The two gates were + # intentionally split — collapsing them into a single trigger_reason + # would lose the scheduling semantics. + trigger_reason = "document_threshold" + delay_reason = ( + "idle_timeout" if settings.DREAM.IDLE_TIMEOUT_MINUTES > 0 else "immediate" + ) if last_dream_at: try: last_dream_time = datetime.fromisoformat(last_dream_at) @@ -286,6 +323,9 @@ async def check_and_schedule_dream( f"Skipping dream for {collection.observer}/{collection.observed}: only {hours_since_last_dream:.1f} hours " + f"since last dream (minimum: {settings.DREAM.MIN_HOURS_BETWEEN_DREAMS})" ) + # delay_reason = "min_hours_gate" if we DID schedule, but + # we don't — return early. Telemetry only records dreams + # that actually fire. return False except (ValueError, TypeError) as e: logger.warning( @@ -345,6 +385,10 @@ async def check_and_schedule_dream( dream_type=DreamType(dream_type), observer=collection.observer, observed=collection.observed, + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream, + document_threshold=settings.DREAM.DOCUMENT_THRESHOLD, ) logger.debug( "Scheduled dream", diff --git a/src/dreamer/orchestrator.py b/src/dreamer/orchestrator.py index d09e5b67..0b529cc6 100644 --- a/src/dreamer/orchestrator.py +++ b/src/dreamer/orchestrator.py @@ -15,12 +15,12 @@ from __future__ import annotations import logging import time -import uuid from dataclasses import dataclass from datetime import datetime, timezone from typing import Any import sentry_sdk +from nanoid import generate as generate_nanoid from sqlalchemy import func, select from src import crud, models @@ -28,7 +28,7 @@ from src.config import settings from src.dependencies import tracked_db from src.dreamer.specialists import SPECIALISTS, SpecialistResult from src.dreamer.surprisal import SurprisalScore # type: ignore -from src.exceptions import SpecialistExecutionError, SurprisalError +from src.exceptions import SurprisalError from src.schemas import DreamType from src.telemetry.events import DreamRunEvent, emit from src.telemetry.logging import ( @@ -69,6 +69,12 @@ async def run_dream( observer: str, observed: str, session_name: str | None = None, + *, + dream_type: str | None = None, + trigger_reason: str | None = None, + delay_reason: str | None = None, + documents_since_last_dream_at_schedule: int | None = None, + document_threshold: int | None = None, ) -> DreamResult | None: """ Run a full dream cycle with optional surprisal-based sampling. @@ -89,7 +95,7 @@ async def run_dream( if not settings.DREAM.ENABLED: return None - run_id = str(uuid.uuid4())[:8] + run_id = generate_nanoid() task_name = f"dream_orchestrator_{run_id}" start_time = time.perf_counter() @@ -121,12 +127,12 @@ async def run_dream( deduction_result: SpecialistResult | None = None induction_result: SpecialistResult | None = None - # Phase 0: Surprisal-based sampling (if enabled) + # Surprisal-based sampling (if enabled) # Specialists are self-directed by default - hints are optional suggestions exploration_hints: list[str] | None = None if settings.DREAM.SURPRISAL.ENABLED: - logger.info(f"[{run_id}] Phase 0: Computing surprisal scores") + logger.info(f"[{run_id}] Computing surprisal scores") try: from src.dreamer.surprisal import sample_observations_with_surprisal @@ -165,91 +171,127 @@ async def run_dream( accumulate_metric(task_name, "surprisal_error", str(e), "blob") # Specialists will explore freely without hints - # Phase 1: Run deduction specialist (manages its own DB sessions) - logger.info(f"[{run_id}] Phase 1: Running deduction specialist") - deduction_specialist = SPECIALISTS["deduction"] + # Specialist phase wrapped in try/finally so DreamRunEvent ALWAYS emits — + # both for graceful failures (specialist raises Exception) AND unexpected + # exceptions including CancelledError. The orphaned-child-event problem + # before this fix: specialists.py:350 catches BaseException and re-raises, + # but the orchestrator's old `except SpecialistExecutionError` clauses + # didn't match anything actually raised in src/, so any real failure + # propagated past the emit at the bottom of this function. Now: the + # specialist try blocks catch Exception (CancelledError still propagates + # correctly), and the outer try/finally guarantees the parent event fires. + # + # Pre-init aggregate locals so the function-level return below and the + # finally's emit both see defined values even on an early exception. + duration_ms = 0.0 + total_iterations = 0 + total_input_tokens = 0 + total_output_tokens = 0 try: - deduction_result = await deduction_specialist.run( - workspace_name=workspace_name, - observer=observer, - observed=observed, - session_name=session_name, - hints=exploration_hints, - configuration=configuration, - parent_run_id=run_id, - ) - logger.info( - f"[{run_id}] Deduction completed: {deduction_result.content[:200]}..." - ) - accumulate_metric( - task_name, "deduction_result", deduction_result.content, "blob" - ) - deduction_success = deduction_result.success - except SpecialistExecutionError as e: - logger.error(f"[{run_id}] Deduction specialist failed: {e}", exc_info=True) - accumulate_metric(task_name, "deduction_error", str(e), "blob") + # Run deduction specialist (manages its own DB sessions) + logger.info(f"[{run_id}] Running deduction specialist") + deduction_specialist = SPECIALISTS["deduction"] + try: + deduction_result = await deduction_specialist.run( + workspace_name=workspace_name, + observer=observer, + observed=observed, + session_name=session_name, + hints=exploration_hints, + configuration=configuration, + parent_run_id=run_id, + ) + logger.info( + f"[{run_id}] Deduction completed: {deduction_result.content[:200]}..." + ) + accumulate_metric( + task_name, "deduction_result", deduction_result.content, "blob" + ) + deduction_success = deduction_result.success + except Exception as e: + # `Exception` (not `BaseException`) — CancelledError must still + # propagate so the worker can shut down. SpecialistExecutionError + # is no longer raised by `src/`, but the catch is broad enough to + # cover provider/DB/tool/validation errors. + logger.error(f"[{run_id}] Deduction specialist failed: {e}", exc_info=True) + accumulate_metric(task_name, "deduction_error", str(e), "blob") - # Phase 2: Run induction specialist (after deduction so it can see new deductive obs) - logger.info(f"[{run_id}] Phase 2: Running induction specialist") - induction_specialist = SPECIALISTS["induction"] - try: - induction_result = await induction_specialist.run( - workspace_name=workspace_name, - observer=observer, - observed=observed, - session_name=session_name, - hints=exploration_hints, - configuration=configuration, - parent_run_id=run_id, - ) - logger.info( - f"[{run_id}] Induction completed: {induction_result.content[:200]}..." - ) - accumulate_metric( - task_name, "induction_result", induction_result.content, "blob" - ) - induction_success = induction_result.success - except SpecialistExecutionError as e: - logger.error(f"[{run_id}] Induction specialist failed: {e}", exc_info=True) - accumulate_metric(task_name, "induction_error", str(e), "blob") + # Run induction specialist (after deduction so it can see new deductive obs) + logger.info(f"[{run_id}] Running induction specialist") + induction_specialist = SPECIALISTS["induction"] + try: + induction_result = await induction_specialist.run( + workspace_name=workspace_name, + observer=observer, + observed=observed, + session_name=session_name, + hints=exploration_hints, + configuration=configuration, + parent_run_id=run_id, + ) + logger.info( + f"[{run_id}] Induction completed: {induction_result.content[:200]}..." + ) + accumulate_metric( + task_name, "induction_result", induction_result.content, "blob" + ) + induction_success = induction_result.success + except Exception as e: + logger.error(f"[{run_id}] Induction specialist failed: {e}", exc_info=True) + accumulate_metric(task_name, "induction_error", str(e), "blob") - # Log final metrics - duration_ms = (time.perf_counter() - start_time) * 1000 - accumulate_metric(task_name, "total_duration", duration_ms, "ms") + # Log final metrics + duration_ms = (time.perf_counter() - start_time) * 1000 + accumulate_metric(task_name, "total_duration", duration_ms, "ms") - logger.info(f"[{run_id}] Dream cycle completed in {duration_ms:.0f}ms") - log_performance_metrics("dream_orchestrator", run_id) - - # Aggregate metrics from specialist results - total_iterations = (deduction_result.iterations if deduction_result else 0) + ( - induction_result.iterations if induction_result else 0 - ) - total_input_tokens = (deduction_result.input_tokens if deduction_result else 0) + ( - induction_result.input_tokens if induction_result else 0 - ) - total_output_tokens = ( - deduction_result.output_tokens if deduction_result else 0 - ) + (induction_result.output_tokens if induction_result else 0) - - # Emit DreamRunEvent with aggregated metrics - emit( - DreamRunEvent( - run_id=run_id, - workspace_name=workspace_name, - session_name=session_name, - observer=observer, - observed=observed, - specialists_run=["deduction", "induction"], - deduction_success=deduction_success, - induction_success=induction_success, - surprisal_enabled=settings.DREAM.SURPRISAL.ENABLED, - surprisal_conclusion_count=surprisal_observation_count, - total_iterations=total_iterations, - total_input_tokens=total_input_tokens, - total_output_tokens=total_output_tokens, - total_duration_ms=duration_ms, - ) - ) + logger.info(f"[{run_id}] Dream cycle completed in {duration_ms:.0f}ms") + log_performance_metrics("dream_orchestrator", run_id) + finally: + # Emit DreamRunEvent unconditionally so analytics see a parent for + # every DreamSpecialistEvent. Aggregation guards None specialist + # results. Emit-side errors are swallowed by the global telemetry + # path; a defensive try around event construction protects against + # schema-validation surprises during partial state. + if duration_ms == 0.0: + duration_ms = (time.perf_counter() - start_time) * 1000 + try: + total_iterations = ( + deduction_result.iterations if deduction_result else 0 + ) + (induction_result.iterations if induction_result else 0) + total_input_tokens = ( + deduction_result.input_tokens if deduction_result else 0 + ) + (induction_result.input_tokens if induction_result else 0) + total_output_tokens = ( + deduction_result.output_tokens if deduction_result else 0 + ) + (induction_result.output_tokens if induction_result else 0) + emit( + DreamRunEvent( + run_id=run_id, + workspace_name=workspace_name, + session_name=session_name, + observer=observer, + observed=observed, + specialists_run=["deduction", "induction"], + deduction_success=deduction_success, + induction_success=induction_success, + surprisal_enabled=settings.DREAM.SURPRISAL.ENABLED, + surprisal_conclusion_count=surprisal_observation_count, + total_iterations=total_iterations, + total_input_tokens=total_input_tokens, + total_output_tokens=total_output_tokens, + total_duration_ms=duration_ms, + # scheduling context threaded through the + # queue payload by check_and_schedule_dream. + dream_type=dream_type, + enabled_types_count=len(settings.DREAM.ENABLED_TYPES), + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule, + document_threshold=document_threshold, + ) + ) + except Exception: # pragma: no cover - telemetry must not raise + logger.debug("Failed to emit DreamRunEvent", exc_info=True) return DreamResult( run_id=run_id, @@ -315,6 +357,12 @@ DREAM: {payload.dream_type} documents for {workspace_name}/{payload.observer}/{p observer=payload.observer, observed=payload.observed, session_name=payload.session_name, + # scheduling context — propagated to DreamRunEvent. + dream_type=payload.dream_type.value, + trigger_reason=payload.trigger_reason, + delay_reason=payload.delay_reason, + documents_since_last_dream_at_schedule=payload.documents_since_last_dream_at_schedule, + document_threshold=payload.document_threshold, ) # Log completion (telemetry event already emitted in run_dream) diff --git a/src/dreamer/specialists.py b/src/dreamer/specialists.py index c7277586..4d3ecc97 100644 --- a/src/dreamer/specialists.py +++ b/src/dreamer/specialists.py @@ -12,17 +12,20 @@ from __future__ import annotations import logging import time -import uuid from abc import ABC, abstractmethod +from collections import Counter from collections.abc import Callable from dataclasses import dataclass -from typing import Any +from typing import Any, cast + +from nanoid import generate as generate_nanoid from src import crud, schemas from src.config import ConfiguredModelSettings, settings from src.dependencies import tracked_db from src.exceptions import ValidationException from src.llm import HonchoLLMCallResponse, honcho_llm_call +from src.llm.types import LLMTelemetryContext from src.schemas import ResolvedConfiguration from src.telemetry import prometheus_metrics from src.telemetry.events import DreamSpecialistEvent, emit @@ -153,154 +156,269 @@ If you update it, send the full deduplicated list and remove stale entries. Returns: SpecialistResult with metrics and content """ - run_id = parent_run_id or str(uuid.uuid4())[:8] + run_id = parent_run_id or generate_nanoid() task_name = f"dreamer_{self.name}_{run_id}" start_time = time.perf_counter() - # Short-lived DB session for preflight operations - async with tracked_db("dream.specialist.preflight") as db: - await crud.get_peer(db, workspace_name, schemas.PeerCreate(name=observer)) - if observer != observed: + # Telemetry state initialized BEFORE the try so the finally block can + # always read consistent values. Without this, a failure in preflight + # (peer lookup, peer-card preload, create_tool_executor, get_model_config, + # prompt construction) would bypass the finally entirely and the run + # would disappear from failure-path analytics — orphaning the + # downstream DreamRunEvent. + specialist_success = False + specialist_error_class: str | None = None + response: HonchoLLMCallResponse[str] | None = None + + # Rollups initialized here so they're accessible from the finally + # block on the failure path (where they stay at defaults). + created_observation_count = 0 + deleted_observation_count = 0 + peer_card_updated = False + search_tool_calls_count = 0 + duration_ms = 0.0 + # Per-level rollups — accumulated from each create/delete_observations + # tool call's metadata.levels list. Counter rather than list[str] so + # the emitted dict stays compact even when the specialist produces + # many observations. + created_counts_by_level: Counter[str] = Counter() + deleted_counts_by_level: Counter[str] = Counter() + + try: + # Short-lived DB session for preflight operations + async with tracked_db("dream.specialist.preflight") as db: await crud.get_peer( - db, workspace_name, schemas.PeerCreate(name=observed) + db, workspace_name, schemas.PeerCreate(name=observer) + ) + if observer != observed: + await crud.get_peer( + db, workspace_name, schemas.PeerCreate(name=observed) + ) + + # Determine if peer card tools should be included + peer_card_enabled = ( + configuration is None or configuration.peer_card.create ) - # Determine if peer card tools should be included - peer_card_enabled = configuration is None or configuration.peer_card.create + # Fetch current peer card to inject into prompt (saves a tool call) + current_peer_card: list[str] | None = None + if peer_card_enabled: + current_peer_card = await crud.get_peer_card( + db, + workspace_name=workspace_name, + observer=observer, + observed=observed, + ) + # DB session closed — LLM calls happen without holding a connection - # Fetch current peer card to inject into prompt (saves a tool call) - current_peer_card: list[str] | None = None - if peer_card_enabled: - current_peer_card = await crud.get_peer_card( - db, - workspace_name=workspace_name, - observer=observer, - observed=observed, - ) - # DB session closed — LLM calls happen without holding a connection + # Build messages + messages: list[dict[str, str]] = [ + { + "role": "system", + "content": self.build_system_prompt( + observed, peer_card_enabled=peer_card_enabled + ), + }, + { + "role": "user", + "content": self.build_user_prompt(hints, current_peer_card), + }, + ] - # Build messages - messages: list[dict[str, str]] = [ - { - "role": "system", - "content": self.build_system_prompt( - observed, peer_card_enabled=peer_card_enabled - ), - }, - { - "role": "user", - "content": self.build_user_prompt(hints, current_peer_card), - }, - ] - - # Create tool executor with telemetry context - tool_executor: Callable[ - [str, dict[str, Any]], Any - ] = await create_tool_executor( - workspace_name=workspace_name, - observer=observer, - observed=observed, - session_name=session_name, - include_observation_ids=True, - history_token_limit=settings.DREAM.HISTORY_TOKEN_LIMIT, - configuration=configuration, - run_id=run_id, - agent_type=self.name, - parent_category="dream", - ) - - model_config = self.get_model_config() - - # Respect operator-configured max_output_tokens on the specialist's - # ModelConfig (e.g. DREAM_DEDUCTION_MODEL_CONFIG__MAX_OUTPUT_TOKENS). - # Only fall back to the specialist's hardcoded default when the - # config leaves max_output_tokens unset or non-positive. - configured_max = model_config.max_output_tokens - effective_max_tokens = ( - configured_max - if configured_max and configured_max > 0 - else self.get_max_tokens() - ) - - # Track iterations via callback - iteration_count = 0 - - def iteration_callback(data: Any) -> None: - nonlocal iteration_count - iteration_count = data.iteration - - # Run the agent loop - response: HonchoLLMCallResponse[str] = await honcho_llm_call( - model_config=model_config, - prompt="", # Ignored since we pass messages - max_tokens=effective_max_tokens, - tools=self.get_tools(peer_card_enabled=peer_card_enabled), - tool_choice=None, - tool_executor=tool_executor, - max_tool_iterations=self.get_max_iterations(), - messages=messages, - track_name=f"Dreamer/{self.name}", - iteration_callback=iteration_callback, - ) - - # Log metrics - duration_ms = (time.perf_counter() - start_time) * 1000 - accumulate_metric(task_name, "total_duration", duration_ms, "ms") - accumulate_metric( - task_name, "tool_calls", len(response.tool_calls_made), "count" - ) - accumulate_metric(task_name, "input_tokens", response.input_tokens, "count") - accumulate_metric(task_name, "output_tokens", response.output_tokens, "count") - - # Prometheus metrics - if settings.METRICS.ENABLED: - prometheus_metrics.record_dreamer_tokens( - count=response.input_tokens, - specialist_name=self.name, - token_type=TokenTypes.INPUT.value, - ) - prometheus_metrics.record_dreamer_tokens( - count=response.output_tokens, - specialist_name=self.name, - token_type=TokenTypes.OUTPUT.value, - ) - - logger.info( - f"{self.name}: Completed in {duration_ms:.0f}ms, " - + f"{len(response.tool_calls_made)} tool calls, " - + f"{response.input_tokens} in / {response.output_tokens} out" - ) - - log_performance_metrics(f"dreamer_{self.name}", run_id) - - # Emit telemetry event - emit( - DreamSpecialistEvent( - run_id=run_id, - specialist_type=self.name, + # Create tool executor with telemetry context + tool_executor: Callable[ + [str, dict[str, Any]], Any + ] = await create_tool_executor( workspace_name=workspace_name, observer=observer, observed=observed, - iterations=iteration_count, + session_name=session_name, + include_observation_ids=True, + history_token_limit=settings.DREAM.HISTORY_TOKEN_LIMIT, + configuration=configuration, + run_id=run_id, + agent_type=self.name, + parent_category="dream", + ) + + model_config = self.get_model_config() + + # Respect operator-configured max_output_tokens on the specialist's + # ModelConfig (e.g. DREAM_DEDUCTION_MODEL_CONFIG__MAX_OUTPUT_TOKENS). + # Only fall back to the specialist's hardcoded default when the + # config leaves max_output_tokens unset or non-positive. + configured_max = model_config.max_output_tokens + effective_max_tokens = ( + configured_max + if configured_max and configured_max > 0 + else self.get_max_tokens() + ) + + # call_purpose maps "deduction"/"induction" specialist names onto the + # closed CallPurpose enum slugs without importing the enum here. + call_purpose_slug = f"dream.{self.name}" + + # Run the agent loop + response = await honcho_llm_call( + model_config=model_config, + prompt="", # Ignored since we pass messages + max_tokens=effective_max_tokens, + tools=self.get_tools(peer_card_enabled=peer_card_enabled), + tool_choice=None, + tool_executor=tool_executor, + max_tool_iterations=self.get_max_iterations(), + messages=messages, + track_name=f"Dreamer/{self.name}", + telemetry=LLMTelemetryContext( + workspace_name=workspace_name, + call_purpose=call_purpose_slug, + parent_category="dream", + agent_type=self.name, + run_id=run_id, + observer=observer, + observed=observed, + ), + ) + + # Log metrics + duration_ms = (time.perf_counter() - start_time) * 1000 + accumulate_metric(task_name, "total_duration", duration_ms, "ms") + accumulate_metric( + task_name, "tool_calls", len(response.tool_calls_made), "count" + ) + accumulate_metric(task_name, "input_tokens", response.input_tokens, "count") + accumulate_metric( + task_name, "output_tokens", response.output_tokens, "count" + ) + + # Prometheus metrics + if settings.METRICS.ENABLED: + prometheus_metrics.record_dreamer_tokens( + count=response.input_tokens, + specialist_name=self.name, + token_type=TokenTypes.INPUT.value, + ) + prometheus_metrics.record_dreamer_tokens( + count=response.output_tokens, + specialist_name=self.name, + token_type=TokenTypes.OUTPUT.value, + ) + + logger.info( + f"{self.name}: Completed in {duration_ms:.0f}ms, " + + f"{len(response.tool_calls_made)} tool calls, " + + f"{response.input_tokens} in / {response.output_tokens} out" + ) + + log_performance_metrics(f"dreamer_{self.name}", run_id) + + # count actual observations created/deleted from the + # ToolResult.metadata that stashed on `all_tool_calls[i]`. + # Counting tool-name occurrences would mis-attribute: a single + # create_observations call can produce N (or zero) observations. The + # truth lives in the handler's returned metadata. + _search_tools = { + "search_memory", + "search_messages", + "search_messages_temporal", + } + for tc in response.tool_calls_made: + tool_name_any: Any = tc.get("tool_name") or tc.get("name") + meta_any: Any = tc.get("tool_result_metadata") or {} + if tool_name_any in _search_tools: + search_tool_calls_count += 1 + if isinstance(meta_any, dict): + # `meta_any` is `dict[Unknown, Unknown]` after the isinstance + # narrow because tool_calls_made is typed list[dict[str, Any]]. + # Cast to the expected dict shape to silence the partial-known + # warning without losing runtime safety. + meta_dict = cast(dict[str, Any], meta_any) + created_val: Any = meta_dict.get("created_count") or 0 + deleted_val: Any = meta_dict.get("deleted_count") or 0 + created_observation_count += int(created_val) + deleted_observation_count += int(deleted_val) + if meta_dict.get("peer_card_updated"): + peer_card_updated = True + # Accumulate per-level counts from create/delete observations. + # Both handlers stash `{"levels": ["explicit", "deductive", ...]}` + # in metadata (agent_tools.py:1373 + agent_tools.py:2011). + levels_any: Any = meta_dict.get("levels") + if isinstance(levels_any, list): + levels_list = cast(list[Any], levels_any) + level_strs = [ + str(level) for level in levels_list if level is not None + ] + # Tool-name dispatch decides which counter to update — + # create_observations metadata has `created_count`, + # delete_observations has `deleted_count`. + if "created_count" in meta_dict: + created_counts_by_level.update(level_strs) + elif "deleted_count" in meta_dict: + deleted_counts_by_level.update(level_strs) + + specialist_success = True + + return SpecialistResult( + run_id=run_id, + specialist_type=self.name, + iterations=response.iterations, tool_calls_count=len(response.tool_calls_made), input_tokens=response.input_tokens, output_tokens=response.output_tokens, duration_ms=duration_ms, success=True, + content=response.content, ) - ) - - return SpecialistResult( - run_id=run_id, - specialist_type=self.name, - iterations=iteration_count, - tool_calls_count=len(response.tool_calls_made), - input_tokens=response.input_tokens, - output_tokens=response.output_tokens, - duration_ms=duration_ms, - success=True, - content=response.content, - ) + except BaseException as e: + # BaseException (not Exception) — asyncio.CancelledError doesn't + # inherit from Exception in py3.8+, and we want the failure + # telemetry populated for cancellations too (worker shutdown, + # client disconnect). `raise` preserves cancellation semantics. + specialist_error_class = type(e).__name__ + if duration_ms == 0.0: + duration_ms = (time.perf_counter() - start_time) * 1000 + raise + finally: + # Emit DreamSpecialistEvent unconditionally so the success=False + # path of the schema is actually populated. Telemetry must not + # raise from inside finally during exception propagation; the + # emitter itself swallows errors but we add a defensive try + # in case event construction fails (e.g. schema validation). + try: + tool_calls_count = ( + len(response.tool_calls_made) if response is not None else 0 + ) + input_tokens = response.input_tokens if response is not None else 0 + output_tokens = response.output_tokens if response is not None else 0 + iterations = response.iterations if response is not None else 0 + emit( + DreamSpecialistEvent( + run_id=run_id, + specialist_type=self.name, + workspace_name=workspace_name, + observer=observer, + observed=observed, + iterations=iterations, + tool_calls_count=tool_calls_count, + input_tokens=input_tokens, + output_tokens=output_tokens, + duration_ms=duration_ms, + success=specialist_success, + error_class=specialist_error_class, + # denormalized rollups (all 0 on the failure path) + created_observation_count=created_observation_count, + deleted_observation_count=deleted_observation_count, + peer_card_updated=peer_card_updated, + search_tool_calls_count=search_tool_calls_count, + # Per-level breakdowns — `dict(Counter)` keeps the + # serialized event compact (zero-count levels are + # omitted, not enumerated). + created_counts_by_level=dict(created_counts_by_level), + deleted_counts_by_level=dict(deleted_counts_by_level), + ) + ) + except Exception: # pragma: no cover - telemetry must not raise + logger.debug("Failed to emit DreamSpecialistEvent", exc_info=True) class DeductionSpecialist(BaseSpecialist): @@ -552,8 +670,8 @@ Use `create_observations_inductive`. "content": "The pattern or generalization", "source_ids": ["id1", "id2", "id3"], "sources": ["evidence 1", "evidence 2"], - "pattern_type": "tendency", // preference|behavior|personality|tendency|correlation - "confidence": "medium" // low (2 sources), medium (3-4), high (5+) + "pattern_type": "tendency", // preference|behavior|personality|tendency|correlation + "confidence": "medium" // low (2 sources), medium (3-4), high (5+) }}] }} ``` diff --git a/src/embedding_client.py b/src/embedding_client.py index 850597f5..60516bc5 100644 --- a/src/embedding_client.py +++ b/src/embedding_client.py @@ -1,8 +1,10 @@ import asyncio import logging import threading +import time from collections import defaultdict -from typing import Any, NamedTuple +from collections.abc import Awaitable, Callable +from typing import Any, Literal, NamedTuple, TypeVar import tiktoken from google import genai @@ -13,6 +15,115 @@ from .config import EmbeddingModelConfig, resolve_embedding_model_config, settin logger = logging.getLogger(__name__) +_T = TypeVar("_T") + + +async def _emit_embedding_call( + *, + provider: str, + model: str, + texts: list[str], + input_tokens_estimate: int, + fn: Callable[[], Awaitable[_T]], + is_final_attempt: bool = True, +) -> _T: + """time a single embedding-provider call, emit + `embedding.call.completed` on both success and exception, and return the + call's result. Errors propagate unchanged — telemetry never bleeds into + the caller's control flow. + + Caller-supplied `texts` is used only for `input_count`; we don't keep the + list around for the event to avoid leaking content into telemetry. + + `is_final_attempt` defaults to True so one-shot callers (`embed`, + `simple_batch_embed`) get correct semantics without changes. Retry-loop + callers (`_process_batch`) pass the real attempt index so dashboards + can distinguish exhausted retries from mid-retry failures. + """ + start = time.perf_counter() + error: BaseException | None = None + try: + return await fn() + except BaseException as exc: + error = exc + raise + finally: + if error is None: + outcome: Literal["success", "error", "cancelled"] = "success" + elif isinstance(error, asyncio.CancelledError): + outcome = "cancelled" + else: + outcome = "error" + _publish_embedding_event( + provider=provider, + model=model, + input_count=len(texts), + input_tokens_estimate=input_tokens_estimate, + duration_ms=(time.perf_counter() - start) * 1000, + outcome=outcome, + error=error, + is_final_attempt=is_final_attempt, + ) + + +def _publish_embedding_event( + *, + provider: str, + model: str, + input_count: int, + input_tokens_estimate: int, + duration_ms: float, + outcome: Literal["success", "error", "cancelled"], + error: BaseException | None, + is_final_attempt: bool, +) -> None: + """Build and emit the EmbeddingCallCompletedEvent. Best-effort.""" + try: + from src.telemetry.events import ( + EmbeddingCallCompletedEvent, + EmbeddingCallPurpose, + emit, + ) + from src.utils.types import ( + get_embedding_call_purpose, + get_embedding_parent_category, + get_embedding_run_id, + get_embedding_workspace_name, + ) + + # call_purpose travels via ContextVar so embedding callers don't have + # to thread it through every call site. Unknown values drop to None + # rather than raising — keeps telemetry resilient to drift. + purpose_slug = get_embedding_call_purpose() + call_purpose: EmbeddingCallPurpose | None = None + if purpose_slug: + try: + call_purpose = EmbeddingCallPurpose(purpose_slug) + except ValueError: + logger.debug( + "Unknown embedding_call_purpose=%r; emitting without", + purpose_slug, + ) + + emit( + EmbeddingCallCompletedEvent( + workspace_name=get_embedding_workspace_name(), + call_purpose=call_purpose, + parent_category=get_embedding_parent_category(), + provider=provider, + model=model, + input_count=input_count, + input_tokens_estimate=input_tokens_estimate, + duration_ms=duration_ms, + outcome=outcome, + is_final_attempt=is_final_attempt, + error_class=type(error).__name__ if error is not None else None, + run_id=get_embedding_run_id(), + ) + ) + except Exception: # pragma: no cover - telemetry must not raise + logger.debug("Failed to emit EmbeddingCallCompletedEvent", exc_info=True) + class BatchItem(NamedTuple): """A single item in a batch with its metadata.""" @@ -20,6 +131,7 @@ class BatchItem(NamedTuple): text: str text_id: str chunk_index: int + token_count: int class _EmbeddingClient: @@ -93,22 +205,49 @@ class _EmbeddingClient: f"Query exceeds maximum token limit of {self.max_embedding_tokens} tokens (got {token_count} tokens)" ) + # Bind the typed client at the dispatch site so pyright can narrow it + # for the closures without needing `assert isinstance(...)` (bandit + # B101). The closures close over the narrowed local, not `self.client`. if isinstance(self.client, genai.Client): - response = await self.client.aio.models.embed_content( + gemini_client = self.client + + async def _call_gemini() -> list[float]: + response = await gemini_client.aio.models.embed_content( + model=self.model, + contents=query, + config={"output_dimensionality": self.vector_dimensions}, + ) + if not response.embeddings or not response.embeddings[0].values: + raise ValueError("No embedding returned from Gemini API") + return self._validate_embedding_dimensions( + response.embeddings[0].values + ) + + return await _emit_embedding_call( + provider=self.transport, model=self.model, - contents=query, - config={"output_dimensionality": self.vector_dimensions}, + texts=[query], + input_tokens_estimate=token_count, + fn=_call_gemini, ) - if not response.embeddings or not response.embeddings[0].values: - raise ValueError("No embedding returned from Gemini API") - return self._validate_embedding_dimensions(response.embeddings[0].values) - else: # openai + + openai_client = self.client + + async def _call_openai() -> list[float]: openai_kwargs: dict[str, Any] = {"model": self.model, "input": [query]} if self.send_dimensions: openai_kwargs["dimensions"] = self.vector_dimensions - response = await self.client.embeddings.create(**openai_kwargs) + response = await openai_client.embeddings.create(**openai_kwargs) return self._validate_embedding_dimensions(response.data[0].embedding) + return await _emit_embedding_call( + provider=self.transport, + model=self.model, + texts=[query], + input_tokens_estimate=token_count, + fn=_call_openai, + ) + async def simple_batch_embed(self, texts: list[str]) -> list[list[float]]: """ Simple batch embedding for a list of text strings. @@ -126,7 +265,11 @@ class _EmbeddingClient: for i in range(0, len(texts), self.max_batch_size): batch = texts[i : i + self.max_batch_size] - try: + + async def _embed_batch(batch: list[str] = batch) -> list[list[float]]: + """One provider call for one batch. Lifted into a closure so + _emit_embedding_call can time + emit + propagate errors.""" + batch_embeddings: list[list[float]] = [] if isinstance(self.client, genai.Client): # Type cast needed due to genai type signature complexity response = await self.client.aio.models.embed_content( @@ -137,7 +280,7 @@ class _EmbeddingClient: if response.embeddings: for emb in response.embeddings: if emb.values: - embeddings.append( + batch_embeddings.append( self._validate_embedding_dimensions(emb.values) ) else: # openai @@ -148,12 +291,26 @@ class _EmbeddingClient: if self.send_dimensions: openai_kwargs["dimensions"] = self.vector_dimensions response = await self.client.embeddings.create(**openai_kwargs) - embeddings.extend( + batch_embeddings.extend( [ self._validate_embedding_dimensions(data.embedding) for data in response.data ] ) + return batch_embeddings + + try: + # Pre-compute the tiktoken estimate ONCE for telemetry; the + # batch contents don't change between attempts. + tokens_estimate = sum(len(self.encoding.encode(t)) for t in batch) + batch_embeddings = await _emit_embedding_call( + provider=self.transport, + model=self.model, + texts=batch, + input_tokens_estimate=tokens_estimate, + fn=_embed_batch, + ) + embeddings.extend(batch_embeddings) except Exception as e: # Check if it's a token limit error and re-raise as ValueError for consistency if "token" in str(e).lower(): @@ -248,7 +405,9 @@ class _EmbeddingClient: current_batch = [] current_tokens = 0 - current_batch.append(BatchItem(chunk_text, text_id, chunk_idx)) + current_batch.append( + BatchItem(chunk_text, text_id, chunk_idx, chunk_tokens) + ) current_tokens += chunk_tokens if current_batch: @@ -271,42 +430,53 @@ class _EmbeddingClient: """ last_exception: Exception | None = None + async def _call_provider() -> dict[str, dict[int, list[float]]]: + """One provider call. Lifted out of the retry loop so + _emit_embedding_call emits a separate event per attempt — each + attempt is a distinct provider hit and shows up as its own line + item in analytics.""" + result: dict[str, dict[int, list[float]]] = defaultdict(dict) + if isinstance(self.client, genai.Client): + response = await self.client.aio.models.embed_content( + model=self.model, + contents=[item.text for item in batch], + config={"output_dimensionality": self.vector_dimensions}, + ) + if response.embeddings: + for item, embedding in zip(batch, response.embeddings, strict=True): + if embedding.values: + result[item.text_id][item.chunk_index] = ( + self._validate_embedding_dimensions(embedding.values) + ) + else: # openai + openai_kwargs: dict[str, Any] = { + "model": self.model, + "input": [item.text for item in batch], + } + if self.send_dimensions: + openai_kwargs["dimensions"] = self.vector_dimensions + response = await self.client.embeddings.create(**openai_kwargs) + for item, embedding_data in zip(batch, response.data, strict=True): + result[item.text_id][item.chunk_index] = ( + self._validate_embedding_dimensions(embedding_data.embedding) + ) + return result + + # Token counts were computed during chunk prep; reuse them here so the + # provider call doesn't re-encode every chunk just for the size proxy. + batch_tokens_estimate = sum(item.token_count for item in batch) + batch_texts = [item.text for item in batch] + for attempt in range(max_retries): try: - # Organize embeddings by text_id and chunk_index - result: dict[str, dict[int, list[float]]] = defaultdict(dict) - - if isinstance(self.client, genai.Client): - response = await self.client.aio.models.embed_content( - model=self.model, - contents=[item.text for item in batch], - config={"output_dimensionality": self.vector_dimensions}, - ) - if response.embeddings: - for item, embedding in zip( - batch, response.embeddings, strict=True - ): - if embedding.values: - result[item.text_id][item.chunk_index] = ( - self._validate_embedding_dimensions( - embedding.values - ) - ) - else: # openai - openai_kwargs: dict[str, Any] = { - "model": self.model, - "input": [item.text for item in batch], - } - if self.send_dimensions: - openai_kwargs["dimensions"] = self.vector_dimensions - response = await self.client.embeddings.create(**openai_kwargs) - for item, embedding_data in zip(batch, response.data, strict=True): - result[item.text_id][item.chunk_index] = ( - self._validate_embedding_dimensions( - embedding_data.embedding - ) - ) - + result = await _emit_embedding_call( + provider=self.transport, + model=self.model, + texts=batch_texts, + input_tokens_estimate=batch_tokens_estimate, + fn=_call_provider, + is_final_attempt=(attempt >= max_retries - 1), + ) return dict(result) except Exception as e: diff --git a/src/llm/api.py b/src/llm/api.py index 2a8f0529..9c9a628d 100644 --- a/src/llm/api.py +++ b/src/llm/api.py @@ -38,6 +38,7 @@ from .types import ( HonchoLLMCallResponse, HonchoLLMCallStreamChunk, IterationCallback, + LLMTelemetryContext, ReasoningEffortType, StreamingResponseWithMetadata, ) @@ -73,6 +74,7 @@ async def honcho_llm_call( max_input_tokens: int | None = None, trace_name: str | None = None, iteration_callback: IterationCallback | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> HonchoLLMCallResponse[M]: ... @@ -102,6 +104,7 @@ async def honcho_llm_call( max_input_tokens: int | None = None, trace_name: str | None = None, iteration_callback: IterationCallback | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> HonchoLLMCallResponse[str]: ... @@ -131,6 +134,7 @@ async def honcho_llm_call( max_input_tokens: int | None = None, trace_name: str | None = None, iteration_callback: IterationCallback | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> AsyncIterator[HonchoLLMCallStreamChunk] | StreamingResponseWithMetadata: ... @@ -160,6 +164,7 @@ async def honcho_llm_call( max_input_tokens: int | None = None, trace_name: str | None = None, iteration_callback: IterationCallback | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> ( HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk] @@ -236,6 +241,8 @@ async def honcho_llm_call( tools=tools, tool_choice=tool_choice, selected_config=plan.selected_config, + plan=plan, + telemetry=telemetry, ) return await honcho_llm_call_inner( plan.provider, @@ -254,6 +261,8 @@ async def honcho_llm_call( tools=tools, tool_choice=tool_choice, selected_config=plan.selected_config, + plan=plan, + telemetry=telemetry, ) decorated = _call_with_provider_selection @@ -306,9 +315,105 @@ async def honcho_llm_call( # Tool-less path: call once and return. if not tools or not tool_executor: - result: ( - HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk] - ) = await decorated() + # enforce `max_input_tokens` for tool-less calls too. Before + # this change, only `execute_tool_loop` consumed the kwarg — the + # deriver passed it but it was silently dropped, so the cap-hit + # signal it needed for RepresentationCompletedEvent could not be + # measured. Now we run the same message-list truncation helper + # and surface a `hit_input_token_cap` boolean on the response. + # + # The signal is purely token-based ("did the input exceed cap?") + # rather than message-count-based — the helper deliberately keeps + # the last conversation unit even when it's oversized (see + # truncate_messages_to_fit), so a single-message over-cap input + # (the deriver's prompt-only case) would otherwise silently fly + # through with hit=False. Token-based comparison catches it. + toolless_hit_input_token_cap = False + toolless_messages = messages + if max_input_tokens is not None: + from .conversation import count_message_tokens, truncate_messages_to_fit + + base_messages = messages or [{"role": "user", "content": prompt}] + toolless_hit_input_token_cap = ( + count_message_tokens(base_messages) > max_input_tokens + ) + toolless_messages = truncate_messages_to_fit( + base_messages, max_input_tokens + ) + + # Re-bind the closure to use the truncated message list. + if toolless_messages is not None: + captured_messages = toolless_messages + + async def _toolless_call() -> ( + HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk] + ): + plan = _get_attempt_plan() + # Branch on stream so each call site lands on the right + # `Literal[True]/False` overload — basedpyright won't infer + # which overload a runtime `bool` matches. + if stream: + return await honcho_llm_call_inner( + plan.provider, + plan.model, + prompt, + max_tokens, + response_model=response_model, + json_mode=json_mode, + temperature=effective_temperature(temperature), + stop_seqs=stop_seqs, + reasoning_effort=plan.reasoning_effort, + verbosity=verbosity, + thinking_budget_tokens=plan.thinking_budget_tokens, + stream=True, + client_override=plan.client, + tools=tools, + tool_choice=tool_choice, + selected_config=plan.selected_config, + plan=plan, + telemetry=telemetry, + messages=captured_messages, + ) + return await honcho_llm_call_inner( + plan.provider, + plan.model, + prompt, + max_tokens, + response_model=response_model, + json_mode=json_mode, + temperature=effective_temperature(temperature), + stop_seqs=stop_seqs, + reasoning_effort=plan.reasoning_effort, + verbosity=verbosity, + thinking_budget_tokens=plan.thinking_budget_tokens, + stream=False, + client_override=plan.client, + tools=tools, + tool_choice=tool_choice, + selected_config=plan.selected_config, + plan=plan, + telemetry=telemetry, + messages=captured_messages, + ) + + wrapped = _toolless_call + if track_name: + wrapped = ai_track(track_name)(wrapped) + if enable_retry: + wrapped = retry( + stop=stop_after_attempt(retry_attempts), + wait=wait_exponential(multiplier=1, min=4, max=10), + before_sleep=before_retry_callback, + )(wrapped) + result: ( + HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk] + ) = await wrapped() + else: + result = await decorated() + + if toolless_hit_input_token_cap and isinstance(result, HonchoLLMCallResponse): + result.hit_input_token_cap = True + if trace_name and isinstance(result, HonchoLLMCallResponse): log_reasoning_trace( task_type=trace_name, @@ -346,6 +451,7 @@ async def honcho_llm_call( before_retry_callback=before_retry_callback, stream_final=stream_final_only, iteration_callback=iteration_callback, + telemetry=telemetry, ) if trace_name and isinstance(result, HonchoLLMCallResponse): log_reasoning_trace( diff --git a/src/llm/executor.py b/src/llm/executor.py index d96008af..87db8dd0 100644 --- a/src/llm/executor.py +++ b/src/llm/executor.py @@ -11,6 +11,9 @@ Used by: from __future__ import annotations +import asyncio +import logging +import time from collections.abc import AsyncIterator from typing import Any, Literal, TypeVar, overload @@ -23,17 +26,35 @@ from .backend import StreamChunk as BackendStreamChunk from .backend import ToolCallResult from .registry import CLIENTS, backend_for_provider from .request_builder import execute_completion, execute_stream -from .runtime import effective_config_for_call +from .runtime import AttemptPlan, effective_config_for_call from .types import ( HonchoLLMCallResponse, HonchoLLMCallStreamChunk, + LLMTelemetryContext, ProviderClient, ReasoningEffortType, ) +logger = logging.getLogger(__name__) + M = TypeVar("M", bound=BaseModel) +def _outcome_from_error( + err: BaseException | None, +) -> Literal["success", "error", "cancelled"]: + """Map a finally-block error into the telemetry outcome literal. + + CancelledError is a normal control-flow event (client disconnect, server + shutdown) — surface it distinctly so it doesn't pollute error-rate alerts. + """ + if err is None: + return "success" + if isinstance(err, asyncio.CancelledError): + return "cancelled" + return "error" + + def _tool_call_result_to_dict(tool_call: ToolCallResult) -> dict[str, Any]: result = { "id": tool_call.id, @@ -45,6 +66,101 @@ def _tool_call_result_to_dict(tool_call: ToolCallResult) -> dict[str, Any]: return result +def _emit_llm_call_completed( + *, + plan: AttemptPlan | None, + telemetry: LLMTelemetryContext | None, + provider: ModelTransport, + model: str, + max_tokens: int, + duration_ms: float, + has_tools: bool, + was_stream: bool, + outcome: Literal["success", "error", "cancelled"], + result: BackendCompletionResult | None, + error: BaseException | None, +) -> None: + """Build and emit an LLMCallCompletedEvent. Best-effort; swallows errors so + telemetry failures never bleed into the LLM call path.""" + try: + from src.telemetry.events import CallPurpose, LLMCallCompletedEvent, emit + + # call_purpose is a string slug on LLMTelemetryContext; validate against + # the enum here (silent drop on unknown values keeps telemetry resilient). + call_purpose: CallPurpose | None = None + if telemetry is not None and telemetry.call_purpose: + try: + call_purpose = CallPurpose(telemetry.call_purpose) + except ValueError: + logger.debug( + "Unknown LLMTelemetryContext.call_purpose=%r; emitting without", + telemetry.call_purpose, + ) + + attempt = plan.attempt if plan is not None else 1 + retry_attempts = plan.retry_attempts if plan is not None else 1 + was_fallback = plan.is_fallback if plan is not None else False + + emit( + LLMCallCompletedEvent( + workspace_name=(telemetry.workspace_name if telemetry else None), + call_purpose=call_purpose, + parent_category=(telemetry.parent_category if telemetry else None), + transport=provider, + provider_label=_infer_provider_label(provider, model, plan), + model=model, + effective_max_output_tokens=max_tokens, + provider_input_tokens=(result.input_tokens if result else 0), + provider_output_tokens=(result.output_tokens if result else 0), + cache_read_tokens=(result.cache_read_input_tokens if result else 0), + cache_creation_tokens=( + result.cache_creation_input_tokens if result else 0 + ), + finish_reason=(result.finish_reason if result else None), + outcome=outcome, + is_final_attempt=(attempt >= retry_attempts), + error_class=(type(error).__name__ if error else None), + attempt=attempt, + retry_attempts=retry_attempts, + was_fallback=was_fallback, + duration_ms=duration_ms, + has_tools=has_tools, + tool_call_count=(len(result.tool_calls) if result else 0), + was_stream=was_stream, + run_id=(telemetry.run_id if telemetry else None), + iteration=(telemetry.iteration if telemetry else None), + ) + ) + except Exception: # pragma: no cover - telemetry must not raise + logger.debug("Failed to emit LLMCallCompletedEvent", exc_info=True) + + +def _infer_provider_label( + _transport: ModelTransport, model: str, plan: AttemptPlan | None +) -> str | None: + """Best-effort vendor inference for relay setups. + + When the model name carries a vendor prefix (OpenRouter convention: + "anthropic/claude-..." routed through the openai transport), surface that + as the provider label so analytics can distinguish "openai-the-vendor" + from "openai-the-transport-pointing-at-openrouter". + + `_transport` is currently unused but kept on the signature so callers stay + explicit about which transport produced the call — future inference rules + (e.g. anthropic-direct vs anthropic-via-relay) may need it. + """ + if "/" in model: + return model.split("/", 1)[0] + # Defensive getattr — selected_config may be a stub in tests or a config + # without an explicit base_url. Either way the inference is best-effort. + base_url = ( + getattr(plan.selected_config, "base_url", None) if plan is not None else None + ) + if base_url and "openrouter" in base_url.lower(): + return "openrouter" + return None + + def completion_result_to_response( result: BackendCompletionResult, ) -> HonchoLLMCallResponse[Any]: @@ -92,6 +208,8 @@ async def honcho_llm_call_inner( tool_choice: str | dict[str, Any] | None = None, messages: list[dict[str, Any]] | None = None, selected_config: ModelConfig | None = None, + plan: AttemptPlan | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> HonchoLLMCallResponse[M]: ... @@ -114,6 +232,8 @@ async def honcho_llm_call_inner( tool_choice: str | dict[str, Any] | None = None, messages: list[dict[str, Any]] | None = None, selected_config: ModelConfig | None = None, + plan: AttemptPlan | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> HonchoLLMCallResponse[str]: ... @@ -136,6 +256,8 @@ async def honcho_llm_call_inner( tool_choice: str | dict[str, Any] | None = None, messages: list[dict[str, Any]] | None = None, selected_config: ModelConfig | None = None, + plan: AttemptPlan | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> AsyncIterator[HonchoLLMCallStreamChunk]: ... @@ -157,11 +279,22 @@ async def honcho_llm_call_inner( tool_choice: str | dict[str, Any] | None = None, messages: list[dict[str, Any]] | None = None, selected_config: ModelConfig | None = None, + plan: AttemptPlan | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk]: """One backend call. No retry, no fallback, no tool loop. The outer src/llm/api.py `honcho_llm_call` handles retry + fallback + tool orchestration on top of this. + + Emits one LLMCallCompletedEvent per call. On the stream path, setup + runs inside the awaited coroutine (so it sits inside any outer retry + wrapper) and emits its own event on failure; the wrapping generator + emits a second event from its finally block after drain completes or + raises. `was_stream` is True for streamed calls. Token counts are + zero on the stream path because provider token totals aren't surfaced + post-stream at this layer; aggregate envelopes (DialecticCompletedEvent + etc.) carry the accurate totals. """ client = client_override or CLIENTS.get(provider) if client is None: @@ -187,8 +320,18 @@ async def honcho_llm_call_inner( call_extras: dict[str, Any] = {"json_mode": json_mode, "verbosity": verbosity} if stream: - - async def _stream() -> AsyncIterator[HonchoLLMCallStreamChunk]: + # Stream path: setup must run inside the awaited coroutine so it + # sits inside the outer retry wrapper (tool_loop.stream_final_response + # wraps `await honcho_llm_call_inner(stream=True)` with tenacity). + # If we deferred `execute_stream` into the generator body, a transient + # setup failure (rate-limit, auth, network) would surface at first + # iteration — outside retry — and crash the request. + # + # Drain failures stay unretried by design (chunks may have already + # been sent to the client) and report via the wrapper's finally. + # Token counts are 0 on this path; aggregate envelopes carry totals. + stream_start = time.perf_counter() + try: stream_iter = await execute_stream( backend, effective_config, @@ -200,23 +343,80 @@ async def honcho_llm_call_inner( cache_policy=effective_config.cache_policy, extra_params=call_extras, ) - async for chunk in stream_iter: - yield stream_chunk_to_response_chunk(chunk) + except BaseException as exc: + _emit_llm_call_completed( + plan=plan, + telemetry=telemetry, + provider=provider, + model=model, + max_tokens=max_tokens, + duration_ms=(time.perf_counter() - stream_start) * 1000, + has_tools=bool(tools), + was_stream=True, + outcome=_outcome_from_error(exc), + result=None, + error=exc, + ) + raise - return _stream() + async def _wrap_stream() -> AsyncIterator[HonchoLLMCallStreamChunk]: + stream_error: BaseException | None = None + try: + async for chunk in stream_iter: + yield stream_chunk_to_response_chunk(chunk) + except BaseException as exc: + stream_error = exc + raise + finally: + _emit_llm_call_completed( + plan=plan, + telemetry=telemetry, + provider=provider, + model=model, + max_tokens=max_tokens, + duration_ms=(time.perf_counter() - stream_start) * 1000, + has_tools=bool(tools), + was_stream=True, + outcome=_outcome_from_error(stream_error), + result=None, + error=stream_error, + ) - result = await execute_completion( - backend, - effective_config, - messages=messages, - max_tokens=max_tokens, - tools=tools, - tool_choice=tool_choice, - response_format=response_model, - cache_policy=effective_config.cache_policy, - extra_params=call_extras, - ) - return completion_result_to_response(result) + return _wrap_stream() + + start = time.perf_counter() + backend_result: BackendCompletionResult | None = None + error: BaseException | None = None + try: + backend_result = await execute_completion( + backend, + effective_config, + messages=messages, + max_tokens=max_tokens, + tools=tools, + tool_choice=tool_choice, + response_format=response_model, + cache_policy=effective_config.cache_policy, + extra_params=call_extras, + ) + return completion_result_to_response(backend_result) + except BaseException as exc: + error = exc + raise + finally: + _emit_llm_call_completed( + plan=plan, + telemetry=telemetry, + provider=provider, + model=model, + max_tokens=max_tokens, + duration_ms=(time.perf_counter() - start) * 1000, + has_tools=bool(tools), + was_stream=False, + outcome=_outcome_from_error(error), + result=backend_result, + error=error, + ) __all__ = [ diff --git a/src/llm/runtime.py b/src/llm/runtime.py index 2c29f397..ae551378 100644 --- a/src/llm/runtime.py +++ b/src/llm/runtime.py @@ -75,6 +75,9 @@ class AttemptPlan: thinking_budget_tokens: int | None reasoning_effort: ReasoningEffortType selected_config: ModelConfig + attempt: int + retry_attempts: int + is_fallback: bool def resolve_runtime_model_config( @@ -166,6 +169,9 @@ def plan_attempt( thinking_budget_tokens=attempt_thinking_budget, reasoning_effort=attempt_reasoning_effort, selected_config=selected, + attempt=attempt, + retry_attempts=retry_attempts, + is_fallback=not is_primary, ) diff --git a/src/llm/tool_loop.py b/src/llm/tool_loop.py index 2db87e9a..734af8b3 100644 --- a/src/llm/tool_loop.py +++ b/src/llm/tool_loop.py @@ -11,16 +11,24 @@ from __future__ import annotations +import dataclasses +import functools import logging -from collections.abc import AsyncIterator, Callable -from typing import Any +from collections.abc import AsyncIterator, Awaitable, Callable +from typing import Any, ParamSpec, TypeVar from pydantic import BaseModel from tenacity import retry, stop_after_attempt, wait_exponential from src.config import ModelTransport from src.exceptions import ValidationException -from src.utils.types import set_current_iteration +from src.utils.types import ( + get_last_tool_metadata, + iteration_scope, + set_current_iteration, + set_current_tool_call_seq, + set_last_tool_metadata, +) from .executor import honcho_llm_call_inner from .registry import history_adapter_for_provider @@ -34,10 +42,106 @@ from .types import ( HonchoLLMCallStreamChunk, IterationCallback, IterationData, + LLMTelemetryContext, StreamingResponseWithMetadata, VerbosityType, ) +_P = ParamSpec("_P") +_R = TypeVar("_R") + + +def _with_iteration_scope( + fn: Callable[_P, Awaitable[_R]], +) -> Callable[_P, Awaitable[_R]]: + """Wrap an async tool-loop entry point in `iteration_scope()` so the + per-iteration ContextVars (iteration, tool_call_seq, provider id, last + tool metadata) are reset to their pre-call values on exit. Defensive + against subsequent loops in the same asyncio Task observing stale state. + """ + + @functools.wraps(fn) + async def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _R: + with iteration_scope(): + return await fn(*args, **kwargs) + + return wrapper + + +def _telemetry_for_iteration( + base: LLMTelemetryContext | None, iteration: int +) -> LLMTelemetryContext | None: + """Return a copy of `base` with `iteration` set, or None if no base. + + We always copy rather than mutate the caller-supplied context so callers + that pass the same context into multiple `honcho_llm_call` invocations + don't see drift across concurrent runs. + """ + if base is None: + return None + return LLMTelemetryContext( + workspace_name=base.workspace_name, + call_purpose=base.call_purpose, + parent_category=base.parent_category, + run_id=base.run_id, + iteration=iteration, + observer=base.observer, + observed=base.observed, + peer_name=base.peer_name, + agent_type=base.agent_type, + ) + + +def _emit_agent_iteration( + telemetry: LLMTelemetryContext | None, + iteration: int, + response: HonchoLLMCallResponse[Any], +) -> None: + """emit AgentIterationEvent after each per-iteration LLM response. + + Fired immediately after `response = await call_func()` in the per-iteration + loop AND after the max-iteration synthesis call. Emitted regardless of + whether the model requested tool calls — the no-tool terminating iteration + still counts as an iteration for cost calibration. + + Skipped when telemetry context is missing or lacks the required agent + identifiers (no agent → no agent.iteration event). + """ + if telemetry is None or not telemetry.run_id: + return + if not telemetry.parent_category or not telemetry.agent_type: + # Without agent_type / parent_category we can't fill the event's + # required fields. Skip rather than emit a half-populated event. + return + if not telemetry.workspace_name: + return + try: + # Local import: keeps src/llm/ free of a hard dependency on telemetry + # at import time so the LLM layer remains usable in unit tests that + # don't initialize the telemetry stack. + from src.telemetry.events import AgentIterationEvent, emit + + emit( + AgentIterationEvent( + run_id=telemetry.run_id, + parent_category=telemetry.parent_category, + agent_type=telemetry.agent_type, + workspace_name=telemetry.workspace_name, + observer=telemetry.observer, + observed=telemetry.observed, + peer_name=telemetry.peer_name, + iteration=iteration, + tool_calls=[tc["name"] for tc in response.tool_calls_made], + input_tokens=response.input_tokens, + output_tokens=response.output_tokens, + cache_read_tokens=response.cache_read_input_tokens or 0, + cache_creation_tokens=response.cache_creation_input_tokens or 0, + ) + ) + except Exception: # pragma: no cover - telemetry must not raise + logger.debug("Failed to emit AgentIterationEvent", exc_info=True) + + logger = logging.getLogger(__name__) # Bounds for max_tool_iterations to prevent runaway loops. @@ -98,6 +202,7 @@ async def stream_final_response( enable_retry: bool, retry_attempts: int, before_retry_callback: Callable[[Any], None], + telemetry: LLMTelemetryContext | None = None, ) -> AsyncIterator[HonchoLLMCallStreamChunk]: """Stream the final response after tool execution is complete. @@ -110,7 +215,24 @@ async def stream_final_response( streaming call against the same pinned model for transient errors. """ + # Bump the per-retry attempt index inside `_setup_stream`. The pinned + # `winning_plan.attempt` is frozen from before retries started; without + # this counter, every retried stream-setup emit reports the same attempt + # value — telemetry can't tell the retry sequence apart. + stream_attempt = 0 + async def _setup_stream() -> AsyncIterator[HonchoLLMCallStreamChunk]: + nonlocal stream_attempt + stream_attempt += 1 + # `dataclasses.replace` produces a per-attempt plan with the bumped + # `attempt` and the real `retry_attempts` budget so the executor's + # LLMCallCompletedEvent reports attempt=1/2/3 and is_final_attempt + # correctly across the retry sequence. + plan_for_attempt = dataclasses.replace( + winning_plan, + attempt=stream_attempt, + retry_attempts=retry_attempts, + ) return await honcho_llm_call_inner( winning_plan.provider, winning_plan.model, @@ -129,6 +251,8 @@ async def stream_final_response( tool_choice=None, messages=conversation_messages, selected_config=winning_plan.selected_config, + plan=plan_for_attempt, + telemetry=telemetry, ) if enable_retry: @@ -145,6 +269,7 @@ async def stream_final_response( yield chunk +@_with_iteration_scope async def execute_tool_loop( *, prompt: str, @@ -166,6 +291,7 @@ async def execute_tool_loop( before_retry_callback: Callable[[Any], None], stream_final: bool = False, iteration_callback: IterationCallback | None = None, + telemetry: LLMTelemetryContext | None = None, ) -> HonchoLLMCallResponse[Any] | StreamingResponseWithMetadata: """Run the iterative tool calling loop for agentic LLM interactions. @@ -179,7 +305,7 @@ async def execute_tool_loop( Final HonchoLLMCallResponse with accumulated token counts and tool call history, or a StreamingResponseWithMetadata if stream_final=True. """ - from .conversation import truncate_messages_to_fit + from .conversation import count_message_tokens, truncate_messages_to_fit if not MIN_TOOL_ITERATIONS <= max_tool_iterations <= MAX_TOOL_ITERATIONS: raise ValidationException( @@ -199,6 +325,14 @@ async def execute_tool_loop( total_cache_creation_tokens = 0 total_cache_read_tokens = 0 empty_response_retries = 0 + # Latch — set when any iteration's input exceeded `max_input_tokens`. + # Token-based rather than message-count-based: catches both "messages + # got dropped" and "couldn't drop the last unit but still over cap." + # Stamped onto the final response so + # RepresentationCompletedEvent.hit_input_token_cap and + # DialecticCompletedEvent.hit_input_token_cap reflect the cap hit + # (the toolless path tracks this in src/llm/api.py:325-340). + hit_input_token_cap = False # Track effective tool_choice — switches from "required"/"any" to "auto" after iter 1. effective_tool_choice = tool_choice @@ -208,6 +342,8 @@ async def execute_tool_loop( logger.debug(f"Tool execution iteration {iteration + 1}/{max_tool_iterations}") if max_input_tokens is not None: + if count_message_tokens(conversation_messages) > max_input_tokens: + hit_input_token_cap = True conversation_messages = truncate_messages_to_fit( conversation_messages, max_input_tokens ) @@ -215,6 +351,7 @@ async def execute_tool_loop( async def _call_with_messages( effective_tool_choice: str | dict[str, Any] | None = effective_tool_choice, conversation_messages: list[dict[str, Any]] = conversation_messages, + iteration_for_call: int = iteration + 1, ) -> HonchoLLMCallResponse[Any]: plan = get_attempt_plan() return await honcho_llm_call_inner( @@ -235,6 +372,8 @@ async def execute_tool_loop( tool_choice=effective_tool_choice, messages=conversation_messages, selected_config=plan.selected_config, + plan=plan, + telemetry=_telemetry_for_iteration(telemetry, iteration_for_call), ) if enable_retry: @@ -253,6 +392,11 @@ async def execute_tool_loop( total_cache_creation_tokens += response.cache_creation_input_tokens total_cache_read_tokens += response.cache_read_input_tokens + # emit one AgentIterationEvent per LLM response BEFORE the + # no-tool early return. The terminating iteration counts too — it has + # an empty tool_calls list and is essential for cost calibration. + _emit_agent_iteration(telemetry, iteration + 1, response) + if not response.tool_calls_made: logger.debug("No tool calls in response, finishing") @@ -293,6 +437,7 @@ async def execute_tool_loop( enable_retry=enable_retry, retry_attempts=retry_attempts, before_retry_callback=before_retry_callback, + telemetry=_telemetry_for_iteration(telemetry, iteration + 1), ) return StreamingResponseWithMetadata( stream=stream, @@ -303,6 +448,7 @@ async def execute_tool_loop( cache_read_input_tokens=total_cache_read_tokens, thinking_content=response.thinking_content, iterations=iteration + 1, + hit_input_token_cap=hit_input_token_cap, ) response.tool_calls_made = all_tool_calls @@ -311,6 +457,9 @@ async def execute_tool_loop( response.cache_creation_input_tokens = total_cache_creation_tokens response.cache_read_input_tokens = total_cache_read_tokens response.iterations = iteration + 1 + response.hit_input_token_cap = ( + response.hit_input_token_cap or hit_input_token_cap + ) return response current_provider = get_attempt_plan().provider @@ -328,15 +477,27 @@ async def execute_tool_loop( set_current_iteration(iteration + 1) tool_results: list[dict[str, Any]] = [] - for tool_call in response.tool_calls_made: + for seq, tool_call in enumerate(response.tool_calls_made): tool_name = tool_call["name"] tool_input = tool_call["input"] tool_id = tool_call.get("id", "") logger.debug(f"Executing tool: {tool_name}") + # the executor closure reads these from + # ContextVars to populate AgentToolCallCompletedEvent. Set BEFORE + # the executor call so two calls to the same tool in one iteration + # get distinct seq values. Reset last-tool metadata so we never + # observe stale state from a prior call. + set_current_tool_call_seq(seq, tool_id or None) + set_last_tool_metadata({}) + try: tool_result = await tool_executor(tool_name, tool_input) + # Stash ToolResult.metadata on all_tool_calls so + # specialist rollups can read created/deleted observation + # counts without round-tripping through the event store. + tool_result_metadata = get_last_tool_metadata() tool_results.append( { "tool_id": tool_id, @@ -349,6 +510,7 @@ async def execute_tool_loop( "tool_name": tool_name, "tool_input": tool_input, "tool_result": tool_result, + "tool_result_metadata": tool_result_metadata, } ) except Exception as e: @@ -391,6 +553,10 @@ async def execute_tool_loop( f"Tool execution loop reached max iterations ({max_tool_iterations})" ) + # The max-iteration synthesis call gets iteration N+1 in telemetry so 's + # AgentIterationEvent and this LLMCallCompletedEvent line up sequentially. + synthesis_iteration = iteration + 1 + synthesis_prompt = ( "You have reached the maximum number of tool calls. " "Based on all the information you have gathered, provide your final response now. " @@ -401,6 +567,8 @@ async def execute_tool_loop( # Truncate again — the per-iteration truncate ran before the last tool # call, so appending synthesis_prompt could nudge us back over the cap. if max_input_tokens is not None: + if count_message_tokens(conversation_messages) > max_input_tokens: + hit_input_token_cap = True conversation_messages = truncate_messages_to_fit( conversation_messages, max_input_tokens ) @@ -422,6 +590,7 @@ async def execute_tool_loop( enable_retry=enable_retry, retry_attempts=retry_attempts, before_retry_callback=before_retry_callback, + telemetry=_telemetry_for_iteration(telemetry, synthesis_iteration), ) return StreamingResponseWithMetadata( stream=stream, @@ -432,6 +601,7 @@ async def execute_tool_loop( cache_read_input_tokens=total_cache_read_tokens, thinking_content=None, iterations=iteration + 1, + hit_input_token_cap=hit_input_token_cap, ) current_attempt.set(1) @@ -456,6 +626,8 @@ async def execute_tool_loop( tool_choice=None, messages=conversation_messages, selected_config=plan.selected_config, + plan=plan, + telemetry=_telemetry_for_iteration(telemetry, synthesis_iteration), ) if enable_retry: @@ -468,6 +640,16 @@ async def execute_tool_loop( final_call_func = _final_call final_response = await final_call_func() + + # emit the synthesis-call iteration event BEFORE merging cumulative + # totals onto final_response below — otherwise the event's per-iteration + # token counts would double-count the running totals. + _emit_agent_iteration( + _telemetry_for_iteration(telemetry, synthesis_iteration), + synthesis_iteration, + final_response, + ) + final_response.tool_calls_made = all_tool_calls final_response.iterations = iteration + 1 final_response.input_tokens = total_input_tokens + final_response.input_tokens @@ -478,6 +660,9 @@ async def execute_tool_loop( final_response.cache_read_input_tokens = ( total_cache_read_tokens + final_response.cache_read_input_tokens ) + final_response.hit_input_token_cap = ( + final_response.hit_input_token_cap or hit_input_token_cap + ) return final_response diff --git a/src/llm/types.py b/src/llm/types.py index 7af5372d..a81e6e43 100644 --- a/src/llm/types.py +++ b/src/llm/types.py @@ -45,6 +45,38 @@ class IterationData: """Tokens written to cache in this iteration.""" +@dataclass +class LLMTelemetryContext: + """Context threaded through honcho_llm_call → honcho_llm_call_inner so the + LLMCallCompletedEvent emitter (and AgentIterationEvent emitter) + can attribute calls to the right workspace / agent / iteration without + re-deriving any of it from ambient state. + + Iteration is mutable: tool_loop updates this field before each inner call. + NOT read from set_current_iteration ContextVar — that fires after the LLM + call returns, so reading it from the executor would yield stale values. + """ + + workspace_name: str | None = None + # call_purpose carries the same string as src.telemetry.events.llm.CallPurpose values. + # Stored as str rather than importing the enum here to keep src/llm/ free of + # telemetry imports — the emitter validates against the enum. + call_purpose: str | None = None + parent_category: str | None = None + run_id: str | None = None + iteration: int | None = None + # Optional peer context (dream agents pass observer/observed; dialectic + # passes peer_name). Kept here so AgentIterationEvent can populate + # them without a separate threading path. + observer: str | None = None + observed: str | None = None + peer_name: str | None = None + # Tool-related context: agent_type is the human-readable identifier of the + # agent — dialectic/deduction/induction. Used by agent iteration + # event and tool call event. + agent_type: str | None = None + + IterationCallback = Callable[[IterationData], None] @@ -71,6 +103,12 @@ class HonchoLLMCallResponse(BaseModel, Generic[T]): thinking_blocks: list[dict[str, Any]] = Field(default_factory=list) # OpenRouter reasoning_details for Gemini models — must be preserved across turns. reasoning_details: list[dict[str, Any]] = Field(default_factory=list) + # True when the original input exceeded `max_input_tokens` — covers + # both "messages were dropped" and "couldn't drop the last unit and + # remaining tokens still exceeded the cap" (the deriver's prompt-only + # case). Maps 1:1 to `RepresentationCompletedEvent.hit_input_token_cap` + # and `DialecticCompletedEvent.hit_input_token_cap`. + hit_input_token_cap: bool = False class HonchoLLMCallStreamChunk(BaseModel): @@ -87,6 +125,15 @@ class StreamingResponseWithMetadata: Lets callers read tool_calls_made / token counts / thinking_content from the tool-execution phase while still iterating the final streamed answer. + + `output_tokens` is updated AS THE STREAM DRAINS — `__aiter__` wraps the + underlying iterator and accumulates the latest non-None `output_tokens` + value reported by chunk usage. Providers like OpenAI (with + `stream_options.include_usage`) and Anthropic emit a final usage chunk + with the cumulative count, so the post-drain `output_tokens` value + reflects tool-loop output + final-stream output. Callers that read + `output_tokens` AFTER fully iterating the stream get the true total; + callers that read it before drain see only the tool-loop portion. """ _stream: AsyncIterator[HonchoLLMCallStreamChunk] @@ -97,6 +144,7 @@ class StreamingResponseWithMetadata: cache_read_input_tokens: int thinking_content: str | None iterations: int + hit_input_token_cap: bool def __init__( self, @@ -108,6 +156,7 @@ class StreamingResponseWithMetadata: cache_read_input_tokens: int, thinking_content: str | None = None, iterations: int = 0, + hit_input_token_cap: bool = False, ): self._stream = stream self.tool_calls_made = tool_calls_made @@ -117,9 +166,31 @@ class StreamingResponseWithMetadata: self.cache_read_input_tokens = cache_read_input_tokens self.thinking_content = thinking_content self.iterations = iterations + self.hit_input_token_cap = hit_input_token_cap def __aiter__(self) -> AsyncIterator[HonchoLLMCallStreamChunk]: - return self._stream.__aiter__() + # Wrap the underlying iterator to capture final-stream output_tokens + # from chunks as they arrive. Providers emit a usage chunk at end-of- + # stream with the cumulative output_tokens count; we fold it into + # self.output_tokens (which carries the tool-loop running total at + # construction) so the post-drain value reflects the true cost. + return self._iterate_with_usage_capture() + + async def _iterate_with_usage_capture( + self, + ) -> AsyncIterator[HonchoLLMCallStreamChunk]: + final_stream_output_tokens = 0 + async for chunk in self._stream: + if chunk.output_tokens is not None: + # Take the LATEST value, not the sum — providers report + # the cumulative usage in the final chunk, not deltas. + final_stream_output_tokens = chunk.output_tokens + yield chunk + # Stream drained — fold the final-stream output tokens into the + # tool-loop totals so DialecticCompletedEvent / downstream readers + # see the true cost. + if final_stream_output_tokens > 0: + self.output_tokens += final_stream_output_tokens async def __anext__(self) -> HonchoLLMCallStreamChunk: return await self._stream.__anext__() @@ -130,6 +201,7 @@ __all__ = [ "HonchoLLMCallStreamChunk", "IterationCallback", "IterationData", + "LLMTelemetryContext", "ProviderClient", "ReasoningEffortType", "StreamingResponseWithMetadata", diff --git a/src/main.py b/src/main.py index 02f377a3..d08d1164 100644 --- a/src/main.py +++ b/src/main.py @@ -15,6 +15,7 @@ from pydantic import ValidationError from sentry_sdk.integrations.fastapi import FastApiIntegration from sentry_sdk.integrations.starlette import StarletteIntegration +from src._version import HONCHO_VERSION from src.cache.client import close_cache, init_cache from src.config import settings from src.db import engine, request_context @@ -161,7 +162,7 @@ app = FastAPI( title="Honcho API", summary="The Identity Layer for the Agentic World", description="""Honcho is a platform for giving agents user-centric memory and social cognition.""", - version="3.0.6", + version=HONCHO_VERSION, contact={ "name": "Plastic Labs", "url": "https://honcho.dev", diff --git a/src/reconciler/sync_vectors.py b/src/reconciler/sync_vectors.py index 34a2fab4..0dada25d 100644 --- a/src/reconciler/sync_vectors.py +++ b/src/reconciler/sync_vectors.py @@ -22,6 +22,8 @@ from src.config import settings from src.dependencies import tracked_db from src.embedding_client import embedding_client from src.exceptions import VectorStoreError +from src.telemetry.events import EmbeddingCallPurpose +from src.utils.types import embedding_call_purpose from src.vector_store import VectorRecord, VectorStore, get_external_vector_store logger = logging.getLogger(__name__) @@ -210,7 +212,11 @@ async def _sync_documents( if docs_needing_embed: try: contents = [doc.content for doc in docs_needing_embed] - new_embeddings = await embedding_client.simple_batch_embed(contents) + with embedding_call_purpose( + EmbeddingCallPurpose.VECTOR_SYNC.value, + parent_category="reconciliation", + ): + new_embeddings = await embedding_client.simple_batch_embed(contents) if len(new_embeddings) != len(docs_needing_embed): logger.warning( @@ -332,7 +338,11 @@ async def _sync_message_embeddings( if embs_needing_embed: try: contents = [emb.content for emb in embs_needing_embed] - new_embeddings = await embedding_client.simple_batch_embed(contents) + with embedding_call_purpose( + EmbeddingCallPurpose.VECTOR_SYNC.value, + parent_category="reconciliation", + ): + new_embeddings = await embedding_client.simple_batch_embed(contents) if len(new_embeddings) != len(embs_needing_embed): logger.warning( diff --git a/src/routers/conclusions.py b/src/routers/conclusions.py index 6ba2ed67..20aadb75 100644 --- a/src/routers/conclusions.py +++ b/src/routers/conclusions.py @@ -9,6 +9,8 @@ from src import crud, schemas from src.dependencies import db from src.exceptions import ResourceNotFoundException, ValidationException from src.security import require_auth +from src.telemetry.events import EmbeddingCallPurpose +from src.utils.types import embedding_call_purpose logger = logging.getLogger(__name__) @@ -111,16 +113,21 @@ async def query_conclusions( "observer and observed must be specified for semantic search" ) - documents = await crud.query_documents( - db, + with embedding_call_purpose( + EmbeddingCallPurpose.GENERIC_DOCUMENT_SEARCH.value, workspace_name=workspace_id, - query=body.query, - observer=observer, - observed=observed, - filters=body.filters, - max_distance=body.distance, - top_k=body.top_k, - ) + parent_category="api", + ): + documents = await crud.query_documents( + db, + workspace_name=workspace_id, + query=body.query, + observer=observer, + observed=observed, + filters=body.filters, + max_distance=body.distance, + top_k=body.top_k, + ) return [schemas.Conclusion.model_validate(doc) for doc in documents] diff --git a/src/routers/messages.py b/src/routers/messages.py index 3acdc1e5..917ca713 100644 --- a/src/routers/messages.py +++ b/src/routers/messages.py @@ -23,6 +23,7 @@ from src.deriver import enqueue from src.exceptions import FileTooLargeError, ResourceNotFoundException from src.security import require_auth from src.telemetry import prometheus_metrics +from src.telemetry.events import FileUploadedEvent, MessageCreatedEvent, emit from src.utils.files import process_file_uploads_for_messages logger = logging.getLogger(__name__) @@ -107,6 +108,17 @@ async def create_messages_for_session( workspace_name=workspace_id, ) + emit( + MessageCreatedEvent( + workspace_name=workspace_id, + session_name=session_id, + message_count=len(created_messages), + total_tokens=sum(message.token_count for message in created_messages), + source="api", + last_message_id=created_messages[-1].public_id, + ) + ) + # Enqueue for processing (existing logic) payloads = [ { @@ -206,6 +218,35 @@ async def create_messages_with_file( workspace_name=workspace_id, ) + # An empty extracted file (no chunks) leaves both lists empty. Skip the + # telemetry in that case rather than indexing into []. + if all_message_data and created_messages: + file_metadata = all_message_data[0]["file_metadata"] + total_tokens = sum(message.token_count for message in created_messages) + emit( + FileUploadedEvent( + workspace_name=workspace_id, + session_name=session_id, + peer_name=form_data.peer_id, + file_id=str(file_metadata["file_id"]), + filename=file.filename, + content_type=file.content_type, + file_size_bytes=file.size, + message_count=len(created_messages), + total_tokens=total_tokens, + ) + ) + emit( + MessageCreatedEvent( + workspace_name=workspace_id, + session_name=session_id, + message_count=len(created_messages), + total_tokens=total_tokens, + source="file_upload", + last_message_id=created_messages[-1].public_id, + ) + ) + return created_messages diff --git a/src/routers/peers.py b/src/routers/peers.py index fb765737..99014455 100644 --- a/src/routers/peers.py +++ b/src/routers/peers.py @@ -1,6 +1,8 @@ import json import logging from collections.abc import AsyncIterator +from contextlib import suppress +from time import perf_counter from fastapi import APIRouter, Body, Depends, Path, Query, Response from fastapi.responses import StreamingResponse @@ -12,10 +14,13 @@ from src import crud, schemas from src.config import settings from src.dependencies import db, tracked_db from src.dialectic.chat import agentic_chat, agentic_chat_stream +from src.embedding_client import embedding_client from src.exceptions import AuthenticationException, ResourceNotFoundException from src.security import JWTParams, require_auth from src.telemetry import prometheus_metrics +from src.telemetry.events import EmbeddingCallPurpose, GetContextEvent, emit from src.utils.search import search +from src.utils.types import embedding_call_purpose logger = logging.getLogger(__name__) @@ -252,6 +257,18 @@ async def get_representation( If no target is provided, we get the omniscient Honcho Representation of the Peer. """ try: + embedding: list[float] | None = None + if options.search_query: + with ( + suppress(Exception), + embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MEMORY.value, + workspace_name=workspace_id, + parent_category="api", + ), + ): + embedding = await embedding_client.embed(options.search_query) + # If no target specified, get global representation (omniscient Honcho perspective) representation = await crud.get_working_representation( workspace_id, @@ -259,6 +276,7 @@ async def get_representation( observed=options.target if options.target is not None else peer_id, session_name=options.session_id, include_semantic_query=options.search_query, + embedding=embedding, semantic_search_top_k=options.search_top_k, semantic_search_max_distance=options.search_max_distance, include_most_derived=options.include_most_frequent @@ -267,6 +285,7 @@ async def get_representation( max_observations=options.max_conclusions if options.max_conclusions is not None else settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS, + parent_category="api", ) return schemas.RepresentationResponse( representation=representation.format_as_markdown() @@ -399,8 +418,21 @@ async def get_peer_context( """ # If no target specified, get the peer's own context (self-observation) observed = target if target is not None else peer_id + context_started = perf_counter() try: + embedding: list[float] | None = None + if search_query: + with ( + suppress(Exception), + embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MEMORY.value, + workspace_name=workspace_id, + parent_category="api", + ), + ): + embedding = await embedding_client.embed(search_query) + # Get the working representation representation = await crud.get_working_representation( workspace_id, @@ -408,12 +440,14 @@ async def get_peer_context( observed=observed, session_name=None, # Peer context is global, not session-scoped include_semantic_query=search_query, + embedding=embedding, semantic_search_top_k=search_top_k, semantic_search_max_distance=search_max_distance, include_most_derived=include_most_frequent, max_observations=max_conclusions if max_conclusions is not None else settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS, + parent_category="api", ) # Get the peer card @@ -421,12 +455,29 @@ async def get_peer_context( db, workspace_id, observer=peer_id, observed=observed ) - return schemas.PeerContext( + response = schemas.PeerContext( peer_id=peer_id, target_id=observed, representation=representation.format_as_markdown(), peer_card=peer_card, ) + emit( + GetContextEvent( + workspace_name=workspace_id, + context_scope="peer", + peer_name=peer_id, + target_name=observed, + has_representation=bool(response.representation), + has_peer_card=peer_card is not None, + search_query_provided=search_query is not None, + search_top_k=search_top_k, + search_max_distance=search_max_distance, + include_most_frequent=include_most_frequent, + max_conclusions=max_conclusions, + total_duration_ms=(perf_counter() - context_started) * 1000, + ) + ) + return response except ValueError as e: logger.warning(f"Failed to get context for peer {peer_id}: {str(e)}") raise ResourceNotFoundException("Peer not found") from e diff --git a/src/routers/sessions.py b/src/routers/sessions.py index 9071aef1..98f669df 100644 --- a/src/routers/sessions.py +++ b/src/routers/sessions.py @@ -1,5 +1,6 @@ import logging from contextlib import suppress +from time import perf_counter from fastapi import APIRouter, Body, Depends, Path, Query, Response from fastapi_pagination import Page @@ -18,10 +19,12 @@ from src.exceptions import ( ValidationException, ) from src.security import JWTParams, require_auth +from src.telemetry.events import EmbeddingCallPurpose, GetContextEvent, emit from src.utils import summarizer from src.utils.representation import Representation from src.utils.search import search from src.utils.tokens import estimate_tokens +from src.utils.types import embedding_call_purpose logger = logging.getLogger(__name__) @@ -78,6 +81,8 @@ async def _get_working_representation_task( max_observations=max_observations if max_observations is not None else config.settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS, + parent_category="api", + embedding_purpose=EmbeddingCallPurpose.SESSION_CONTEXT_SEARCH, ) @@ -669,6 +674,7 @@ async def get_session_context( token_limit = ( tokens if tokens is not None else config.settings.GET_CONTEXT_MAX_TOKENS ) + context_started = perf_counter() if peer_perspective and not peer_target: raise ValidationException( @@ -680,11 +686,30 @@ async def get_session_context( summary, messages = await _get_session_context_task( db, workspace_id, session_id, token_limit, include_summary ) - return schemas.SessionContext( + response = schemas.SessionContext( name=session_id, messages=messages, summary=summary, ) + emit( + GetContextEvent( + workspace_name=workspace_id, + context_scope="session", + session_name=session_id, + tokens_requested=tokens, + message_count=len(messages), + has_summary=summary is not None, + search_query_provided=search_query is not None, + search_top_k=search_top_k, + search_max_distance=search_max_distance, + include_most_frequent=include_most_frequent, + max_conclusions=max_conclusions, + include_summary=include_summary, + limit_to_session=limit_to_session, + total_duration_ms=(perf_counter() - context_started) * 1000, + ) + ) + return response observer = peer_perspective or peer_target observed = peer_target @@ -692,7 +717,14 @@ async def get_session_context( # Pre-compute embedding outside the DB session (best-effort) embedding: list[float] | None = None if search_query: - with suppress(Exception): + with ( + suppress(Exception), + embedding_call_purpose( + EmbeddingCallPurpose.SESSION_CONTEXT_SEARCH.value, + workspace_name=workspace_id, + parent_category="api", + ), + ): embedding = await embedding_client.embed(search_query) # Sequential calls on shared DB session @@ -731,13 +763,37 @@ async def get_session_context( db, workspace_id, session_id, messages_start_id, messages_budget ) - return schemas.SessionContext( + response = schemas.SessionContext( name=session_id, messages=messages, summary=summary, peer_representation=representation.format_as_markdown(), peer_card=card, ) + emit( + GetContextEvent( + workspace_name=workspace_id, + context_scope="session", + session_name=session_id, + peer_name=observer, + target_name=observed, + tokens_requested=tokens, + message_count=len(messages), + has_summary=summary is not None, + has_representation=bool(response.peer_representation), + has_peer_card=card is not None, + search_query_provided=search_query is not None, + search_top_k=search_top_k, + search_max_distance=search_max_distance, + include_most_frequent=include_most_frequent, + max_conclusions=max_conclusions, + include_summary=include_summary, + limit_to_session=limit_to_session, + peer_perspective_provided=peer_perspective is not None, + total_duration_ms=(perf_counter() - context_started) * 1000, + ) + ) + return response @router.get( diff --git a/src/routers/workspaces.py b/src/routers/workspaces.py index 2ce7cdda..6671afbc 100644 --- a/src/routers/workspaces.py +++ b/src/routers/workspaces.py @@ -228,6 +228,13 @@ async def schedule_dream( observed=observed, dream_type=dream_type, session_name=request.session_id, + # Manual route — explicit sentinels for the DreamRunEvent + # scheduling-context fields. Auto-schedule threads concrete + # threshold/delay reasons (see src/dreamer/dream_scheduler.py); + # without these, manual dreams arrive with both null and break + # analytics joins on `trigger_reason`. + trigger_reason="manual", + delay_reason="immediate", ) logger.info( diff --git a/src/telemetry/emitter.py b/src/telemetry/emitter.py index 56a70e7c..2413c7c2 100644 --- a/src/telemetry/emitter.py +++ b/src/telemetry/emitter.py @@ -13,18 +13,64 @@ import contextlib import json import logging from collections import deque -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any import httpx from cloudevents.conversion import to_json # pyright: ignore[reportUnknownVariableType] from cloudevents.http import CloudEvent +from src._version import HONCHO_VERSION + if TYPE_CHECKING: from src.telemetry.events.base import BaseEvent logger = logging.getLogger(__name__) +def _should_sample( + event: "BaseEvent", rate: object, *, event_id: str | None = None +) -> bool: + """Trace-coherent deterministic sampler for high-volume events. + + `rate` is typed as `object` (rather than `float`) because the caller + reads it straight from `settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE`, + which in tests gets MagicMock'd. A MagicMock comparison against 1.0 + raises TypeError, so we validate at the boundary and fall back to + passthrough on anything non-numeric. + + When the event carries a `run_id`, sampling decisions hash on that id — + so every event in an agent run either passes or fails the sampler, and + join queries downstream don't see half-traces. Events without `run_id` + (summarizer, deriver — non-agentic call sites) sample independently per + event using the event's deterministic id. Callers that have already + computed `event.generate_id()` can pass it as `event_id` to avoid the + redundant sha256 hash. + """ + if not isinstance(rate, int | float): + return True + rate_f = float(rate) + if rate_f >= 1.0: + return True + if rate_f <= 0.0: + return False + run_id = getattr(event, "run_id", None) + if isinstance(run_id, str) and run_id: + key = run_id + elif event_id is not None: + key = event_id + else: + key = event.generate_id() + # Stable hash → 0..9999 → compare against rate * 10000. + bucket = int.from_bytes(_stable_hash(key)[:4], "big") % 10_000 + return bucket < int(rate_f * 10_000) + + +def _stable_hash(value: str) -> bytes: + import hashlib + + return hashlib.sha256(value.encode("utf-8")).digest() + + class TelemetryEmitter: """Buffered, async CloudEvents emitter with retry logic. @@ -59,6 +105,8 @@ class TelemetryEmitter: _client: httpx.AsyncClient | None _running: bool _lock: asyncio.Lock + _capacity_warning_active: bool + _pending_flush_tasks: set[asyncio.Task[None]] def __init__( self, @@ -97,6 +145,8 @@ class TelemetryEmitter: self._client = None self._running = False self._lock = asyncio.Lock() + self._capacity_warning_active = False + self._pending_flush_tasks = set() async def start(self) -> None: """Start the emitter background tasks. @@ -121,8 +171,14 @@ class TelemetryEmitter: async def shutdown(self) -> None: """Gracefully shutdown the emitter. - Stops the periodic flush task, flushes remaining events, - and closes the HTTP client. + Stops the periodic flush task, drains any in-flight threshold + flushes, flushes remaining events, and closes the HTTP client. + + Threshold flushes are spawned from emit() and pop their batch + under lock before releasing it for the HTTP send. If we don't + await those tasks first, the final flush() can see an empty + buffer and return while the in-flight task is still mid-send — + closing the HTTP client then orphans that batch. """ if not self.enabled: return @@ -135,6 +191,12 @@ class TelemetryEmitter: with contextlib.suppress(asyncio.CancelledError): await self._flush_task + # Drain in-flight threshold flushes. Snapshot the set first because + # the done-callback mutates it. Exceptions here mustn't block shutdown. + pending = list(self._pending_flush_tasks) + if pending: + await asyncio.gather(*pending, return_exceptions=True) + # Final flush of remaining events await self.flush() @@ -159,9 +221,45 @@ class TelemetryEmitter: return from src.config import settings + from src.telemetry.prometheus.metrics import prometheus_metrics - # Generate deterministic event ID - event_id = event.generate_id() + # High-volume events are subject to HIGH_VOLUME_SAMPLE_RATE. Aggregate + # envelopes (representation.completed, dialectic.completed, dream.run, + # etc.) declare _volume_class="ground_truth" and skip the sampler. + # Sampling is deterministic on run_id when available so an entire + # agentic trace is either fully kept or fully dropped — never a + # half-sampled run that breaks join queries downstream. + # + # Trade-off: at rate < 1.0, ground_truth aggregates still emit but + # their high-volume children get sampled out. Downstream JOIN ... ON + # run_id sees orphaned parents; aggregates carry totals so this is + # intentional, but per-call analytics rebuilt from the sampled + # children alone will undercount. See HIGH_VOLUME_SAMPLE_RATE + # docstring in src/config.py for the full implications. + # Lazy event_id generation. The sampler only needs it for high-volume + # events without a run_id (run_id events sample on run_id directly). + # For sampled-out children of an agent run, deferring saves a sha256 + # hash per event; for events that pass the sampler or skip it, we + # still only compute the id once and reuse it for the CloudEvent. + event_id: str | None = None + + if event.volume_class() == "high_volume": + run_id = getattr(event, "run_id", None) + has_run_id = isinstance(run_id, str) and bool(run_id) + if not has_run_id: + event_id = event.generate_id() + if not _should_sample( + event, + settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE, + event_id=event_id, + ): + prometheus_metrics.record_telemetry_event_sampled_out( + event_type=event.event_type() + ) + return + + if event_id is None: + event_id = event.generate_id() # Build source with namespace for tenant routing # Format: /honcho/{namespace}/{category} or /honcho/{category} @@ -179,28 +277,64 @@ class TelemetryEmitter: "dataschema": f"https://honcho.dev/schemas/{event.event_type()}/v{event.schema_version()}", } + # Build body and inject envelope-level identity. We do NOT mutate the event + # instance — tests and callers that observe the event after emit() see it + # unchanged. Only the serialized body that hits the wire carries the extras. + body: dict[str, Any] = event.model_dump(mode="json") + body["honcho_version"] = HONCHO_VERSION + + # Buffer-full check happens here because deque(maxlen=) silently evicts. + # Detect by length-before-append; if at capacity, the append will displace + # the oldest event — that's a drop. + will_drop_oldest = len(self._buffer) >= self.max_buffer_size + # Create CloudEvent - cloud_event = CloudEvent(attributes, event.model_dump(mode="json")) + cloud_event = CloudEvent(attributes, body) + + if will_drop_oldest: + prometheus_metrics.record_telemetry_event_dropped(reason="buffer_full") self._buffer.append(cloud_event) buffer_size = len(self._buffer) + prometheus_metrics.record_telemetry_event_emitted(event_type=event.event_type()) + prometheus_metrics.set_telemetry_buffer_size(size=buffer_size) logger.debug("Queued event %s (buffer size: %d)", event_id, buffer_size) - # Warning logs as buffer approaches max capacity + # Warning logs as buffer approaches max capacity. Edge-triggered so + # sustained backpressure doesn't spam thousands of WARN lines per + # second — exactly when log pipelines are most fragile. capacity_ratio = buffer_size / self.max_buffer_size if capacity_ratio >= 0.8: - logger.warning( - "Telemetry buffer at %.0f%% capacity (%d/%d events)", - capacity_ratio * 100, - buffer_size, - self.max_buffer_size, - ) + if not self._capacity_warning_active: + self._capacity_warning_active = True + logger.warning( + "Telemetry buffer at %.0f%% capacity (%d/%d events)", + capacity_ratio * 100, + buffer_size, + self.max_buffer_size, + ) + else: + self._capacity_warning_active = False logger.debug("Event added to emitter (buffer size: %d)", buffer_size) - # Threshold-based flush trigger + # Threshold-based flush trigger. emit() is sync — guard against the + # case where a future caller invokes it from outside an event loop; + # the periodic flush task will still pick up the buffered events. + # Track spawned tasks so shutdown() can await them before closing the + # HTTP client — otherwise an in-flight threshold flush (which pops its + # batch under lock, then sends without the lock) can be orphaned and + # lose its batch when the client closes underneath it. if buffer_size >= self.flush_threshold and self._running: logger.debug("Triggering flush (buffer size: %d)", buffer_size) - asyncio.create_task(self.flush()) + try: + asyncio.get_running_loop() + flush_task = asyncio.create_task(self.flush()) + self._pending_flush_tasks.add(flush_task) + flush_task.add_done_callback(self._pending_flush_tasks.discard) + except RuntimeError: + logger.debug( + "emit() called outside an event loop; deferring flush to periodic task" + ) async def flush(self) -> None: """Flush buffered events to the endpoint. @@ -208,31 +342,47 @@ class TelemetryEmitter: Sends events in batches up to batch_size. Uses exponential backoff retry on failure. Events are returned to the buffer on permanent failure. + + The lock is held only for buffer mutations (pop batch / restore on + failure) — never across the HTTP send. A failing endpoint can spend + tens of seconds in retry + backoff; keeping that out of the lock + lets concurrent flushers make progress on disjoint batches. """ if not self.enabled or not self._buffer or self._client is None: return - async with self._lock: - while self._buffer: - # Extract a batch + while True: + async with self._lock: + if not self._buffer: + return batch: list[CloudEvent] = [] while self._buffer and len(batch) < self.batch_size: batch.append(self._buffer.popleft()) - if not batch: - break + if not batch: + return - # Try to send the batch - success = await self._send_batch(batch) - if not success: - # Put events back at the front of the buffer - for event in reversed(batch): - self._buffer.appendleft(event) - logger.warning( - "Failed to send batch of %d events, returned to buffer", - len(batch), - ) - break + success = await self._send_batch(batch) + if success: + continue + + from src.telemetry.prometheus.metrics import prometheus_metrics + + async with self._lock: + # Put events back at the front of the buffer. If the buffer is + # already full, deque.appendleft silently evicts from the right + # — those events are lost. Count the eviction as send_failed. + for event in reversed(batch): + if len(self._buffer) >= self.max_buffer_size: + prometheus_metrics.record_telemetry_event_dropped( + reason="send_failed" + ) + self._buffer.appendleft(event) + logger.warning( + "Failed to send batch of %d events, returned to buffer", + len(batch), + ) + return async def _send_batch(self, batch: list[CloudEvent]) -> bool: """Send a batch of events to the endpoint with retry logic. diff --git a/src/telemetry/events/__init__.py b/src/telemetry/events/__init__.py index 697ecec8..3da745dd 100644 --- a/src/telemetry/events/__init__.py +++ b/src/telemetry/events/__init__.py @@ -24,6 +24,11 @@ Event Categories: - AgentToolPeerCardUpdatedEvent: Peer card updated by agent - AgentToolSummaryCreatedEvent: Summary created + api: User-facing API operations + - MessageCreatedEvent: Message batch created + - FileUploadedEvent: File converted into messages + - GetContextEvent: Context retrieved for a session or peer + deletion: Resource removal - DeletionCompletedEvent: Resource deletion completed (with cascade counts) @@ -55,11 +60,17 @@ import logging from src.telemetry.events.agent import ( AgentIterationEvent, + AgentToolCallCompletedEvent, AgentToolConclusionsCreatedEvent, AgentToolConclusionsDeletedEvent, AgentToolPeerCardUpdatedEvent, AgentToolSummaryCreatedEvent, ) +from src.telemetry.events.api import ( + FileUploadedEvent, + GetContextEvent, + MessageCreatedEvent, +) from src.telemetry.events.base import BaseEvent, generate_event_id from src.telemetry.events.deletion import DeletionCompletedEvent from src.telemetry.events.dialectic import DialecticCompletedEvent @@ -67,6 +78,12 @@ from src.telemetry.events.dream import ( DreamRunEvent, DreamSpecialistEvent, ) +from src.telemetry.events.llm import ( + CallPurpose, + EmbeddingCallCompletedEvent, + EmbeddingCallPurpose, + LLMCallCompletedEvent, +) from src.telemetry.events.reconciliation import ( CleanupStaleItemsCompletedEvent, SyncVectorsCompletedEvent, @@ -89,10 +106,20 @@ __all__ = [ "DialecticCompletedEvent", # Agent events "AgentIterationEvent", + "AgentToolCallCompletedEvent", "AgentToolConclusionsCreatedEvent", "AgentToolConclusionsDeletedEvent", "AgentToolPeerCardUpdatedEvent", "AgentToolSummaryCreatedEvent", + # API events + "MessageCreatedEvent", + "FileUploadedEvent", + "GetContextEvent", + # LLM events + "LLMCallCompletedEvent", + "CallPurpose", + "EmbeddingCallCompletedEvent", + "EmbeddingCallPurpose", # Reconciliation events "SyncVectorsCompletedEvent", "CleanupStaleItemsCompletedEvent", diff --git a/src/telemetry/events/agent.py b/src/telemetry/events/agent.py index fe7ff72b..df771709 100644 --- a/src/telemetry/events/agent.py +++ b/src/telemetry/events/agent.py @@ -29,11 +29,12 @@ class AgentIterationEvent(BaseEvent): """ _event_type: ClassVar[str] = "agent.iteration" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "agent" + _volume_class: ClassVar[str] = "high_volume" # Run identification - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") # Context parent_category: str = Field( @@ -82,11 +83,11 @@ class AgentToolConclusionsCreatedEvent(BaseEvent): """ _event_type: ClassVar[str] = "agent.tool.conclusions.created" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "agent" # Run identification - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") iteration: int = Field(..., description="Iteration number when this occurred") # Context @@ -121,11 +122,11 @@ class AgentToolConclusionsDeletedEvent(BaseEvent): """ _event_type: ClassVar[str] = "agent.tool.conclusions.deleted" - _schema_version: ClassVar[int] = 2 + _schema_version: ClassVar[int] = 3 _category: ClassVar[str] = "agent" # Run identification - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") iteration: int = Field(..., description="Iteration number when this occurred") # Context @@ -158,11 +159,11 @@ class AgentToolPeerCardUpdatedEvent(BaseEvent): """ _event_type: ClassVar[str] = "agent.tool.peer_card.updated" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "agent" # Run identification - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") iteration: int = Field(..., description="Iteration number when this occurred") # Context @@ -190,11 +191,11 @@ class AgentToolSummaryCreatedEvent(BaseEvent): """ _event_type: ClassVar[str] = "agent.tool.summary.created" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "agent" # Run identification (may be placeholder if not from an agentic loop) - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") iteration: int = Field(..., description="Iteration number when this occurred") # Context @@ -219,17 +220,140 @@ class AgentToolSummaryCreatedEvent(BaseEvent): # Token usage input_tokens: int = Field( - ..., description="Input tokens used for summary generation" + ..., + description=( + "Provider-side input tokens for the summary LLM call " + "(equivalent to HonchoLLMCallResponse.input_tokens). " + "keeps this field unchanged — adding a duplicate " + "`provider_input_tokens` would only churn analytics queries." + ), ) output_tokens: int = Field(..., description="Output tokens (summary token count)") + # ---- Additive fields ---- + # Breakdown of what *went into* the summary prompt. Lets calibration + # answer "how much of a summary call's cost was the previous-summary + # rollup vs. the new messages vs. the scaffold/instructions" without + # re-deriving from the message corpus. + previous_summary_tokens: int = Field( + default=0, + description=( + "Token count of the previous summary text fed back in as context. " + "0 when this is the first summary for the session." + ), + ) + message_tokens: int = Field( + default=0, + description=( + "Sum of `Message.token_count` across the messages being " + "summarized (excludes scaffold and previous_summary)." + ), + ) + prompt_scaffold_tokens: int = Field( + default=0, + description=( + "Estimated tokens for the static scaffold portion of the prompt " + "(from estimate_short/long_summary_prompt_tokens)." + ), + ) + def get_resource_id(self) -> str: """Resource ID includes run_id and iteration for uniqueness.""" return f"{self.run_id}:{self.iteration}:summary_created" +class AgentToolCallCompletedEvent(BaseEvent): + """generic tool-call event: fires once per tool invocation. + + Complements the four state-changer events (conclusions_created/deleted, + peer_card_updated, summary_created), which carry semantic information + about specific tools, with a lightweight per-call telemetry record that + covers every tool — including read-only tools (`search_memory`, + `get_recent_history`, etc.) that have no dedicated event today. + + Resource id includes `tool_call_seq` so the model can legitimately call + the same tool twice in one iteration (it does) without colliding event + ids — without seq, both calls would deterministically hash to the same + id and dedupe would drop one. + """ + + _event_type: ClassVar[str] = "agent.tool.call.completed" + _schema_version: ClassVar[int] = 1 + _category: ClassVar[str] = "agent" + _volume_class: ClassVar[str] = "high_volume" + + # Run identification + run_id: str = Field(..., description="Run id for correlation") + iteration: int = Field(..., description="Iteration number (1-indexed)") + tool_call_seq: int = Field( + ..., + description="0-indexed position within the iteration's tool batch. Disambiguates two calls to the same tool in one iteration.", + ) + provider_tool_call_id: str | None = Field( + default=None, + description="Provider-supplied tool call id (e.g. Anthropic's toolu_*) when available; lets analytics cross-reference provider logs", + ) + + # Context + parent_category: str = Field( + ..., description="Parent category: 'dream' or 'dialectic'" + ) + agent_type: str = Field( + ..., description="Agent type: 'deduction', 'induction', or 'dialectic'" + ) + workspace_name: str = Field(..., description="Workspace name") + + # What ran + tool_name: str = Field(..., description="Tool name as invoked") + duration_ms: float = Field(..., description="Wall-clock duration of the handler") + is_error: bool = Field(default=False, description="True if the handler raised") + + # Result shape + result_chars: int = Field( + ..., description="Length of the result string returned to the LLM" + ) + result_chars_before_truncation: int | None = Field( + default=None, + description="Original result size when the handler truncated; None when no truncation occurred. Pair with was_truncated for the delta.", + ) + result_tokens_estimate: int = Field( + default=0, + description="tiktoken-based size proxy for the result string; estimate only", + ) + was_truncated: bool = Field( + default=False, + description="True when the handler clamped the result to fit a size budget", + ) + + # Search-specific fields (None for non-search tools). Populated by search + # handlers via the ToolResult.metadata bridge. + query_tokens: int | None = Field( + default=None, description="tiktoken estimate of the search query text" + ) + top_k: int | None = Field( + default=None, description="Caller-supplied top_k for the search" + ) + results_count: int | None = Field( + default=None, description="Number of results returned by the search" + ) + used_embedding: bool | None = Field( + default=None, + description="True when the search ran a vector lookup (vs. metadata-only filter)", + ) + embedding_query_count: int = Field( + default=0, + description="Number of embedding API calls the handler made for this invocation", + ) + + def get_resource_id(self) -> str: + """{run_id}:{iteration}:{tool_call_seq} so duplicate tool calls within + one iteration produce distinct deterministic ids.""" + return f"{self.run_id}:{self.iteration}:{self.tool_call_seq}" + + __all__ = [ "AgentIterationEvent", + "AgentToolCallCompletedEvent", "AgentToolConclusionsCreatedEvent", "AgentToolConclusionsDeletedEvent", "AgentToolPeerCardUpdatedEvent", diff --git a/src/telemetry/events/api.py b/src/telemetry/events/api.py new file mode 100644 index 00000000..ebc25c5e --- /dev/null +++ b/src/telemetry/events/api.py @@ -0,0 +1,167 @@ +""" +API events for Honcho telemetry. + +These events track user-facing API operations +""" + +from typing import ClassVar, Literal + +from pydantic import Field + +from src.telemetry.events.base import BaseEvent + + +class MessageCreatedEvent(BaseEvent): + """Emitted when one or more messages are created. + + This is the canonical API event for counting created messages, including + messages created from file uploads. The resource_id keys on + `last_message_id` (the trailing message's public nanoid) so two batches + of the same size in the same session+source produce distinct event ids. + """ + + _event_type: ClassVar[str] = "message.created" + _schema_version: ClassVar[int] = 1 + _category: ClassVar[str] = "api" + + workspace_name: str = Field(..., description="Workspace name") + session_name: str = Field(..., description="Session name") + message_count: int = Field(..., description="Number of messages created") + total_tokens: int = Field(..., description="Total tokens across created messages") + source: Literal["api", "file_upload"] = Field( + default="api", description="Source of the created messages" + ) + last_message_id: str = Field( + ..., + description="public_id (nanoid) of the trailing message in the batch — used as the stable unique key for this emission", + ) + + def get_resource_id(self) -> str: + """Resource ID keys on the trailing message's public_id so two batches + of the same size in the same session+source produce distinct event ids. + """ + return ( + f"{self.workspace_name}:{self.session_name}:" + f"{self.source}:{self.last_message_id}" + ) + + +class FileUploadedEvent(BaseEvent): + """Emitted when an uploaded file is converted into messages. + + This captures file-side metadata. Message creation counts should use + MessageCreatedEvent to avoid double-counting file uploads. + """ + + _event_type: ClassVar[str] = "file.uploaded" + _schema_version: ClassVar[int] = 1 + _category: ClassVar[str] = "api" + + workspace_name: str = Field(..., description="Workspace name") + session_name: str = Field(..., description="Session name") + peer_name: str = Field(..., description="Peer that uploaded the file") + file_id: str = Field(..., description="Generated file identifier") + filename: str | None = Field(default=None, description="Uploaded filename") + content_type: str | None = Field(default=None, description="Uploaded content type") + file_size_bytes: int | None = Field( + default=None, description="Uploaded file size in bytes" + ) + message_count: int = Field( + ..., description="Number of messages created from the file" + ) + total_tokens: int = Field( + ..., description="Total tokens across messages created from the file" + ) + + def get_resource_id(self) -> str: + """Resource ID includes workspace, session, and generated file ID.""" + return f"{self.workspace_name}:{self.session_name}:{self.file_id}" + + +class GetContextEvent(BaseEvent): + """Emitted when context is retrieved for a session or peer.""" + + _event_type: ClassVar[str] = "context.retrieved" + _schema_version: ClassVar[int] = 1 + _category: ClassVar[str] = "api" + + workspace_name: str = Field(..., description="Workspace name") + context_scope: Literal["session", "peer"] = Field( + ..., description="Context endpoint scope" + ) + session_name: str | None = Field( + default=None, description="Session name for session-scoped context" + ) + peer_name: str | None = Field(default=None, description="Observer peer name") + target_name: str | None = Field(default=None, description="Observed peer name") + tokens_requested: int | None = Field( + default=None, + description="Caller-supplied tokens query parameter (None = endpoint default applied)", + ) + message_count: int = Field( + default=0, description="Number of messages returned in context" + ) + has_summary: bool = Field( + default=False, description="Whether a summary was returned" + ) + has_representation: bool = Field( + default=False, description="Whether a representation was returned" + ) + has_peer_card: bool = Field( + default=False, description="Whether a peer card was returned" + ) + search_query_provided: bool = Field( + default=False, description="Whether semantic search query text was provided" + ) + search_top_k: int | None = Field( + default=None, + description="Caller-supplied search_top_k (None = endpoint default)", + ) + search_max_distance: float | None = Field( + default=None, + description="Caller-supplied search_max_distance (None = endpoint default)", + ) + include_most_frequent: bool | None = Field( + default=None, + description="Caller-supplied include_most_frequent (None = endpoint default; defaults differ between peer and session endpoints)", + ) + max_conclusions: int | None = Field( + default=None, + description="Caller-supplied max_conclusions (None = endpoint default)", + ) + include_summary: bool | None = Field( + default=None, + description="Whether summary inclusion was requested; None when unsupported by endpoint", + ) + limit_to_session: bool = Field( + default=False, + description="Whether representation retrieval was session-limited", + ) + peer_perspective_provided: bool = Field( + default=False, + description="Whether peer_perspective was supplied for session context", + ) + total_duration_ms: float = Field(..., description="Total processing time") + + def get_resource_id(self) -> str: + """Resource ID identifies the requested context scope. + + Uses empty string (illegal in peer names — nanoid-derived) as the + absent-peer sentinel so that a peer literally named "none" does not + collide with the absent-peer case. + """ + peer_name = self.peer_name if self.peer_name is not None else "" + target_name = self.target_name if self.target_name is not None else "" + if self.context_scope == "session": + return ( + f"{self.workspace_name}:session:{self.session_name}:" + f"{peer_name}:{target_name}" + ) + return f"{self.workspace_name}:peer:{peer_name}:{target_name}" + + +__all__ = [ + "FileUploadedEvent", + "GetContextEvent", + "MessageCreatedEvent", +] diff --git a/src/telemetry/events/base.py b/src/telemetry/events/base.py index ea2371d9..358dc6b2 100644 --- a/src/telemetry/events/base.py +++ b/src/telemetry/events/base.py @@ -18,21 +18,28 @@ def generate_event_id( event_type: str, timestamp: datetime, resource_id: str, + honcho_version: str | None = None, ) -> str: """Generate a deterministic event ID for idempotency. Same inputs always produce the same ID, so retries are automatically - deduplicated on the receiving end. + deduplicated on the receiving end. `honcho_version` is folded into the + payload so two deploys emitting the same logical event produce distinct + IDs — protects against silently merging events whose payload shape may + have shifted between versions. Args: event_type: The CloudEvents type (e.g., "honcho.work.representation.completed") timestamp: When the event occurred resource_id: A unique identifier for the resource (can include workspace_id if relevant) + honcho_version: The honcho package version emitting the event; + included in the hash so cross-deploy events don't dedupe. Returns: A deterministic event ID in the format "evt_{base64_hash}" """ - payload = f"{event_type}:{resource_id}:{timestamp.isoformat()}" + version_segment = honcho_version or "" + payload = f"{event_type}:{resource_id}:{timestamp.isoformat()}:{version_segment}" hash_bytes = hashlib.sha256(payload.encode()).digest()[:16] # Use URL-safe base64 encoding, strip padding encoded = base64.urlsafe_b64encode(hash_bytes).decode().rstrip("=") @@ -57,6 +64,12 @@ class BaseEvent(BaseModel): _schema_version: ClassVar[int] _category: ClassVar[str] # "work", "activity", or "resource" + # Volume class for sampling decisions: + # - "ground_truth": always emitted at rate 1.0 (aggregates, calibration keys) + # - "high_volume": subject to TELEMETRY.HIGH_VOLUME_SAMPLE_RATE + # Default is ground_truth so existing events keep firing unconditionally. + _volume_class: ClassVar[str] = "ground_truth" + # Common timestamp field present in all events timestamp: datetime = Field( default_factory=lambda: datetime.now(UTC), @@ -78,6 +91,11 @@ class BaseEvent(BaseModel): """Return the event category (work, activity, or resource).""" return cls._category + @classmethod + def volume_class(cls) -> str: + """Return the volume class for sampling: 'ground_truth' or 'high_volume'.""" + return cls._volume_class + def get_resource_id(self) -> str: """Return the resource ID for idempotency key generation. @@ -89,9 +107,17 @@ class BaseEvent(BaseModel): raise NotImplementedError("Subclasses must implement get_resource_id()") def generate_id(self) -> str: - """Generate a deterministic event ID for this event instance.""" + """Generate a deterministic event ID for this event instance. + + Folds in the honcho package version so the same logical event from + two different deploys produces distinct ids — downstream dedupe by + id won't silently merge events whose body shape may have shifted. + """ + from src._version import HONCHO_VERSION + return generate_event_id( event_type=self.event_type(), timestamp=self.timestamp, resource_id=self.get_resource_id(), + honcho_version=HONCHO_VERSION, ) diff --git a/src/telemetry/events/dialectic.py b/src/telemetry/events/dialectic.py index 89ab9044..a83bd804 100644 --- a/src/telemetry/events/dialectic.py +++ b/src/telemetry/events/dialectic.py @@ -25,11 +25,11 @@ class DialecticCompletedEvent(BaseEvent): """ _event_type: ClassVar[str] = "dialectic.completed" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "dialectic" # Run identification (for correlating with iteration/tool events) - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") # Workspace context workspace_name: str = Field(..., description="Workspace name") @@ -67,6 +67,16 @@ class DialecticCompletedEvent(BaseEvent): default=0, description="Tokens written to prompt cache" ) + # Cap hit flag + hit_input_token_cap: bool = Field( + default=False, + description=( + "True when an iteration's input exceeded " + "settings.DIALECTIC.MAX_INPUT_TOKENS. Token-based — fires for the " + "single-oversized-message case too, not just message-list shrinkage." + ), + ) + def get_resource_id(self) -> str: """Resource ID is the run_id for uniqueness.""" return self.run_id diff --git a/src/telemetry/events/dream.py b/src/telemetry/events/dream.py index 33ce19d3..d9e8716f 100644 --- a/src/telemetry/events/dream.py +++ b/src/telemetry/events/dream.py @@ -22,11 +22,11 @@ class DreamRunEvent(BaseEvent): """ _event_type: ClassVar[str] = "dream.run" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "dream" # Run identification (for correlating with specialist/iteration/tool events) - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") # Workspace context workspace_name: str = Field(..., description="Workspace name") @@ -69,6 +69,43 @@ class DreamRunEvent(BaseEvent): ) total_duration_ms: float = Field(..., description="Total processing time") + # ---- Additive fields ---- + dream_type: str | None = Field( + default=None, + description="DreamType slug (currently 'omni'; future: 'deductive'/'inductive')", + ) + enabled_types_count: int = Field( + default=0, + description="len(settings.DREAM.ENABLED_TYPES) at run start — how many dream types this deploy was producing", + ) + trigger_reason: str | None = Field( + default=None, + description=( + "What tripped the schedule: 'document_threshold' | 'manual' | 'surprisal'. " + "Captured at schedule time and threaded through the queue payload." + ), + ) + delay_reason: str | None = Field( + default=None, + description=( + "What governed when this dream actually fired: 'idle_timeout' | " + "'immediate' | 'min_hours_gate'. Disambiguates from trigger_reason " + "to preserve the two-gate scheduler semantics in analytics." + ), + ) + documents_since_last_dream_at_schedule: int | None = Field( + default=None, + description=( + "Document count at the moment check_and_schedule_dream made the decision. " + "Named _at_schedule because the live count changes between schedule and fire " + "(idle delay) — this is the snapshot, not the current value." + ), + ) + document_threshold: int | None = Field( + default=None, + description="settings.DREAM.DOCUMENT_THRESHOLD snapshot at schedule time", + ) + def get_resource_id(self) -> str: """Resource ID is the run_id for uniqueness.""" return self.run_id @@ -82,11 +119,11 @@ class DreamSpecialistEvent(BaseEvent): """ _event_type: ClassVar[str] = "dream.specialist" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "dream" # Run identification (correlates with parent dream.run) - run_id: str = Field(..., description="8-char UUID prefix for run correlation") + run_id: str = Field(..., description="Nanoid for run correlation") # Specialist info specialist_type: str = Field( @@ -108,6 +145,52 @@ class DreamSpecialistEvent(BaseEvent): duration_ms: float = Field(..., description="Processing time") success: bool = Field(..., description="Whether the specialist succeeded") + # ---- Additive fields ---- + # Denormalized rollups so analytics can answer "how many observations did + # this specialist actually produce" without re-aggregating per-tool events. + # Sourced from ToolResult.metadata via tool_loop's all_tool_calls, NOT + # from tool-name counting — `create_observations` calls can produce zero + # observations when all entries fail validation. + created_observation_count: int = Field( + default=0, + description="Actual observations created across all create_observations calls (from ToolResult.metadata.created_count)", + ) + deleted_observation_count: int = Field( + default=0, + description="Actual observations deleted across all delete_observations calls (from ToolResult.metadata.deleted_count)", + ) + created_counts_by_level: dict[str, int] = Field( + default_factory=dict, + description=( + "Counts of created observations per level (explicit / deductive / " + "inductive / contradiction), aggregated across all " + "create_observations tool calls in this specialist run. Levels " + "with zero count may be omitted; queries should treat missing " + "keys as 0. Dict-of-counts rather than list[str] because dream " + "specialists can produce 10-20+ observations per run — a flat " + "list becomes noisy at that scale." + ), + ) + deleted_counts_by_level: dict[str, int] = Field( + default_factory=dict, + description=( + "Counts of deleted observations per level, aggregated across all " + "delete_observations tool calls in this specialist run." + ), + ) + peer_card_updated: bool = Field( + default=False, + description="True when at least one update_peer_card tool call succeeded", + ) + search_tool_calls_count: int = Field( + default=0, + description="Number of search_memory / search_messages / search_messages_temporal invocations", + ) + error_class: str | None = Field( + default=None, + description="Exception class name when success=False; None on success.", + ) + def get_resource_id(self) -> str: """Resource ID includes run_id and specialist type for uniqueness.""" return f"{self.run_id}:{self.specialist_type}" diff --git a/src/telemetry/events/llm.py b/src/telemetry/events/llm.py new file mode 100644 index 00000000..9cbe7901 --- /dev/null +++ b/src/telemetry/events/llm.py @@ -0,0 +1,264 @@ +"""LLM-call events for Honcho telemetry. + +These events fire once per provider hit (each iteration of an agentic tool loop, +each deriver/summarizer LLM call, etc.) and carry the full cost-attribution +context: model/provider/transport, token counts with cache breakdown, finish +reason, outcome (success/error), retry/fallback state, and run correlation. + +Unlike the existing aggregate `*Completed` events (representation, dialectic, +dream), this event is high-volume. It participates in the +`settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE` sampler so per-iteration emission +can be tuned against a budget. +""" + +from __future__ import annotations + +from enum import Enum +from typing import ClassVar, Literal + +from pydantic import Field + +from src.config import ModelTransport +from src.telemetry.events.base import BaseEvent + + +class CallPurpose(str, Enum): + """Closed taxonomy for LLM call purposes. + + The schema lint enforces that all `LLMCallCompletedEvent` emissions use a + value from this enum. Adding a new call site requires adding a value here + first — keeps the analytics taxonomy stable. + """ + + DERIVER_REPRESENTATION = "deriver.representation" + DIALECTIC_ANSWER = "dialectic.answer" + DREAM_DEDUCTION = "dream.deduction" + DREAM_INDUCTION = "dream.induction" + SUMMARY_SHORT = "summary.short" + SUMMARY_LONG = "summary.long" + + +class LLMCallCompletedEvent(BaseEvent): + """Emitted once per provider hit by `honcho_llm_call_inner`. + + Covers success, failure, and cancellation via `outcome`. The last attempt + of a tenacity retry chain is flagged with `is_final_attempt=True` regardless + of outcome — calibration queries for "exhausted" use + `outcome='error' AND is_final_attempt`. Cancellations (typically client + disconnect mid-stream or server shutdown) are distinct from errors and + should not feed error-rate alerting. + + Streaming note: when `was_stream=True`, the token counts are placeholders + (0) because token totals aren't knowable until the stream drains. Use the + aggregate envelopes (`DialecticCompletedEvent` etc.) for streamed-call + accuracy until streaming completion is wired through. + """ + + _event_type: ClassVar[str] = "llm.call.completed" + _schema_version: ClassVar[int] = 1 + _category: ClassVar[str] = "llm" + _volume_class: ClassVar[str] = "high_volume" + + # Context (None for system calls without workspace context) + workspace_name: str | None = Field(default=None, description="Workspace name") + call_purpose: CallPurpose | None = Field( + default=None, + description="Closed enum identifying the call site (deriver, dialectic, etc.)", + ) + parent_category: str | None = Field( + default=None, + description="Parent category for analytics joins: 'representation' | 'dialectic' | 'dream' | 'summary'", + ) + + # Provider info + transport: ModelTransport = Field( + ..., description="SDK transport: 'anthropic' | 'openai' | 'gemini'" + ) + provider_label: str | None = Field( + default=None, + description="Best-effort vendor inference for relay setups (e.g. 'anthropic' when an OpenRouter base_url + 'anthropic/claude-...' model is used); None when not reliably inferable", + ) + model: str = Field(..., description="Model identifier as sent to the provider") + effective_max_output_tokens: int = Field( + ..., description="max_tokens value used for this call" + ) + + # Token usage (zero on was_stream=True placeholder) + provider_input_tokens: int = Field(default=0, description="Provider input_tokens") + provider_output_tokens: int = Field(default=0, description="Provider output_tokens") + cache_read_tokens: int = Field( + default=0, description="Tokens read from prompt cache" + ) + cache_creation_tokens: int = Field( + default=0, description="Tokens written to prompt cache" + ) + + # Outcome + finish_reason: str | None = Field( + default=None, + description="First finish reason from the response (None on error)", + ) + outcome: Literal["success", "error", "cancelled"] = Field( + ..., + description="'success' when the provider returned a result, 'error' when it raised, 'cancelled' when the awaitable was cancelled (client disconnect, server shutdown). Cancellations should be excluded from error-rate alerting.", + ) + is_final_attempt: bool = Field( + ..., + description="True when this is the last allowed attempt (attempt == retry_attempts). Combine with outcome='error' to identify retry-exhausted calls. Cancellations are not retried so this reflects the attempt at cancellation time.", + ) + error_class: str | None = Field( + default=None, + description="Exception class name when outcome is 'error' or 'cancelled' (e.g. 'CancelledError')", + ) + + # Retry/fallback state + attempt: int = Field(..., description="1-indexed tenacity attempt number") + retry_attempts: int = Field(..., description="Total attempts allowed by caller") + was_fallback: bool = Field( + ..., description="True when this attempt used the fallback ModelConfig" + ) + + # Timing + duration_ms: float = Field( + ..., description="Wall-clock duration of the provider call" + ) + + # Shape + has_tools: bool = Field(default=False, description="True if tools were provided") + tool_call_count: int = Field( + default=0, description="Number of tool calls the model requested" + ) + was_stream: bool = Field( + default=False, + description="True for the stream_final_response path. Token counts are 0 placeholders — see class docstring.", + ) + + # Agent correlation (None for non-agent calls like summarizer / deriver) + run_id: str | None = Field( + default=None, + description="Agent run id (ULID when widened in follow-up)", + ) + iteration: int | None = Field( + default=None, + description="1-indexed iteration within an agentic tool loop. Passed explicitly via LLMTelemetryContext — NOT read from set_current_iteration (that fires after the LLM call)", + ) + + def get_resource_id(self) -> str: + """Resource id includes run_id + iteration + attempt + transport/model + so multi-attempt retries within one iteration get distinct ids.""" + run = self.run_id or "none" + iteration = self.iteration if self.iteration is not None else 0 + return f"{run}:{iteration}:{self.attempt}:{self.transport}:{self.model}" + + +class EmbeddingCallPurpose(str, Enum): + """Closed taxonomy for embedding call purposes. + + Mirrors `CallPurpose` for LLM calls. Adding a new embedding call site + requires adding a value here first — keeps the analytics taxonomy stable + and prevents free-form `track_name` drift from leaking into queries. + """ + + SEARCH_MEMORY = "search_memory" + SEARCH_MESSAGES = "search_messages" + CREATE_OBSERVATIONS = "create_observations" + VECTOR_SYNC = "vector_sync" + SUMMARY = "summary" + MESSAGE_CREATE = "message_create" + # Added so previously-unattributed call sites land on a distinct slug + # instead of None. Closed taxonomy — coordinate with analytics before + # adding more. + DIALECTIC_PREFETCH = "dialectic_prefetch" + SESSION_CONTEXT_SEARCH = "session_context_search" + PREFERENCE_EXTRACTION = "preference_extraction" + GENERIC_DOCUMENT_SEARCH = "generic_document_search" + + +class EmbeddingCallCompletedEvent(BaseEvent): + """Emitted once per embedding-provider call. + + Embedding calls are real provider spend (per-token like LLM calls). + Search tools, observation creation, the message-embedding sync, and + the deriver/summarizer paths all hit the embedding API; this event + captures cost-attribution context for all of them. + + Volume note: this event is high-volume. Interactive paths + (`search_memory` / `search_messages`) emit one event per query, so under + a search-heavy dialectic load this can match or exceed the LLM call + rate. The shared `HIGH_VOLUME_SAMPLE_RATE` covers both. + """ + + _event_type: ClassVar[str] = "embedding.call.completed" + _schema_version: ClassVar[int] = 1 + _category: ClassVar[str] = "llm" + _volume_class: ClassVar[str] = "high_volume" + + workspace_name: str | None = Field(default=None, description="Workspace name") + call_purpose: EmbeddingCallPurpose | None = Field( + default=None, + description=( + "Closed enum identifying the call site. Set by callers via the " + "`embedding_call_purpose` ContextVar; None when the call originated " + "outside an instrumented path." + ), + ) + parent_category: str | None = Field( + default=None, + description="Parent category for analytics joins (e.g. 'dialectic', 'representation')", + ) + + provider: str = Field(..., description="'openai' | 'gemini'") + model: str = Field(..., description="Model identifier") + input_count: int = Field( + ..., description="Number of texts embedded in this call (batch size)" + ) + input_tokens_estimate: int = Field( + default=0, + description=( + "tiktoken-based size proxy for the embedded text. ESTIMATE only — " + "the embedding client uses encoding_for_model() with a cl100k_base " + "fallback (see embedding_client.py:68-71), which is exact for " + "older OpenAI models, an approximation for newer ones, and a " + "rough proxy for Gemini (which has its own tokenizer)." + ), + ) + duration_ms: float = Field( + ..., description="Wall-clock duration of the provider call" + ) + + outcome: Literal["success", "error", "cancelled"] = Field( + ..., + description="'success' when the provider returned a result, 'error' when it raised, 'cancelled' when the awaitable was cancelled. Cancellations should be excluded from error-rate alerting.", + ) + is_final_attempt: bool = Field( + default=False, + description=( + "True on the last retry attempt. Mirrors LLMCallCompletedEvent's " + "convention: combine with outcome='error' to identify exhausted " + "embedding calls. Cancellations are not retried." + ), + ) + error_class: str | None = Field( + default=None, + description="Exception class name when outcome is 'error' or 'cancelled'", + ) + + run_id: str | None = Field( + default=None, + description="Agent run id when called from an agentic loop; None for sync/CRUD paths", + ) + + def get_resource_id(self) -> str: + """Resource id includes timestamp-derived components implicitly via + generate_id(); we just stake out a non-empty identifier scope.""" + run = self.run_id or "none" + purpose = self.call_purpose.value if self.call_purpose else "unknown" + return f"{run}:{purpose}:{self.provider}:{self.model}:{self.input_count}" + + +__all__ = [ + "CallPurpose", + "EmbeddingCallCompletedEvent", + "EmbeddingCallPurpose", + "LLMCallCompletedEvent", +] diff --git a/src/telemetry/events/representation.py b/src/telemetry/events/representation.py index 5dff20f2..77d6b292 100644 --- a/src/telemetry/events/representation.py +++ b/src/telemetry/events/representation.py @@ -22,7 +22,7 @@ class RepresentationCompletedEvent(BaseEvent): """ _event_type: ClassVar[str] = "representation.completed" - _schema_version: ClassVar[int] = 1 + _schema_version: ClassVar[int] = 2 _category: ClassVar[str] = "representation" # Workspace context @@ -57,9 +57,79 @@ class RepresentationCompletedEvent(BaseEvent): total_duration_ms: float = Field(..., description="Total processing time") # Token usage - input_tokens: int = Field(..., description="Input tokens used") + input_tokens: int = Field( + ..., + description=( + "Queued-message tokens (the ones we're actually reasoning ABOUT). " + "This field is the downstream metering key for " + "representation.completed — DO NOT rename or repurpose without " + "coordinating with downstream consumers." + ), + ) + total_input_tokens: int = Field( + ..., + description="Total tokens sent to the LLM (queued + extra context + scaffold)", + ) output_tokens: int = Field(..., description="Output tokens generated") + # ---- Additive fields ---- + # Token breakdown beyond `input_tokens` (queued-message tokens already + # captured above). These break out what made up the LLM prompt so analytics + # can answer "how much did extra context cost us per call". + queued_message_count: int = Field( + default=0, + description="Number of messages in this batch that were the actual queue items being reasoned about", + ) + prompt_message_count: int = Field( + default=0, + description="Total messages in the prompt — queued + extra interleaving context", + ) + prompt_message_tokens: int = Field( + default=0, + description="Sum of token_count across all messages in the prompt", + ) + extra_context_message_count: int = Field( + default=0, + description="prompt_message_count - queued_message_count: the extra-context messages we pulled in", + ) + extra_context_tokens: int = Field( + default=0, + description="prompt_message_tokens - input_tokens: token cost of the extra context", + ) + prompt_scaffold_tokens: int = Field( + default=0, + description="Estimated tokens for the system/scaffold portion of the prompt", + ) + + # Cap configuration + hit flags () + batch_max_tokens: int = Field( + default=0, + description="settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS at fetch time", + ) + max_input_tokens: int = Field( + default=0, description="settings.DERIVER.MAX_INPUT_TOKENS at call time" + ) + was_flush_enabled: bool = Field( + default=False, + description="settings.DERIVER.FLUSH_ENABLED snapshot at batch time", + ) + hit_batch_token_cap: bool = Field( + default=False, + description="True when the queue batcher clamped the batch to fit batch_max_tokens", + ) + hit_input_token_cap: bool = Field( + default=False, + description=( + "True when the LLM call truncated input messages to fit max_input_tokens." + ), + ) + + # Observer fanout + observer_count: int = Field( + default=0, + description="Number of observers this representation was saved against", + ) + def get_resource_id(self) -> str: """Resource ID includes workspace, session, and latest message for uniqueness.""" return f"{self.workspace_name}:{self.session_name}:{self.latest_message_id}" diff --git a/src/telemetry/prometheus/metrics.py b/src/telemetry/prometheus/metrics.py index f64399b7..90082b85 100644 --- a/src/telemetry/prometheus/metrics.py +++ b/src/telemetry/prometheus/metrics.py @@ -10,6 +10,7 @@ from prometheus_client import ( CONTENT_TYPE_LATEST, REGISTRY, Counter, + Gauge, disable_created_metrics, generate_latest, ) @@ -29,6 +30,12 @@ class NamespacedCounter(Counter): return super().labels(**kwargs) # type: ignore[return-value] +class NamespacedGauge(Gauge): + def labels(self, **kwargs: str) -> NamespacedGauge: + kwargs["namespace"] = cast(str, settings.METRICS.NAMESPACE) + return super().labels(**kwargs) # type: ignore[return-value] + + class TokenTypes(Enum): INPUT = "input" OUTPUT = "output" @@ -92,6 +99,32 @@ dreamer_tokens_processed_counter = NamespacedCounter( ["namespace", "specialist_name", "token_type"], ) +# CloudEvents emitter health metrics. Split intentional (sampled out) vs unintentional +# (dropped) so the dropped counter remains a real alert signal. +telemetry_events_emitted_counter = NamespacedCounter( + "telemetry_events_emitted", + "CloudEvents successfully placed on the emitter buffer", + ["namespace", "type"], +) + +telemetry_events_sampled_out_counter = NamespacedCounter( + "telemetry_events_sampled_out", + "CloudEvents intentionally dropped by HIGH_VOLUME_SAMPLE_RATE", + ["namespace", "type"], +) + +telemetry_events_dropped_counter = NamespacedCounter( + "telemetry_events_dropped", + "CloudEvents lost unintentionally (buffer_full or send_failed)", + ["namespace", "reason"], +) + +telemetry_buffer_size_gauge = NamespacedGauge( + "telemetry_buffer_size", + "Current size of the CloudEvents emitter buffer", + ["namespace"], +) + @final class PrometheusMetrics: @@ -217,6 +250,31 @@ class PrometheusMetrics: except Exception as e: self._handle_metric_error("record_dreamer_tokens", e) + def record_telemetry_event_emitted(self, *, event_type: str) -> None: + try: + telemetry_events_emitted_counter.labels(type=event_type).inc() + except Exception as e: + self._handle_metric_error("record_telemetry_event_emitted", e) + + def record_telemetry_event_sampled_out(self, *, event_type: str) -> None: + try: + telemetry_events_sampled_out_counter.labels(type=event_type).inc() + except Exception as e: + self._handle_metric_error("record_telemetry_event_sampled_out", e) + + def record_telemetry_event_dropped(self, *, reason: str) -> None: + # Reason is one of "buffer_full" | "send_failed". + try: + telemetry_events_dropped_counter.labels(reason=reason).inc() + except Exception as e: + self._handle_metric_error("record_telemetry_event_dropped", e) + + def set_telemetry_buffer_size(self, *, size: int) -> None: + try: + telemetry_buffer_size_gauge.labels().set(size) + except Exception as e: + self._handle_metric_error("set_telemetry_buffer_size", e) + prometheus_metrics = PrometheusMetrics() diff --git a/src/utils/agent_tools.py b/src/utils/agent_tools.py index 0d95cf6d..8cdfd6cb 100644 --- a/src/utils/agent_tools.py +++ b/src/utils/agent_tools.py @@ -20,12 +20,13 @@ from src.telemetry.events import ( AgentToolConclusionsCreatedEvent, AgentToolConclusionsDeletedEvent, AgentToolPeerCardUpdatedEvent, + EmbeddingCallPurpose, emit, ) from src.utils import summarizer from src.utils.formatting import format_new_turn_with_timestamp, utc_now_iso from src.utils.representation import Representation -from src.utils.types import get_current_iteration +from src.utils.types import ToolResult, embedding_call_purpose, get_current_iteration logger = logging.getLogger(__name__) @@ -313,16 +314,46 @@ class ObservationsCreatedResult: failed: list[ObservationFailure] -def _truncate_tool_output(output: str, max_chars: int | None = None) -> str: - """Truncate tool output to prevent token explosion.""" +def _truncate_tool_output( + output: str, max_chars: int | None = None +) -> tuple[str, int, bool]: + """Truncate tool output to prevent token explosion. + + Returns (text, original_chars, was_truncated). Callers thread the + truncation signal into `ToolResult.metadata` so + `AgentToolCallCompletedEvent` can report `was_truncated` and + `result_chars_before_truncation` instead of always emitting them as + None/False. + """ if max_chars is None: max_chars = settings.LLM.MAX_TOOL_OUTPUT_CHARS - if len(output) <= max_chars: - return output - truncated = output[:max_chars] - return ( - truncated - + f"\n\n[OUTPUT TRUNCATED - showing {max_chars:,} of {len(output):,} characters]" + original_chars = len(output) + if original_chars <= max_chars: + return output, original_chars, False + truncated = ( + output[:max_chars] + + f"\n\n[OUTPUT TRUNCATED - showing {max_chars:,} of {original_chars:,} characters]" + ) + return truncated, original_chars, True + + +def _maybe_truncated_result(output: str) -> "str | ToolResult": + """Run `_truncate_tool_output` and wrap in `ToolResult` only when the + output was actually clamped, so the truncation signal reaches the + `AgentToolCallCompletedEvent` emitter (which reads `was_truncated` / + `result_chars_before_truncation` from `ToolResult.metadata`). Returns a + bare `str` in the common no-truncation case to keep the handler + contract unchanged. + """ + content, original_chars, was_truncated = _truncate_tool_output(output) + if not was_truncated: + return content + return ToolResult( + content=content, + metadata={ + "was_truncated": True, + "result_chars_before_truncation": original_chars, + }, ) @@ -785,6 +816,8 @@ async def create_observations( workspace_name: str, message_ids: list[int], message_created_at: str, + run_id: str | None = None, + parent_category: str | None = None, ) -> ObservationsCreatedResult: """ Create multiple observations (documents) in the memory system in a single call. @@ -799,6 +832,9 @@ async def create_observations( workspace_name: Workspace identifier message_ids: List of message IDs these observations are based on message_created_at: Timestamp of the message that triggered these observations + run_id: Agent run id, threaded onto the embedding-call ContextVar so + EmbeddingCallCompletedEvents emitted here can be joined back to + the originating agent run. Returns: ObservationsCreatedResult with created count and any per-observation failures @@ -808,13 +844,15 @@ async def create_observations( return ObservationsCreatedResult(created_count=0, created_levels=[], failed=[]) normalized_observations = [ - _normalized_observation_input(obs) for obs in observations if obs.content.strip() + _normalized_observation_input(obs) + for obs in observations + if obs.content.strip() ] if not normalized_observations: logger.info("No non-empty observations to create") return ObservationsCreatedResult(created_count=0, created_levels=[], failed=[]) - # Phase 1: Ensure collection exists (short DB scope) + # Ensure collection exists (short DB scope) async with tracked_db("create_observations.collection") as db: await crud.get_or_create_collection( db, @@ -823,11 +861,17 @@ async def create_observations( observed=observed, ) - # Phase 2: Compute embeddings (no DB needed) + # Compute embeddings (no DB needed) contents = [obs.content for obs in normalized_observations] embeddings_by_index: dict[int, list[float]] | None = None try: - embeddings = await embedding_client.simple_batch_embed(contents) + with embedding_call_purpose( + EmbeddingCallPurpose.CREATE_OBSERVATIONS.value, + workspace_name=workspace_name, + run_id=run_id, + parent_category=parent_category, + ): + embeddings = await embedding_client.simple_batch_embed(contents) embeddings_by_index = dict( zip(range(len(normalized_observations)), embeddings, strict=True) ) @@ -846,7 +890,13 @@ async def create_observations( embedding = embeddings_by_index[i] else: try: - embedding = await embedding_client.embed(obs.content) + with embedding_call_purpose( + EmbeddingCallPurpose.CREATE_OBSERVATIONS.value, + workspace_name=workspace_name, + run_id=run_id, + parent_category=parent_category, + ): + embedding = await embedding_client.embed(obs.content) except Exception as e: logger.warning( "Error embedding observation content for level '%s': %s", @@ -890,7 +940,7 @@ async def create_observations( ) documents.append(doc) - # Phase 3: Bulk create all documents (short DB scope) + # Bulk create all documents (short DB scope) accepted: list[schemas.DocumentCreate] = [] if documents: async with tracked_db("create_observations.save") as db: @@ -1207,7 +1257,7 @@ async def _handle_create_observations_impl( tool_input: dict[str, Any], *, forced_level: str | None = None, -) -> str: +) -> "str | ToolResult": """Handle create_observations tool.""" raw_observations = tool_input.get("observations", []) @@ -1271,6 +1321,8 @@ async def _handle_create_observations_impl( workspace_name=ctx.workspace_name, message_ids=message_ids, message_created_at=message_created_at, + run_id=ctx.run_id, + parent_category=ctx.parent_category, ) # Merge validation and embedding failures @@ -1311,18 +1363,27 @@ async def _handle_create_observations_impl( ) response += f"\nFailed {len(all_failures)}: {failure_details}" - return response + # +5: surface created_count so DreamSpecialistEvent can sum actual + # observations across the run rather than just counting create_observations + # calls (which would conflate "1 call that made 5 observations" with + # "5 calls that each made 1"). + from src.utils.types import ToolResult + + return ToolResult( + content=response, + metadata={"created_count": result.created_count, "levels": levels}, + ) async def _handle_create_observations( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": return await _handle_create_observations_impl(ctx, tool_input) async def _handle_create_observations_deductive( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": return await _handle_create_observations_impl( ctx, tool_input, @@ -1332,7 +1393,7 @@ async def _handle_create_observations_deductive( async def _handle_create_observations_inductive( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": return await _handle_create_observations_impl( ctx, tool_input, @@ -1340,7 +1401,9 @@ async def _handle_create_observations_inductive( ) -async def _handle_update_peer_card(ctx: ToolContext, tool_input: dict[str, Any]) -> str: +async def _handle_update_peer_card( + ctx: ToolContext, tool_input: dict[str, Any] +) -> "str | ToolResult": """Handle update_peer_card tool.""" # Check if peer card creation is disabled via configuration if ctx.configuration is not None and not ctx.configuration.peer_card.create: @@ -1427,12 +1490,19 @@ async def _handle_update_peer_card(ctx: ToolContext, tool_input: dict[str, Any]) ) ) - return f"Updated peer card for {ctx.observed} by {ctx.observer}" + # signal a successful peer_card update so DreamSpecialistEvent + # can set its `peer_card_updated` flag without name-counting. + from src.utils.types import ToolResult + + return ToolResult( + content=f"Updated peer card for {ctx.observed} by {ctx.observer}", + metadata={"peer_card_updated": True, "facts_count": len(normalized_peer_card)}, + ) async def _handle_get_recent_history( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": """Handle get_recent_history tool.""" _ = tool_input async with tracked_db("tool.get_recent_history") as db: @@ -1454,21 +1524,39 @@ async def _handle_get_recent_history( else f"from {ctx.observed} across sessions" ) output = f"Conversation history ({len(history)} messages {scope}):\n{history_text}" - return _truncate_tool_output(output) + return _maybe_truncated_result(output) -async def _handle_search_memory(ctx: ToolContext, tool_input: dict[str, Any]) -> str: +async def _handle_search_memory( + ctx: ToolContext, tool_input: dict[str, Any] +) -> "str | ToolResult": """Handle search_memory tool.""" + from src.utils.types import ToolResult + top_k = min(_safe_int(tool_input.get("top_k"), 20), 40) query = tool_input["query"] try: - query_embedding = await embedding_client.embed(query) + with embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MEMORY.value, + workspace_name=ctx.workspace_name, + run_id=ctx.run_id, + parent_category=ctx.parent_category, + ): + query_embedding = await embedding_client.embed(query) except ValueError: return ( "ERROR: Query exceeds maximum token limit of " + f"{settings.EMBEDDING.MAX_INPUT_TOKENS}. Please use a shorter query." ) + # Base telemetry metadata; results_count gets filled in below. + search_meta: dict[str, Any] = { + "top_k": top_k, + "used_embedding": True, + "embedding_query_count": 1, + "query_tokens": _estimate_tokens_safe(query), + } + documents = await crud.query_documents( db=None, workspace_name=ctx.workspace_name, @@ -1481,11 +1569,13 @@ async def _handle_search_memory(ctx: ToolContext, tool_input: dict[str, Any]) -> mem = Representation.from_documents(documents) total_count = mem.len() if total_count == 0: - # fallback behavior: if the memory is *empty*, that means we're quite - # early in a workspace/peer/session -- in order to give good answers in - # this stage, and be efficient with tool calls, and make sure the model - # doesn't short-circuit and think there's nothing here, we - # automatically search the message history for relevant information. + # Empty-memory fallback: if the memory is *empty*, that means we're + # quite early in a workspace/peer/session -- in order to give good + # answers in this stage, and be efficient with tool calls, and make + # sure the model doesn't short-circuit and think there's nothing + # here, we automatically search the message history for relevant + # information. + zero_hit_meta = {**search_meta, "results_count": 0} if ctx.agent_type == "dialectic": limit = min(_safe_int(tool_input.get("top_k"), 20), 20) message_output = None @@ -1503,21 +1593,33 @@ async def _handle_search_memory(ctx: ToolContext, tool_input: dict[str, Any]) -> snippets, f"for query '{query}'" ) if message_output: - return ( - f"No observations yet. Message search results:\n\n{message_output}" + fallback_meta = {**zero_hit_meta, "results_count": len(snippets)} + return ToolResult( + content=f"No observations yet. Message search results:\n\n{message_output}", + metadata=fallback_meta, ) - return ( - f"No observations found for query '{query}', and no messages found in " - "history. Try a different phrasing or use grep_messages for exact text." + return ToolResult( + content=( + f"No observations found for query '{query}', and no messages found in " + "history. Try a different phrasing or use grep_messages for exact text." + ), + metadata=zero_hit_meta, ) - return f"No observations found for query '{query}'" + return ToolResult( + content=f"No observations found for query '{query}'", + metadata=zero_hit_meta, + ) mem_str = mem.str_with_ids() if ctx.include_observation_ids else str(mem) - return f"Found {total_count} observations for query '{query}':\n\n{mem_str}" + search_meta["results_count"] = total_count + return ToolResult( + content=f"Found {total_count} observations for query '{query}':\n\n{mem_str}", + metadata=search_meta, + ) async def _handle_get_observation_context( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": """Handle get_observation_context tool.""" async with tracked_db("tool.get_observation_context") as db: messages = await get_observation_context( @@ -1540,16 +1642,26 @@ async def _handle_get_observation_context( ] ) output = f"Retrieved {len(messages)} messages with context:\n{messages_text}" - return _truncate_tool_output(output) + return _maybe_truncated_result(output) -async def _handle_search_messages(ctx: ToolContext, tool_input: dict[str, Any]) -> str: +async def _handle_search_messages( + ctx: ToolContext, tool_input: dict[str, Any] +) -> "str | ToolResult": """Handle search_messages tool.""" + from src.utils.types import ToolResult + query = tool_input["query"] limit = min(_safe_int(tool_input.get("limit"), 10), 20) # Cap at 20 # Pre-compute embedding outside DB session to avoid holding a connection # during the external API call (same pattern as _handle_search_memory). - query_embedding = await embedding_client.embed(query) + with embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MESSAGES.value, + workspace_name=ctx.workspace_name, + run_id=ctx.run_id, + parent_category=ctx.parent_category, + ): + query_embedding = await embedding_client.embed(query) snippets = await crud.search_messages( workspace_name=ctx.workspace_name, session_name=ctx.session_name, @@ -1559,13 +1671,25 @@ async def _handle_search_messages(ctx: ToolContext, tool_input: dict[str, Any]) embedding=query_embedding, observer=ctx.observer, ) + search_meta: dict[str, Any] = { + "top_k": limit, + "used_embedding": True, + "embedding_query_count": 1, + "query_tokens": _estimate_tokens_safe(query), + "results_count": len(snippets), + } if not snippets: - return f"No messages found for query '{query}'" + return ToolResult( + content=f"No messages found for query '{query}'", + metadata=search_meta, + ) formatted = _format_message_snippets(snippets, f"for query '{query}'") - return formatted + return ToolResult(content=formatted, metadata=search_meta) -async def _handle_grep_messages(ctx: ToolContext, tool_input: dict[str, Any]) -> str: +async def _handle_grep_messages( + ctx: ToolContext, tool_input: dict[str, Any] +) -> "str | ToolResult": """Handle grep_messages tool.""" text = tool_input.get("text", "") if not text: @@ -1606,7 +1730,7 @@ async def _handle_grep_messages(ctx: ToolContext, tool_input: dict[str, Any]) -> f"Found {total_matches} messages containing '{text}' in {len(snippets)} conversation snippets:\n\n" + "\n\n".join(snippet_texts) ) - return _truncate_tool_output(output) + return _maybe_truncated_result(output) def _parse_date(date_str: str | None, param_name: str) -> datetime | None | str: @@ -1621,7 +1745,7 @@ def _parse_date(date_str: str | None, param_name: str) -> datetime | None | str: async def _handle_get_messages_by_date_range( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": """Handle get_messages_by_date_range tool.""" after_date_str = tool_input.get("after_date") before_date_str = tool_input.get("before_date") @@ -1677,12 +1801,12 @@ async def _handle_get_messages_by_date_range( output = ( f"Found {msg_count} messages ({range_desc}, {order_desc}):\n\n{messages_text}" ) - return _truncate_tool_output(output) + return _maybe_truncated_result(output) async def _handle_search_messages_temporal( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": """Handle search_messages_temporal tool.""" query = tool_input.get("query", "") if not query: @@ -1703,7 +1827,13 @@ async def _handle_search_messages_temporal( # Pre-compute embedding outside DB session to avoid holding a connection # during the external API call. - query_embedding = await embedding_client.embed(query) + with embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MESSAGES.value, + workspace_name=ctx.workspace_name, + run_id=ctx.run_id, + parent_category=ctx.parent_category, + ): + query_embedding = await embedding_client.embed(query) snippets = await crud.search_messages_temporal( workspace_name=ctx.workspace_name, session_name=ctx.session_name, @@ -1722,11 +1852,25 @@ async def _handle_search_messages_temporal( date_filter.append(f"before {before_date_str}") filter_desc = f" ({' and '.join(date_filter)})" if date_filter else "" + # Matches the search_messages metadata shape so analytics can filter + # AgentToolCallCompletedEvent uniformly across all embedding-backed + # search tools (search_memory / search_messages / search_messages_temporal). + search_meta: dict[str, Any] = { + "top_k": limit, + "used_embedding": True, + "embedding_query_count": 1, + "query_tokens": _estimate_tokens_safe(query), + "results_count": len(snippets), + } + if not snippets: - return f"No messages found for query '{query}'{filter_desc}" + return ToolResult( + content=f"No messages found for query '{query}'{filter_desc}", + metadata=search_meta, + ) formatted = _format_message_snippets(snippets, f"for query '{query}'{filter_desc}") - return formatted + return ToolResult(content=formatted, metadata=search_meta) async def _handle_get_recent_observations( @@ -1820,7 +1964,7 @@ async def _handle_get_peer_card(ctx: ToolContext, tool_input: dict[str, Any]) -> async def _handle_delete_observations( ctx: ToolContext, tool_input: dict[str, Any] -) -> str: +) -> "str | ToolResult": """Handle delete_observations tool.""" observation_ids = tool_input.get("observation_ids", []) if not observation_ids: @@ -1859,7 +2003,16 @@ async def _handle_delete_observations( ) ) - return f"Deleted {deleted_count} observations" + # +5: surface deleted_count + levels for DreamSpecialistEvent rollups. + from src.utils.types import ToolResult + + return ToolResult( + content=f"Deleted {deleted_count} observations", + metadata={ + "deleted_count": deleted_count, + "levels": [level for _, level in deleted], + }, + ) async def _handle_finish_consolidation( @@ -1876,12 +2029,20 @@ async def _handle_extract_preferences( ) -> str: """Handle extract_preferences tool.""" _ = tool_input - results = await extract_preferences( + # Wrap so the batch-embed + downstream search_messages embedding calls + # all carry preference-extraction attribution. + with embedding_call_purpose( + EmbeddingCallPurpose.PREFERENCE_EXTRACTION.value, workspace_name=ctx.workspace_name, - session_name=ctx.session_name, - observed=ctx.observed, - observer=ctx.observer, - ) + run_id=ctx.run_id, + parent_category=ctx.parent_category, + ): + results = await extract_preferences( + workspace_name=ctx.workspace_name, + session_name=ctx.session_name, + observed=ctx.observed, + observer=ctx.observer, + ) messages = results.get("messages", []) @@ -1901,7 +2062,13 @@ async def _handle_extract_preferences( def _format_message_snippets( snippets: list[tuple[list[models.Message], list[models.Message]]], desc: str ) -> str: - """Format message snippets for output.""" + """Format message snippets for output. + + Returns bare `str` because callers concatenate it into other strings + or place it into `ToolResult.content`. Callers that need the + truncation telemetry signal route their own output through + `_maybe_truncated_result` themselves. + """ snippet_texts: list[str] = [] total_matches = sum(len(matches) for matches, _ in snippets) for i, (matches, context) in enumerate(snippets, 1): @@ -1921,7 +2088,10 @@ def _format_message_snippets( f"Found {total_matches} matching messages in {len(snippets)} conversation snippets {desc}:\n\n" + "\n\n".join(snippet_texts) ) - return _truncate_tool_output(output) + # `[0]` extracts the truncated text — telemetry signal is discarded here + # because callers wrap the result into ToolResult themselves (and so any + # downstream truncation telemetry should come from the caller's path). + return _truncate_tool_output(output)[0] async def _handle_get_reasoning_chain( @@ -1960,9 +2130,7 @@ async def _handle_get_reasoning_chain( premise_lines: list[Any] = [] for p in premises: p_level = p.level or "explicit" - premise_lines.append( - f" - [id:{p.id}] ({p_level}): {p.content}" - ) + premise_lines.append(f" - [id:{p.id}] ({p_level}): {p.content}") output_parts.append( f"\n**Premises ({len(premises)}):**\n" + "\n".join(premise_lines) @@ -1979,7 +2147,7 @@ async def _handle_get_reasoning_chain( source_lines: list[Any] = [] for s in sources: s_level = s.level or "explicit" - source_lines.append(f" - [id:{s.id}] ({s_level}): {s.content}") + source_lines.append(f" - [id:{s.id}] ({s_level}): {s.content}") output_parts.append( f"\n**Sources ({len(sources)}):**\n" + "\n".join(source_lines) ) @@ -2007,7 +2175,7 @@ async def _handle_get_reasoning_chain( child_lines: list[Any] = [] for c in children: c_level = c.level or "explicit" - child_lines.append(f" - [id:{c.id}] ({c_level}): {c.content}") + child_lines.append(f" - [id:{c.id}] ({c_level}): {c.content}") output_parts.append( f"\n**Derived Conclusions ({len(children)}):**\n" + "\n".join(child_lines) @@ -2110,33 +2278,187 @@ async def create_tool_executor( Returns: String result describing what was done """ - logger.info("[tool call] %s %s", tool_name, tool_input) + import time + + from src.utils.types import ( + ToolResult, + get_current_iteration, + get_current_provider_tool_call_id, + get_current_tool_call_seq, + set_last_tool_metadata, + ) + + # Log nondisclosive call shape only. Raw `tool_input` can carry user + # content (search queries, peer-card text, etc.); the param keys are + # enough to reconstruct the call shape from telemetry without leaking + # content to log sinks. + logger.info("[tool call] %s keys=%s", tool_name, sorted(tool_input.keys())) + + start = time.perf_counter() + # Defaults populated even on early returns / error paths so the + # AgentToolCallCompletedEvent emission below can fire consistently. + result_str: str = "" + metadata: dict[str, Any] = {} + is_error: bool = False try: handler = _TOOL_HANDLERS.get(tool_name) if handler: - result = await handler(ctx, tool_input) - logger.info("[tool result] %s %s", tool_name, result) - return result - return f"Unknown tool: {tool_name}" + handler_result = await handler(ctx, tool_input) + # Handlers return either a plain str (existing contract) or a + # ToolResult(content, metadata) carrying structured fields for + # telemetry and specialist rollups. + if isinstance(handler_result, ToolResult): + result_str = handler_result.content + metadata = handler_result.metadata + else: + result_str = handler_result + # Log shape, not contents — `result_str` can carry retrieved + # observations, message snippets, peer-card text, etc. The + # AgentToolCallCompletedEvent telemetry captures the + # structured metadata for analytics. + logger.info( + "[tool result] %s len=%d metadata_keys=%s", + tool_name, + len(result_str), + sorted(metadata.keys()), + ) + else: + result_str = f"Unknown tool: {tool_name}" + is_error = True + logger.warning(result_str) + except asyncio.CancelledError: + # Cancellation (client disconnect, server shutdown) — populate + # telemetry fields so the finally-block emit records an accurate + # event, then re-raise so cancellation propagates to the caller. + # CancelledError extends BaseException, so the broader except + # clauses below do not catch it. + result_str = f"Tool {tool_name} cancelled" + is_error = True + raise except ValueError as e: # Recoverable errors (bad input, validation failures) - return to LLM - error_msg = f"Tool {tool_name} failed with invalid input: {e}" - logger.warning(error_msg) - return error_msg + result_str = f"Tool {tool_name} failed with invalid input: {e}" + is_error = True + logger.warning(result_str) except KeyError as e: # Missing required parameters - return to LLM - error_msg = f"Tool {tool_name} missing required parameter: {e}" - logger.warning(error_msg) - return error_msg + result_str = f"Tool {tool_name} missing required parameter: {e}" + is_error = True + logger.warning(result_str) except Exception as e: # Unexpected errors - log with full traceback but still return to LLM # We don't re-raise because the LLM should be able to continue with other tools - error_msg = f"Tool {tool_name} failed unexpectedly: {type(e).__name__}: {e}" - logger.error(error_msg, exc_info=True) + result_str = ( + f"Tool {tool_name} failed unexpectedly: {type(e).__name__}: {e}" + ) + is_error = True + logger.error(result_str, exc_info=True) # No explicit rollback needed — each handler uses tracked_db() which # handles rollback in its finally block - return error_msg + finally: + # Emit in finally so CancelledError (and any other BaseException) + # still produces an AgentToolCallCompletedEvent before propagating. + duration_ms = (time.perf_counter() - start) * 1000 + + # Publish ToolResult.metadata for tool_loop to stash on all_tool_calls. + # Reset to {} (rather than leaving stale metadata) so a non-ToolResult + # handler doesn't appear to have leaked metadata from a prior call. + set_last_tool_metadata(metadata) + + _emit_agent_tool_call_completed( + ctx=ctx, + tool_name=tool_name, + duration_ms=duration_ms, + result_str=result_str, + metadata=metadata, + is_error=is_error, + iteration=get_current_iteration(), + tool_call_seq=get_current_tool_call_seq(), + provider_tool_call_id=get_current_provider_tool_call_id(), + ) + + return result_str return execute_tool + + +def _emit_agent_tool_call_completed( + *, + ctx: "ToolContext", + tool_name: str, + duration_ms: float, + result_str: str, + metadata: dict[str, Any], + is_error: bool, + iteration: int, + tool_call_seq: int, + provider_tool_call_id: str | None, +) -> None: + """Build and emit AgentToolCallCompletedEvent. Best-effort; swallows errors. + + Skipped when the executor was constructed without agent identifiers + (run_id / agent_type / parent_category) — telemetry attribution requires + all three. + """ + if not (ctx.run_id and ctx.agent_type and ctx.parent_category): + return + try: + from src.telemetry.events import AgentToolCallCompletedEvent, emit + + emit( + AgentToolCallCompletedEvent( + run_id=ctx.run_id, + iteration=iteration, + tool_call_seq=tool_call_seq, + provider_tool_call_id=provider_tool_call_id, + parent_category=ctx.parent_category, + agent_type=ctx.agent_type, + workspace_name=ctx.workspace_name, + tool_name=tool_name, + duration_ms=duration_ms, + is_error=is_error, + result_chars=len(result_str), + result_chars_before_truncation=metadata.get( + "result_chars_before_truncation" + ), + result_tokens_estimate=_estimate_tokens(result_str), + was_truncated=bool(metadata.get("was_truncated", False)), + query_tokens=metadata.get("query_tokens"), + top_k=metadata.get("top_k"), + results_count=metadata.get("results_count"), + used_embedding=metadata.get("used_embedding"), + embedding_query_count=int(metadata.get("embedding_query_count") or 0), + ) + ) + except Exception: # pragma: no cover - telemetry must not raise + logger.debug("Failed to emit AgentToolCallCompletedEvent", exc_info=True) + + +def _estimate_tokens(text: str) -> int: + """Tiktoken-based size proxy for tool result strings. Best-effort.""" + if not text: + return 0 + try: + import tiktoken + + # Use cl100k_base as a stable default — matches the embedding-client + # fallback. Exact accuracy isn't required; this is a size proxy. + encoding = tiktoken.get_encoding("cl100k_base") + return len(encoding.encode(text)) + except Exception: + # Fall back to a rough char→token ratio so the field is always populated. + return max(1, len(text) // 4) + + +def _estimate_tokens_safe(text: str | None) -> int | None: + """Wrapper around `_estimate_tokens` that returns None on falsy input. + + Used by search-handler metadata where we want `query_tokens=None` + when the query is empty rather than 0 (which could be confused with a + real measurement). + """ + if not text: + return None + return _estimate_tokens(text) diff --git a/src/utils/queue_payload.py b/src/utils/queue_payload.py index 6e09ab47..605cf615 100644 --- a/src/utils/queue_payload.py +++ b/src/utils/queue_payload.py @@ -57,6 +57,15 @@ class DreamPayload(BasePayload): observer: str observed: str session_name: str | None = None + # scheduling context captured at schedule time so the + # eventual DreamRunEvent can attribute the cycle back to *why* it was + # scheduled (which threshold tripped) and *what* governed when it fired + # (idle delay vs. immediate vs. min-hours gate). Defaults preserve + # backward compat for any in-flight payloads from older producers. + trigger_reason: str | None = None + delay_reason: str | None = None + documents_since_last_dream_at_schedule: int | None = None + document_threshold: int | None = None class DeletionPayload(BasePayload): @@ -90,6 +99,10 @@ def create_dream_payload( observer: str, observed: str, session_name: str | None = None, + trigger_reason: str | None = None, + delay_reason: str | None = None, + documents_since_last_dream_at_schedule: int | None = None, + document_threshold: int | None = None, ) -> dict[str, Any]: """Create a dream payload.""" return DreamPayload( @@ -97,6 +110,10 @@ def create_dream_payload( observer=observer, observed=observed, session_name=session_name, + trigger_reason=trigger_reason, + delay_reason=delay_reason, + documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule, + document_threshold=document_threshold, ).model_dump(mode="json", exclude_none=True) diff --git a/src/utils/search.py b/src/utils/search.py index 9c44bb1b..15721933 100644 --- a/src/utils/search.py +++ b/src/utils/search.py @@ -17,8 +17,10 @@ from src.dependencies import tracked_db from src.embedding_client import embedding_client from src.exceptions import ValidationException from src.models import session_peers_table +from src.telemetry.events import EmbeddingCallPurpose from src.utils.filter import apply_filter from src.utils.formatting import ILIKE_ESCAPE_CHAR, escape_ilike_pattern +from src.utils.types import embedding_call_purpose from src.vector_store import get_external_vector_store T = TypeVar("T") @@ -380,7 +382,12 @@ async def search( if settings.EMBED_MESSAGES and isinstance(workspace_name, str): try: - query_embedding = await embedding_client.embed(query) + with embedding_call_purpose( + EmbeddingCallPurpose.SEARCH_MESSAGES.value, + workspace_name=workspace_name, + parent_category="api", + ): + query_embedding = await embedding_client.embed(query) except ValueError as e: raise ValidationException( f"Query exceeds maximum token limit of {settings.EMBEDDING.MAX_INPUT_TOKENS}." diff --git a/src/utils/summarizer.py b/src/utils/summarizer.py index d964402c..42d18bcc 100644 --- a/src/utils/summarizer.py +++ b/src/utils/summarizer.py @@ -16,9 +16,11 @@ from src.crud.session import session_cache_key from src.dependencies import tracked_db from src.exceptions import ResourceNotFoundException from src.llm import HonchoLLMCallResponse, honcho_llm_call +from src.llm.types import LLMTelemetryContext from src.models import Message from src.telemetry import prometheus_metrics from src.telemetry.events import AgentToolSummaryCreatedEvent, emit +from src.telemetry.events.llm import CallPurpose from src.telemetry.logging import accumulate_metric, conditional_observe from src.telemetry.prometheus.metrics import ( DeriverComponents, @@ -198,6 +200,8 @@ async def create_short_summary( formatted_messages: str, input_tokens: int, previous_summary: str | None = None, + *, + workspace_name: str | None = None, ) -> HonchoLLMCallResponse[str]: # input_tokens indicates how many tokens the message list + previous summary take up # we want to optimize short summaries to be smaller than the actual content being summarized @@ -219,6 +223,11 @@ async def create_short_summary( model_config=_get_summary_model_config(), prompt=prompt, max_tokens=settings.SUMMARY.MAX_TOKENS_SHORT, + telemetry=LLMTelemetryContext( + workspace_name=workspace_name, + call_purpose=CallPurpose.SUMMARY_SHORT.value, + parent_category="summary", + ), ) @@ -226,6 +235,8 @@ async def create_short_summary( async def create_long_summary( formatted_messages: str, previous_summary: str | None = None, + *, + workspace_name: str | None = None, ) -> HonchoLLMCallResponse[str]: # the word/token ratio is roughly 4:3 so we multiply by 0.75. # LLMs *seem* to respond better to getting asked for a word count but should workshop this. @@ -244,6 +255,11 @@ async def create_long_summary( model_config=_get_summary_model_config(), prompt=prompt, max_tokens=settings.SUMMARY.MAX_TOKENS_LONG, + telemetry=LLMTelemetryContext( + workspace_name=workspace_name, + call_purpose=CallPurpose.SUMMARY_LONG.value, + parent_category="summary", + ), ) @@ -437,16 +453,19 @@ async def _create_and_save_summary( last_message_id=last_message_id, last_message_content_preview=last_message_content_preview, message_count=message_count, + workspace_name=workspace_name, ) + # Compute scaffold tokens up front (cheap + idempotent) so both the + # save-summary path and the telemetry emit below can use it + # without basedpyright tripping on a possibly-unbound name. + if summary_type == SummaryType.SHORT: + prompt_tokens = estimate_short_summary_prompt_tokens() + else: + prompt_tokens = estimate_long_summary_prompt_tokens() + # Step 3: Save to database with new transaction if not is_fallback: - # Get base prompt tokens based on summary type - if summary_type == SummaryType.SHORT: - prompt_tokens = estimate_short_summary_prompt_tokens() - else: - prompt_tokens = estimate_long_summary_prompt_tokens() - track_deriver_input_tokens( task_type=DeriverTaskTypes.SUMMARY, components={ @@ -499,6 +518,9 @@ async def _create_and_save_summary( # Note: Using AgentToolSummaryCreatedEvent with dummy run_id/iteration since # this is called from the deriver, not from an agentic loop if not is_fallback: + # `prompt_tokens` is set in the `if not is_fallback` block above for + # both SHORT and LONG summary types — we're inside the same branch, so + # it's guaranteed bound here. emit( AgentToolSummaryCreatedEvent( run_id="deriver", # Placeholder - not from an agentic run @@ -513,6 +535,10 @@ async def _create_and_save_summary( summary_type="short" if summary_type == SummaryType.SHORT else "long", input_tokens=llm_input_tokens, output_tokens=llm_output_tokens, + # additive token-breakdown fields + previous_summary_tokens=previous_summary_tokens, + message_tokens=messages_tokens, + prompt_scaffold_tokens=prompt_tokens, ) ) @@ -526,6 +552,8 @@ async def _create_summary( last_message_id: int, last_message_content_preview: str, message_count: int, + *, + workspace_name: str | None = None, ) -> tuple[Summary, bool, int, int]: """ Generate a summary of the provided messages using an LLM. @@ -554,11 +582,16 @@ async def _create_summary( try: if summary_type == SummaryType.SHORT: response = await create_short_summary( - formatted_messages, input_tokens, previous_summary_text + formatted_messages, + input_tokens, + previous_summary_text, + workspace_name=workspace_name, ) else: response = await create_long_summary( - formatted_messages, previous_summary_text + formatted_messages, + previous_summary_text, + workspace_name=workspace_name, ) summary_text = response.content diff --git a/src/utils/types.py b/src/utils/types.py index dd66dccf..2a470d20 100644 --- a/src/utils/types.py +++ b/src/utils/types.py @@ -1,7 +1,8 @@ -from collections.abc import Awaitable, Callable +from collections.abc import Awaitable, Callable, Generator +from contextlib import contextmanager from contextvars import ContextVar from dataclasses import dataclass, field -from typing import Generic, Literal, TypeVar +from typing import Any, Generic, Literal, TypeVar T = TypeVar("T") @@ -20,6 +21,204 @@ def get_current_iteration() -> int: return _current_iteration.get() +# ordinal of the tool call within its iteration. Two calls +# to the same tool in one iteration (the model can do this) need distinct +# resource ids on AgentToolCallCompletedEvent — seq disambiguates. +_current_tool_call_seq: ContextVar[int] = ContextVar("current_tool_call_seq", default=0) +# Optional provider-supplied tool-call id (e.g. Anthropic's `toolu_*`) so the +# emitted event can be cross-referenced with provider logs. +_current_provider_tool_call_id: ContextVar[str | None] = ContextVar( + "current_provider_tool_call_id", default=None +) + + +def set_current_tool_call_seq(seq: int, provider_tool_call_id: str | None) -> None: + """Set the current tool-call ordinal + provider id for telemetry context. + + Called by tool_loop before invoking the tool_executor closure. The seq + starts at 0 within each iteration's tool batch and increments per call. + """ + _current_tool_call_seq.set(seq) + _current_provider_tool_call_id.set(provider_tool_call_id) + + +def get_current_tool_call_seq() -> int: + return _current_tool_call_seq.get() + + +def get_current_provider_tool_call_id() -> str | None: + return _current_provider_tool_call_id.get() + + +# After `execute_tool` finishes, the metadata dict from the handler's +# ToolResult is published here so tool_loop can stash it on the +# `all_tool_calls` entry (which DreamSpecialistEvent reads for rollups). +# Default is None (not {}) per ruff B039 — mutable defaults on +# ContextVars are foot-guns; the getter normalizes None → {}. +_last_tool_metadata: ContextVar[dict[str, Any] | None] = ContextVar( + "last_tool_metadata", default=None +) + + +def set_last_tool_metadata(metadata: dict[str, Any]) -> None: + """Publish the just-finished tool call's ToolResult metadata.""" + _last_tool_metadata.set(metadata) + + +def get_last_tool_metadata() -> dict[str, Any]: + """Read the last tool call's metadata. Returns {} when no ToolResult was returned.""" + return _last_tool_metadata.get() or {} + + +@contextmanager +def iteration_scope() -> Generator[None]: + """Reset per-tool-loop ContextVars on exit. + + Wrap the body of `tool_loop.run_tool_loop` so a subsequent loop in the + same asyncio Task (worker batches, tests using TestClient) starts with + fresh iteration / tool-call state instead of inheriting stale values + from a prior loop. ContextVars are per-Task in asyncio, so cross-request + leakage is unlikely under normal FastAPI use — but this is defensive and + cheap. + """ + iter_token = _current_iteration.set(0) + seq_token = _current_tool_call_seq.set(0) + pid_token = _current_provider_tool_call_id.set(None) + meta_token = _last_tool_metadata.set(None) + try: + yield + finally: + _current_iteration.reset(iter_token) + _current_tool_call_seq.reset(seq_token) + _current_provider_tool_call_id.reset(pid_token) + _last_tool_metadata.reset(meta_token) + + +# embedding-call purpose ContextVar. Callers wrap embedding-driving +# operations in `with embedding_call_purpose("search_memory"): ...` so the +# embedding client can stamp every provider call with the originating intent +# without changing the call signature. None = caller didn't instrument; the +# event still emits but with call_purpose unset. +_embedding_call_purpose: ContextVar[str | None] = ContextVar( + "embedding_call_purpose", default=None +) +# Companion ContextVars so the event can also carry workspace + run +# correlation without threading kwargs through every embedding call site. +_embedding_workspace_name: ContextVar[str | None] = ContextVar( + "embedding_workspace_name", default=None +) +_embedding_run_id: ContextVar[str | None] = ContextVar("embedding_run_id", default=None) +# Parent category for joining EmbeddingCallCompletedEvent against the +# workflow that drove the call (e.g. "dialectic", "deriver", +# "reconciliation", "api"). Optional — None when uninstrumented. +_embedding_parent_category: ContextVar[str | None] = ContextVar( + "embedding_parent_category", default=None +) + + +def get_embedding_call_purpose() -> str | None: + """Read the current embedding call purpose. None when uninstrumented.""" + return _embedding_call_purpose.get() + + +def get_embedding_workspace_name() -> str | None: + """Read the workspace name attached to the current embedding call scope.""" + return _embedding_workspace_name.get() + + +def get_embedding_run_id() -> str | None: + """Read the run_id attached to the current embedding call scope.""" + return _embedding_run_id.get() + + +def get_embedding_parent_category() -> str | None: + """Read the parent category attached to the current embedding call scope.""" + return _embedding_parent_category.get() + + +@contextmanager +def embedding_call_purpose( + purpose: str, + *, + workspace_name: str | None = None, + run_id: str | None = None, + parent_category: str | None = None, +) -> Generator[None]: + """Tag any embedding calls made inside this `with` block. + + `purpose` should match an `EmbeddingCallPurpose` enum value (see + src/telemetry/events/llm.py). Unknown values pass through silently and + land as None on the event — the emitter validates against the enum. + + `workspace_name` and `run_id` let the emitted event correlate back to + a specific workspace and agent run. Both are optional; callers that + don't have one (or have it set further up the stack via a wider + `with` block) can omit it. + + `parent_category` joins the event back to the originating workflow — + typically the same category used by the calling LLM agent ("dialectic", + "deriver", "reconciliation", "api"). Lets analytics pivot embedding + cost/latency by workflow without per-purpose joins. + """ + purpose_token = _embedding_call_purpose.set(purpose) + workspace_token = ( + _embedding_workspace_name.set(workspace_name) + if workspace_name is not None + else None + ) + run_id_token = _embedding_run_id.set(run_id) if run_id is not None else None + parent_category_token = ( + _embedding_parent_category.set(parent_category) + if parent_category is not None + else None + ) + try: + yield + finally: + _embedding_call_purpose.reset(purpose_token) + if workspace_token is not None: + _embedding_workspace_name.reset(workspace_token) + if run_id_token is not None: + _embedding_run_id.reset(run_id_token) + if parent_category_token is not None: + _embedding_parent_category.reset(parent_category_token) + + +@dataclass +class ToolResult: + """Internal return shape used by tool handlers. + + Handlers may continue to return a plain `str` (existing contract). When + they need to carry structured metadata for downstream events — search + `top_k`/`results_count` for AgentToolCallCompletedEvent, or + `created_count`/`deleted_count` for specialist rollups — they + return `ToolResult(content=..., metadata={...})` instead. The + `execute_tool` closure in `create_tool_executor` unwraps the dataclass + before returning the string to `tool_loop`. + + Treat this as a private contract between agent_tools.py and tool_loop.py. + Public callers see only the string content. + + The `__contains__` and `__str__` overrides exist so direct handler-unit + tests that predate the dataclass — e.g. `assert "Created 2" in result` + — keep working without churning every assertion. Anything beyond + substring / str() (like `.lower()`) should access `.content` explicitly. + """ + + content: str + metadata: dict[str, Any] = field(default_factory=dict) + + def __contains__(self, item: object) -> bool: + # Only meaningful for substring checks; matches the legacy str-return + # contract used by handler unit tests. + if not isinstance(item, str): + return False + return item in self.content + + def __str__(self) -> str: + return self.content + + @dataclass class GetOrCreateResult(Generic[T]): """Result of a get_or_create operation indicating whether the resource was created.""" diff --git a/src/vector_store/turbopuffer.py b/src/vector_store/turbopuffer.py index f9adbb01..ded0c691 100644 --- a/src/vector_store/turbopuffer.py +++ b/src/vector_store/turbopuffer.py @@ -11,7 +11,7 @@ from typing import Any, Literal, cast from turbopuffer import AsyncTurbopuffer, InternalServerError, NotFoundError from turbopuffer.lib.namespace import AsyncNamespace -from turbopuffer.types import Filter +from turbopuffer.types import Filter, RowParam from src.config import settings from src.exceptions import VectorStoreError @@ -77,12 +77,23 @@ class TurbopufferVectorStore(VectorStore): ns = self._get_namespace(namespace) - rows: list[dict[str, Any]] = [ - { - "id": v.id, - "vector": v.embedding, - **(v.metadata or {}), - } + # The dict literal carries arbitrary metadata fields, which RowParam supports + # via extra_items=object. basedpyright can't see through the spread, so cast + # via object per its reportInvalidCast guidance. + # Spread metadata first so a caller-supplied "id" or "vector" key + # can never clobber the required upsert fields. + rows: list[RowParam] = [ + cast( + RowParam, + cast( + object, + { + **(v.metadata or {}), + "id": v.id, + "vector": v.embedding, + }, + ), + ) for v in vectors ] diff --git a/tests/bench/molecular.py b/tests/bench/molecular.py index ff1a6ca4..cf62fc7a 100644 --- a/tests/bench/molecular.py +++ b/tests/bench/molecular.py @@ -634,7 +634,7 @@ class MolecularJudge: "required": ["analyses"], } - props_text = "\n".join(f'{i+1}. "{p}"' for i, p in enumerate(propositions)) + props_text = "\n".join(f'{i + 1}. "{p}"' for i, p in enumerate(propositions)) result = await self._call_llm( AMBIGUITY_DETECTION_PROMPT, @@ -717,7 +717,7 @@ class MolecularJudge: "required": ["analyses"], } - props_text = "\n".join(f'{i+1}. "{p}"' for i, p in enumerate(propositions)) + props_text = "\n".join(f'{i + 1}. "{p}"' for i, p in enumerate(propositions)) result = await self._call_llm( DECONTEXTUALITY_PROMPT, @@ -814,7 +814,7 @@ class MolecularJudge: "required": ["analyses"], } - props_text = "\n".join(f'{i+1}. "{p}"' for i, p in enumerate(propositions)) + props_text = "\n".join(f'{i + 1}. "{p}"' for i, p in enumerate(propositions)) result = await self._call_llm( MINIMALITY_PROMPT, @@ -996,16 +996,16 @@ def print_report(report: MolecularReport) -> None: print("CLASSIFICATION DISTRIBUTION:") total = report.proposition_count print( - f" ✓ Molecular: {report.molecular_count:3d} ({report.molecular_count/total*100:5.1f}%)" + f" ✓ Molecular: {report.molecular_count:3d} ({report.molecular_count / total * 100:5.1f}%)" ) print( - f" ⚠ Too Atomic: {report.too_atomic_count:3d} ({report.too_atomic_count/total*100:5.1f}%)" + f" ⚠ Too Atomic: {report.too_atomic_count:3d} ({report.too_atomic_count / total * 100:5.1f}%)" ) print( - f" ⚠ Too Verbose: {report.too_verbose_count:3d} ({report.too_verbose_count/total*100:5.1f}%)" + f" ⚠ Too Verbose: {report.too_verbose_count:3d} ({report.too_verbose_count / total * 100:5.1f}%)" ) print( - f" ~ Borderline: {report.borderline_count:3d} ({report.borderline_count/total*100:5.1f}%)" + f" ~ Borderline: {report.borderline_count:3d} ({report.borderline_count / total * 100:5.1f}%)" ) # Show top issues @@ -1135,7 +1135,7 @@ async def main(): peer = extract_peer_name(trace) conv_id = extract_conversation_id(trace, idx) - print(f"[{idx+1}/{len(all_traces)}] {conv_id} ({len(props)} props)...") + print(f"[{idx + 1}/{len(all_traces)}] {conv_id} ({len(props)} props)...") try: report = await judge.evaluate(props, msgs, peer, conv_id) @@ -1208,7 +1208,7 @@ async def main(): print("\nClassification:") total_props = sum(r.proposition_count for r in results) for k, v in agg["classification_totals"].items(): - print(f" {k:<15} {v:4d} ({v/total_props*100:5.1f}%)") + print(f" {k:<15} {v:4d} ({v / total_props * 100:5.1f}%)") else: print(f"Duration: {format_duration(total_duration)}") else: diff --git a/tests/bench/oolong_common.py b/tests/bench/oolong_common.py index 31ffe2a6..c99842cc 100644 --- a/tests/bench/oolong_common.py +++ b/tests/bench/oolong_common.py @@ -652,20 +652,24 @@ def filter_dataset( if max_context_len is not None: dataset = dataset.filter( - lambda x: x.get( - "context_len", - calculate_context_length(str(x.get("context_window_text", ""))), + lambda x: ( + x.get( + "context_len", + calculate_context_length(str(x.get("context_window_text", ""))), + ) + <= max_context_len ) - <= max_context_len ) if min_context_len is not None: dataset = dataset.filter( - lambda x: x.get( - "context_len", - calculate_context_length(str(x.get("context_window_text", ""))), + lambda x: ( + x.get( + "context_len", + calculate_context_length(str(x.get("context_window_text", ""))), + ) + > min_context_len ) - > min_context_len ) if max_examples is not None and max_examples > 0: diff --git a/tests/bench/runner_common.py b/tests/bench/runner_common.py index 0f9840e7..be262cb0 100644 --- a/tests/bench/runner_common.py +++ b/tests/bench/runner_common.py @@ -99,7 +99,7 @@ def add_common_arguments(parser: argparse.ArgumentParser) -> None: "--base-url", type=str, default=None, - help="Base URL for remote Honcho instance (e.g., https://groudon.fly.dev). Overrides --base-api-port.", + help="Base URL for remote Honcho instance (e.g., https://api.example.com). Overrides --base-api-port.", ) parser.add_argument( diff --git a/tests/crud/test_representation_manager.py b/tests/crud/test_representation_manager.py index ff49ac07..141b61c7 100644 --- a/tests/crud/test_representation_manager.py +++ b/tests/crud/test_representation_manager.py @@ -1,6 +1,5 @@ from contextlib import asynccontextmanager from datetime import datetime, timezone -from types import SimpleNamespace from unittest.mock import AsyncMock, patch import pytest @@ -10,6 +9,13 @@ from sqlalchemy.ext.asyncio import AsyncSession from src import models from src.crud.representation import RepresentationManager +from src.schemas.configuration import ( + ResolvedConfiguration, + ResolvedDreamConfiguration, + ResolvedPeerCardConfiguration, + ResolvedReasoningConfiguration, + ResolvedSummaryConfiguration, +) from src.utils.representation import ( DeductiveObservation, ExplicitObservation, @@ -17,6 +23,20 @@ from src.utils.representation import ( ) +def _resolved_config(*, dream_enabled: bool = False) -> ResolvedConfiguration: + """Build a minimal ResolvedConfiguration for tests that only care about dream.enabled.""" + return ResolvedConfiguration( + reasoning=ResolvedReasoningConfiguration(enabled=False), + peer_card=ResolvedPeerCardConfiguration(use=False, create=False), + summary=ResolvedSummaryConfiguration( + enabled=False, + messages_per_short_summary=20, + messages_per_long_summary=60, + ), + dream=ResolvedDreamConfiguration(enabled=dream_enabled), + ) + + @asynccontextmanager async def _fake_tracked_db(_name: str): yield object() @@ -24,7 +44,7 @@ async def _fake_tracked_db(_name: str): def _saved_observations(mock_save: AsyncMock): call = mock_save.await_args - assert call is not None, "mock was not awaited" + assert call is not None, "mock_save was never awaited" if "all_observations" in call.kwargs: return call.kwargs["all_observations"] if len(call.args) > 1: @@ -205,9 +225,7 @@ class TestRepresentationManagerSave: message_ids=[1], session_name="session", message_created_at=datetime.now(timezone.utc), - message_level_configuration=SimpleNamespace( # pyright: ignore[reportArgumentType] - dream=SimpleNamespace(enabled=False) - ), + message_level_configuration=_resolved_config(), ) assert saved == 1 @@ -261,9 +279,7 @@ class TestRepresentationManagerSave: message_ids=[1], session_name="session", message_created_at=datetime.now(timezone.utc), - message_level_configuration=SimpleNamespace( # pyright: ignore[reportArgumentType] - dream=SimpleNamespace(enabled=False) - ), + message_level_configuration=_resolved_config(), ) assert saved == 1 @@ -314,9 +330,7 @@ class TestRepresentationManagerSave: message_ids=[1], session_name="session", message_created_at=datetime.now(timezone.utc), - message_level_configuration=SimpleNamespace( # pyright: ignore[reportArgumentType] - dream=SimpleNamespace(enabled=False) - ), + message_level_configuration=_resolved_config(), ) assert saved == 0 diff --git a/tests/deriver/test_deriver_processing.py b/tests/deriver/test_deriver_processing.py index 431aec2b..a2058bf9 100644 --- a/tests/deriver/test_deriver_processing.py +++ b/tests/deriver/test_deriver_processing.py @@ -210,6 +210,112 @@ class TestDeriverProcessing: # Verify the methods were called assert mock_representation_manager.save_representation.called # type: ignore[attr-defined] + async def test_warns_when_response_input_tokens_less_than_messages_tokens( + self, + caplog: pytest.LogCaptureFixture, + ) -> None: + """Data-quality invariant: provider should report at least as many + input tokens as we summed from messages. Drift surfaces as a WARNING + so analytics alerting can catch it.""" + import logging + + message = Mock( + id=1, + public_id="msg_drift", + session_name="session-1", + workspace_name="workspace-1", + peer_name="alice", + content="hello", + token_count=100, + created_at=datetime.now(timezone.utc), + ) + configuration = Mock() + configuration.reasoning.enabled = True + + mock_response = HonchoLLMCallResponse( + content=PromptRepresentation(explicit=[]), + input_tokens=10, + output_tokens=5, + finish_reasons=["STOP"], + ) + + with ( + patch( + "src.deriver.deriver.honcho_llm_call", + new_callable=AsyncMock, + return_value=mock_response, + ), + caplog.at_level(logging.WARNING, logger="src.deriver.deriver"), + ): + await process_representation_tasks_batch( + messages=[message], + message_level_configuration=configuration, + observers=["bob"], + observed="alice", + queue_item_message_ids=[1], + ) + + assert any( + "token-breakdown invariant violated" in record.message + and record.levelno == logging.WARNING + for record in caplog.records + ) + + async def test_warns_when_prompt_scaffold_tokens_is_zero( + self, + caplog: pytest.LogCaptureFixture, + ) -> None: + """Data-quality invariant: prompt scaffold estimator returning 0 + signals a silent failure — log WARNING so the metric pipeline can + alert.""" + import logging + + message = Mock( + id=1, + public_id="msg_scaffold_zero", + session_name="session-1", + workspace_name="workspace-1", + peer_name="alice", + content="hello", + token_count=5, + created_at=datetime.now(timezone.utc), + ) + configuration = Mock() + configuration.reasoning.enabled = True + + mock_response = HonchoLLMCallResponse( + content=PromptRepresentation(explicit=[]), + input_tokens=100, + output_tokens=5, + finish_reasons=["STOP"], + ) + + with ( + patch( + "src.deriver.deriver.honcho_llm_call", + new_callable=AsyncMock, + return_value=mock_response, + ), + patch( + "src.deriver.deriver.estimate_deriver_prompt_tokens", + return_value=0, + ), + caplog.at_level(logging.WARNING, logger="src.deriver.deriver"), + ): + await process_representation_tasks_batch( + messages=[message], + message_level_configuration=configuration, + observers=["bob"], + observed="alice", + queue_item_message_ids=[1], + ) + + assert any( + "prompt_scaffold_tokens estimated as 0" in record.message + and record.levelno == logging.WARNING + for record in caplog.records + ) + class TestBackwardsCompatibility: """Test backwards compatibility for queue items created before the deduplication change.""" diff --git a/tests/deriver/test_queue_processing.py b/tests/deriver/test_queue_processing.py index 540dfab5..c293f52c 100644 --- a/tests/deriver/test_queue_processing.py +++ b/tests/deriver/test_queue_processing.py @@ -58,7 +58,7 @@ class TestQueueProcessing: async def test_work_unit_claiming( self, db_session: AsyncSession, - sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] + sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] sample_session_with_peers: tuple[models.Session, list[models.Peer]], ) -> None: """Test that work units can be claimed and are not available to other workers""" @@ -92,7 +92,7 @@ class TestQueueProcessing: @pytest.mark.asyncio async def test_get_and_claim_excludes_already_claimed( self, - sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] + sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] ) -> None: queue_manager = QueueManager() first_batch = await queue_manager.get_and_claim_work_units() @@ -106,7 +106,7 @@ class TestQueueProcessing: async def test_claim_work_unit_conflict_returns_false( self, db_session: AsyncSession, - sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] + sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] ) -> None: # Pre-create an active session for a key queue_manager = QueueManager() @@ -183,29 +183,31 @@ class TestQueueProcessing: await db_session.commit() await db_session.refresh(aqs) - _, items_to_process, _ = await qm.get_queue_item_batch( + batch = await qm.get_queue_item_batch( task_type="representation", work_unit_key=first.work_unit_key, aqs_id=aqs.id, ) + items_to_process = batch.items_to_process nxt = items_to_process[0] if items_to_process else None assert nxt is not None and nxt.id == first.id # Mark first processed, next should be the second first.processed = True await db_session.commit() - _, items_to_process2, _ = await qm.get_queue_item_batch( + batch2 = await qm.get_queue_item_batch( task_type="representation", work_unit_key=first.work_unit_key, aqs_id=aqs.id, ) + items_to_process2 = batch2.items_to_process nxt2 = items_to_process2[0] if items_to_process2 else None assert nxt2 is not None and nxt2.id == second.id @pytest.mark.asyncio async def test_cleanup_work_unit_removes_row( self, - sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] + sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter] db_session: AsyncSession, ) -> None: qm = QueueManager() @@ -358,6 +360,7 @@ class TestQueueProcessing: observed: str | None = None, # pyright: ignore[reportUnusedParameter] observers: list[str] | None = None, # pyright: ignore[reportUnusedParameter] queue_item_message_ids: list[int] | None = None, # pyright: ignore[reportUnusedParameter] + **_extra: Any, # added hit_batch_token_cap / was_flush_enabled / batch_max_tokens ) -> None: processed_batches.append( { @@ -391,6 +394,203 @@ class TestQueueProcessing: assert processed_batches[1]["payload_count"] == 1 assert all(b["task_type"] == "representation" for b in processed_batches) + @pytest.mark.asyncio + async def test_hit_batch_token_cap_reflects_post_filter_batch( + self, + db_session: AsyncSession, + sample_session_with_peers: tuple[models.Session, list[models.Peer]], + create_queue_payload: Callable[..., Any], + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Regression: `hit_batch_token_cap` must reflect the actually-returned + batch (post config-filter), not the pre-filter superset. Previously + the flag used pre-filter `messages_context[-1].id`, which inflated + the range queried for cap detection and produced false positives + when the config-filter trimmed the trailing item from the batch. + """ + from src.deriver import queue_manager as qm_module + + session, peers = sample_session_with_peers + peer = peers[0] + cap = settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS + + # M1 + M2 sum to exactly the cap; M3 pushes over it. After SQL, + # messages_context = [M1, M2]; the cap is genuinely binding *on the + # pre-filter batch*. items_to_process = [QI(M1), QI(M2)]. We then + # simulate a config-filter trim that keeps only QI(M1). The + # actually-returned batch is [M1] alone — sum=400 < cap — so the + # cap-flag must report False. + token_counts = [400, cap - 400, 300] + messages: list[models.Message] = [] + for i, tc in enumerate(token_counts): + m = models.Message( + session_name=session.name, + workspace_name=session.workspace_name, + peer_name=peer.name, + content=f"cap-test message {i}", + token_count=tc, + seq_in_session=i + 1, + ) + db_session.add(m) + messages.append(m) + await db_session.commit() + for m in messages: + await db_session.refresh(m) + + queue_items: list[models.QueueItem] = [] + for m in messages: + payload = create_queue_payload( # type: ignore[reportUnknownArgumentType] + message=m, + task_type="representation", + observed=peer.name, + observer=peer.name, + ) + work_unit_key = construct_work_unit_key(session.workspace_name, payload) + qi = models.QueueItem( + session_id=session.id, + task_type="representation", + work_unit_key=work_unit_key, + payload=payload, + processed=False, + workspace_name=session.workspace_name, + message_id=m.id, + ) + db_session.add(qi) + queue_items.append(qi) + await db_session.commit() + for qi in queue_items: + await db_session.refresh(qi) + + # Trim items_to_process down to the first queue item — mimics + # `_resolve_batch_configuration` cutting at a configuration boundary. + real_resolve = qm_module._resolve_batch_configuration # pyright: ignore[reportPrivateUsage] + + def fake_resolve( + items: list[models.QueueItem], + ) -> tuple[list[models.QueueItem], Any]: + _kept, cfg = real_resolve(items) + return (items[:1] if items else []), cfg + + monkeypatch.setattr(qm_module, "_resolve_batch_configuration", fake_resolve) + + qm = qm_module.QueueManager() + work_unit_key = queue_items[0].work_unit_key + claimed = await qm.claim_work_units(db_session, [work_unit_key]) + aqs_id = claimed[work_unit_key] + await db_session.commit() + + result = await qm.get_queue_item_batch( + task_type="representation", + work_unit_key=work_unit_key, + aqs_id=aqs_id, + ) + + # Returned batch is [M1] alone (config-filter trimmed M2). The cap + # wasn't binding on this batch — sum=400 < cap. Pre-fix code reported + # True (false positive) because it used pre-filter max_kept_id=M2. + assert len(result.messages_context) == 1 + assert result.messages_context[0].id == messages[0].id + assert result.hit_batch_token_cap is False + + @pytest.mark.asyncio + async def test_hit_batch_token_cap_fires_when_trailing_context_trimmed( + self, + db_session: AsyncSession, + sample_session_with_peers: tuple[models.Session, list[models.Peer]], + create_queue_payload: Callable[..., Any], + ) -> None: + """Regression: when SQL kept trailing NON-QUEUE context past the last + queued item, and the config filter trims that context away, the cap + check must still recognize that the SQL cap clamped queue work. + + Pre-fix used `messages_context[-1].id == sql_max_kept_id` which goes + False whenever trailing context is dropped — producing a false + negative for the very case the cap-hit flag exists to report. + Post-fix keys on the queue-item boundary, which is unaffected by + trailing-context trimming. + """ + from src.deriver import queue_manager as qm_module + + session, peers = sample_session_with_peers + peer_a = peers[0] + peer_b = peers[1] if len(peers) > 1 else peers[0] + cap = settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS + + # Layout: 4 messages, ordered. + # M1 (peer_a, queue, 200) + # M2 (peer_a, queue, 200) + # M3 (peer_b, NON-queue context, cap - 300) + # M4 (peer_a, queue, 400) + # Cumulative tokens: M1=200, M2=400, M3=cap+100, M4=cap+500. + # SQL keeps M1+M2 (cumulative <= cap), excludes M3 onwards (over cap). + # Wait — we want SQL to keep through M3 (trailing context) but exclude + # M4 (queue). Adjust so M3 fits but M4 doesn't. + token_counts: list[tuple[models.Peer, int]] = [ + (peer_a, 200), # M1 — queue + (peer_a, 200), # M2 — queue + (peer_b, cap - 700), # M3 — non-queue context; cumulative = cap-300 + (peer_a, 400), # M4 — queue; cumulative cap+100 > cap → excluded + ] + messages: list[models.Message] = [] + for i, (msg_peer, tc) in enumerate(token_counts): + m = models.Message( + session_name=session.name, + workspace_name=session.workspace_name, + peer_name=msg_peer.name, + content=f"cap-test message {i}", + token_count=tc, + seq_in_session=i + 1, + ) + db_session.add(m) + messages.append(m) + await db_session.commit() + for m in messages: + await db_session.refresh(m) + + # Queue items for M1, M2, M4 only — M3 is non-queue context (peer_b). + # observed = peer_a, so the deriver's representation work unit covers + # peer_a messages; peer_b is treated as conversational context. + queue_items: list[models.QueueItem] = [] + for m in [messages[0], messages[1], messages[3]]: + payload = create_queue_payload( # type: ignore[reportUnknownArgumentType] + message=m, + task_type="representation", + observed=peer_a.name, + observer=peer_a.name, + ) + work_unit_key = construct_work_unit_key(session.workspace_name, payload) + qi = models.QueueItem( + session_id=session.id, + task_type="representation", + work_unit_key=work_unit_key, + payload=payload, + processed=False, + workspace_name=session.workspace_name, + message_id=m.id, + ) + db_session.add(qi) + queue_items.append(qi) + await db_session.commit() + for qi in queue_items: + await db_session.refresh(qi) + + qm = qm_module.QueueManager() + work_unit_key = queue_items[0].work_unit_key + claimed = await qm.claim_work_units(db_session, [work_unit_key]) + aqs_id = claimed[work_unit_key] + await db_session.commit() + + result = await qm.get_queue_item_batch( + task_type="representation", + work_unit_key=work_unit_key, + aqs_id=aqs_id, + ) + + # Cap fired: SQL stopped at M3 (cap budget exhausted), excluding the + # queue item M4. Config filter doesn't touch queue items here. The + # flag must report True. + assert result.hit_batch_token_cap is True + @pytest.mark.asyncio async def test_token_batching_filters_by_work_unit( self, @@ -485,11 +685,13 @@ class TestQueueProcessing: await db_session.commit() await db_session.refresh(alice_aqs) - alice_messages, alice_items, _ = await qm.get_queue_item_batch( + alice_batch = await qm.get_queue_item_batch( task_type="representation", work_unit_key=alice_work_unit_key, aqs_id=alice_aqs.id, ) + alice_messages = alice_batch.messages_context + alice_items = alice_batch.items_to_process assert len(alice_messages) == 6 alice_message_ids: set[int] = {m.id for m in alice_messages} @@ -513,11 +715,13 @@ class TestQueueProcessing: await db_session.commit() await db_session.refresh(bob_aqs) - bob_messages, bob_items, _ = await qm.get_queue_item_batch( + bob_batch = await qm.get_queue_item_batch( task_type="representation", work_unit_key=bob_work_unit_key, aqs_id=bob_aqs.id, ) + bob_messages = bob_batch.messages_context + bob_items = bob_batch.items_to_process # Bob should get 5 messages (1..5) - includes preceding alice message for context assert len(bob_messages) == 5 @@ -540,11 +744,13 @@ class TestQueueProcessing: await db_session.commit() await db_session.refresh(steve_aqs) - steve_messages, steve_items, _ = await qm.get_queue_item_batch( + steve_batch = await qm.get_queue_item_batch( task_type="representation", work_unit_key=steve_work_unit_key, aqs_id=steve_aqs.id, ) + steve_messages = steve_batch.messages_context + steve_items = steve_batch.items_to_process # Steve should get 6 messages (2..7) - includes preceding bob message for context assert len(steve_messages) == 6 @@ -658,11 +864,12 @@ class TestQueueProcessing: await db_session.commit() await db_session.refresh(alice_aqs) - alice_messages2, _, _ = await qm.get_queue_item_batch( + alice_batch2 = await qm.get_queue_item_batch( task_type="representation", work_unit_key=alice_work_unit_key, aqs_id=alice_aqs.id, ) + alice_messages2 = alice_batch2.messages_context # Includes preceding steve message for context -> [2,3,4] assert len(alice_messages2) == 3 @@ -682,11 +889,12 @@ class TestQueueProcessing: await db_session.commit() await db_session.refresh(bob_aqs) - bob_messages2, _, _ = await qm.get_queue_item_batch( + bob_batch2 = await qm.get_queue_item_batch( task_type="representation", work_unit_key=bob_work_unit_key, aqs_id=bob_aqs.id, ) + bob_messages2 = bob_batch2.messages_context assert len(bob_messages2) == 1 assert bob_messages2[0].id == messages[0].id # bob only @@ -702,11 +910,12 @@ class TestQueueProcessing: await db_session.commit() await db_session.refresh(steve_aqs) - steve_messages2, _, _ = await qm.get_queue_item_batch( + steve_batch2 = await qm.get_queue_item_batch( task_type="representation", work_unit_key=steve_work_unit_key, aqs_id=steve_aqs.id, ) + steve_messages2 = steve_batch2.messages_context # Includes preceding bob message for context -> [1,2] assert len(steve_messages2) == 2 @@ -926,6 +1135,7 @@ class TestQueueProcessing: observed: str | None = None, # pyright: ignore[reportUnusedParameter] observers: list[str] | None = None, # pyright: ignore[reportUnusedParameter] queue_item_message_ids: list[int] | None = None, # pyright: ignore[reportUnusedParameter] + **_extra: Any, # added hit_batch_token_cap / was_flush_enabled / batch_max_tokens ) -> None: processed_batches.append( { @@ -1046,6 +1256,7 @@ class TestQueueProcessing: observed: str | None = None, # pyright: ignore[reportUnusedParameter] observers: list[str] | None = None, # pyright: ignore[reportUnusedParameter] queue_item_message_ids: list[int] | None = None, # pyright: ignore[reportUnusedParameter] + **_extra: Any, # added hit_batch_token_cap / was_flush_enabled / batch_max_tokens ) -> None: processed_batches.append( { diff --git a/tests/dreamer/test_dream_v2_rollups.py b/tests/dreamer/test_dream_v2_rollups.py new file mode 100644 index 00000000..a20ab04d --- /dev/null +++ b/tests/dreamer/test_dream_v2_rollups.py @@ -0,0 +1,183 @@ +# pyright: reportPrivateUsage=false +"""tests: DreamSpecialistEvent + DreamRunEvent v2 fields. + +Targets: +- Schema bumps to v2 (DreamRunEvent + DreamSpecialistEvent). +- Specialist rollups come from `tool_result_metadata`, NOT tool-name counting. +- DreamRunEvent carries the scheduler reasons threaded through the dream + queue payload. +""" + +from __future__ import annotations + +from src.telemetry.events.dream import DreamRunEvent, DreamSpecialistEvent +from src.utils.queue_payload import DreamPayload + + +class TestSchemaVersionsBumpedToV2: + def test_dream_run_event_at_v2(self): + assert DreamRunEvent.schema_version() == 2 + + def test_dream_specialist_event_at_v2(self): + assert DreamSpecialistEvent.schema_version() == 2 + + +class TestDreamRunEventV2Fields: + def test_scheduler_fields_default(self): + """Existing callers that don't supply scheduling fields must + still construct a valid event — all new fields default to None or 0.""" + event = DreamRunEvent( + run_id="abc", + workspace_name="ws", + session_name=None, + observer="o", + observed="user", + specialists_run=["deduction", "induction"], + deduction_success=True, + induction_success=True, + total_iterations=10, + total_input_tokens=100, + total_output_tokens=20, + total_duration_ms=1000.0, + ) + assert event.dream_type is None + assert event.enabled_types_count == 0 + assert event.trigger_reason is None + assert event.delay_reason is None + assert event.documents_since_last_dream_at_schedule is None + assert event.document_threshold is None + + def test_scheduler_reasons_round_trip(self): + event = DreamRunEvent( + run_id="abc", + workspace_name="ws", + session_name=None, + observer="o", + observed="user", + specialists_run=["deduction"], + deduction_success=True, + induction_success=False, + total_iterations=5, + total_input_tokens=50, + total_output_tokens=10, + total_duration_ms=500.0, + dream_type="omni", + enabled_types_count=1, + trigger_reason="document_threshold", + delay_reason="idle_timeout", + documents_since_last_dream_at_schedule=60, + document_threshold=50, + ) + assert event.dream_type == "omni" + assert event.trigger_reason == "document_threshold" + assert event.delay_reason == "idle_timeout" + assert event.documents_since_last_dream_at_schedule == 60 + assert event.document_threshold == 50 + + def test_threshold_and_delay_are_separate(self): + """The two scheduler gates are intentionally separate fields. The + snapshot semantics differ: trigger_reason describes WHY the dream + was scheduled (which gate tripped); delay_reason describes WHEN it + will fire (idle vs immediate).""" + event = DreamRunEvent( + run_id="abc", + workspace_name="ws", + session_name=None, + observer="o", + observed="user", + specialists_run=["induction"], + deduction_success=False, + induction_success=True, + total_iterations=3, + total_input_tokens=30, + total_output_tokens=5, + total_duration_ms=100.0, + trigger_reason="document_threshold", + delay_reason="immediate", + ) + # trigger_reason captures the WHY; delay_reason captures the WHEN. + # They are separate dimensions — flattening into one field would lose + # the gate semantics that was specifically designed to expose. + assert event.trigger_reason != event.delay_reason + + +class TestDreamSpecialistEventV2Rollups: + def test_rollup_fields_default(self): + event = DreamSpecialistEvent( + run_id="abc", + specialist_type="deduction", + workspace_name="ws", + observer="o", + observed="user", + iterations=3, + tool_calls_count=5, + input_tokens=100, + output_tokens=20, + duration_ms=500.0, + success=True, + ) + assert event.created_observation_count == 0 + assert event.deleted_observation_count == 0 + assert event.peer_card_updated is False + assert event.search_tool_calls_count == 0 + + def test_observation_counts_are_observation_truth_not_call_counts(self): + """The whole point of sourcing rollups from ToolResult.metadata + instead of tool-name counts: a single `create_observations` call can + produce N observations (or zero on validation failure). must + report observation truth, not call truth.""" + event = DreamSpecialistEvent( + run_id="abc", + specialist_type="deduction", + workspace_name="ws", + observer="o", + observed="user", + iterations=2, + # Two create_observations CALLS, but they produced 7 observations + # together (e.g. one batch of 5, one batch of 2). reports + # 7 (the metadata-sourced truth), not 2 (the call count). + tool_calls_count=2, + input_tokens=100, + output_tokens=20, + duration_ms=500.0, + success=True, + created_observation_count=7, + ) + assert event.tool_calls_count == 2 + assert event.created_observation_count == 7 + + +class TestDreamPayloadSchedulerFields: + def test_payload_defaults(self): + from src.schemas import DreamType + + payload = DreamPayload( + dream_type=DreamType.OMNI, + observer="o", + observed="user", + ) + assert payload.trigger_reason is None + assert payload.delay_reason is None + assert payload.documents_since_last_dream_at_schedule is None + assert payload.document_threshold is None + + def test_payload_threads_scheduler_reasons(self): + from src.schemas import DreamType + + payload = DreamPayload( + dream_type=DreamType.OMNI, + observer="o", + observed="user", + trigger_reason="document_threshold", + delay_reason="idle_timeout", + documents_since_last_dream_at_schedule=55, + document_threshold=50, + ) + # Round-trip through serialization → deserialization mimics what + # happens between scheduler enqueue and consumer dequeue. + data = payload.model_dump(mode="json") + restored = DreamPayload(**data) + assert restored.trigger_reason == "document_threshold" + assert restored.delay_reason == "idle_timeout" + assert restored.documents_since_last_dream_at_schedule == 55 + assert restored.document_threshold == 50 diff --git a/tests/dreamer/test_dreamer_integration.py b/tests/dreamer/test_dreamer_integration.py index 5f60196a..fb9b1894 100644 --- a/tests/dreamer/test_dreamer_integration.py +++ b/tests/dreamer/test_dreamer_integration.py @@ -368,6 +368,7 @@ class TestExecuteDreamSessionFilter: observed: str, dream_type: Any, session_name: str, + **_scheduler_extra: Any, # trigger_reason / delay_reason / etc. ) -> None: captured_kwargs.update( { diff --git a/tests/integration/test_telemetry.py b/tests/integration/test_telemetry.py index 9a627885..615a602a 100644 --- a/tests/integration/test_telemetry.py +++ b/tests/integration/test_telemetry.py @@ -142,6 +142,7 @@ def create_representation_event( llm_call_ms=1200.0, total_duration_ms=1300.0, input_tokens=5000, + total_input_tokens=7500, output_tokens=500, ) diff --git a/tests/llm/test_telemetry_agent_iteration.py b/tests/llm/test_telemetry_agent_iteration.py new file mode 100644 index 00000000..382cc2e3 --- /dev/null +++ b/tests/llm/test_telemetry_agent_iteration.py @@ -0,0 +1,295 @@ +# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false +"""tests for AgentIterationEvent emission. + +Targets: +- `src/llm/tool_loop.py::_emit_agent_iteration` fires one event per LLM + response, including the no-tool terminating iteration and the max-iteration + synthesis call. +- Emission is skipped when telemetry context is missing or under-specified + (no agent_type / parent_category / workspace / run_id). +- The caller-supplied `LLMTelemetryContext` is never mutated; per-iteration + copies set the iteration field on a fresh instance. +""" + +from __future__ import annotations + +from typing import Any +from unittest.mock import patch + +from src.llm.tool_loop import ( + _emit_agent_iteration, + _telemetry_for_iteration, +) +from src.llm.types import HonchoLLMCallResponse, LLMTelemetryContext +from src.telemetry.events import AgentIterationEvent, BaseEvent + + +def _response( + *, + tool_calls: list[dict[str, Any]] | None = None, + input_tokens: int = 100, + output_tokens: int = 25, + cache_read: int = 0, + cache_creation: int = 0, +) -> HonchoLLMCallResponse[Any]: + return HonchoLLMCallResponse( + content="", + input_tokens=input_tokens, + output_tokens=output_tokens, + cache_read_input_tokens=cache_read, + cache_creation_input_tokens=cache_creation, + finish_reasons=["stop"], + tool_calls_made=tool_calls or [], + ) + + +class TestTelemetryForIteration: + def test_returns_none_when_base_is_none(self): + assert _telemetry_for_iteration(None, 1) is None + + def test_returns_fresh_copy_with_iteration_set(self): + base = LLMTelemetryContext( + workspace_name="ws", + call_purpose="dialectic.answer", + parent_category="dialectic", + agent_type="dialectic", + run_id="run-xyz", + iteration=None, + peer_name="user_peer", + ) + + copy_a = _telemetry_for_iteration(base, 3) + copy_b = _telemetry_for_iteration(base, 4) + + assert copy_a is not None and copy_b is not None + assert copy_a is not base and copy_b is not base + # Original is never mutated. + assert base.iteration is None + assert copy_a.iteration == 3 + assert copy_b.iteration == 4 + # All other fields round-trip. + assert copy_a.run_id == "run-xyz" + assert copy_a.peer_name == "user_peer" + + +class TestEmitAgentIteration: + def test_emits_event_with_tool_calls(self): + emitted: list[BaseEvent] = [] + telemetry = LLMTelemetryContext( + workspace_name="ws", + parent_category="dream", + agent_type="deduction", + run_id="run-1", + observer="obs", + observed="obj", + ) + response = _response( + tool_calls=[ + {"name": "search_memory", "id": "t1", "input": {}}, + {"name": "create_observations", "id": "t2", "input": {}}, + ], + input_tokens=500, + output_tokens=80, + cache_read=12, + cache_creation=5, + ) + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_iteration(telemetry, iteration=2, response=response) + + assert len(emitted) == 1 + event = emitted[0] + assert isinstance(event, AgentIterationEvent) + assert event.run_id == "run-1" + assert event.parent_category == "dream" + assert event.agent_type == "deduction" + assert event.workspace_name == "ws" + assert event.observer == "obs" + assert event.observed == "obj" + assert event.iteration == 2 + assert event.tool_calls == ["search_memory", "create_observations"] + assert event.input_tokens == 500 + assert event.output_tokens == 80 + assert event.cache_read_tokens == 12 + assert event.cache_creation_tokens == 5 + + def test_emits_terminating_iteration_with_empty_tool_calls(self): + """The no-tool terminating iteration still counts. Empty tool_calls + list must produce a valid AgentIterationEvent.""" + emitted: list[BaseEvent] = [] + telemetry = LLMTelemetryContext( + workspace_name="ws", + parent_category="dialectic", + agent_type="dialectic", + run_id="run-2", + peer_name="user_peer", + ) + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_iteration(telemetry, iteration=4, response=_response()) + + assert len(emitted) == 1 + event = emitted[0] + assert isinstance(event, AgentIterationEvent) + assert event.tool_calls == [] + assert event.iteration == 4 + + def test_skips_when_telemetry_is_none(self): + emitted: list[BaseEvent] = [] + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_iteration(None, iteration=1, response=_response()) + assert emitted == [] + + def test_skips_when_run_id_missing(self): + emitted: list[BaseEvent] = [] + telemetry = LLMTelemetryContext( + workspace_name="ws", + parent_category="dialectic", + agent_type="dialectic", + run_id=None, + ) + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_iteration(telemetry, iteration=1, response=_response()) + assert emitted == [] + + def test_skips_when_agent_type_or_parent_category_missing(self): + """LLMCallCompletedEvent can fire without agent fields (system call), + but agent.iteration is by definition an agent-loop event. If agent + metadata is missing, skip emission rather than send a half-populated + event.""" + emitted: list[BaseEvent] = [] + telemetry = LLMTelemetryContext( + workspace_name="ws", + parent_category=None, + agent_type="dialectic", + run_id="run-3", + ) + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_iteration(telemetry, iteration=1, response=_response()) + assert emitted == [] + + def test_skips_when_workspace_missing(self): + emitted: list[BaseEvent] = [] + telemetry = LLMTelemetryContext( + workspace_name=None, + parent_category="dream", + agent_type="induction", + run_id="run-4", + ) + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_iteration(telemetry, iteration=1, response=_response()) + assert emitted == [] + + def test_swallows_emit_failures(self): + """Telemetry failures must not bleed into the LLM call path.""" + + def explode(*_args: Any, **_kwargs: Any) -> None: + raise RuntimeError("emitter wedged") + + telemetry = LLMTelemetryContext( + workspace_name="ws", + parent_category="dream", + agent_type="deduction", + run_id="run-5", + ) + with patch("src.telemetry.events.emit", side_effect=explode): + # Must not raise. + _emit_agent_iteration(telemetry, iteration=1, response=_response()) + + +def test_volume_class_is_high_volume(): + """emission targets a high-volume event class so the sampler + can throttle iteration events independently of aggregates.""" + assert AgentIterationEvent.volume_class() == "high_volume" + + +class TestIterationScope: + """`iteration_scope()` in src/utils/types.py captures Tokens for the + per-loop ContextVars and resets them on exit. Defensive against a + subsequent tool loop in the same asyncio Task seeing stale state from + a previous loop (worker batches, tests using TestClient). + """ + + def test_resets_iteration_and_tool_call_state_on_exit(self): + from src.utils.types import ( + get_current_iteration, + get_current_provider_tool_call_id, + get_current_tool_call_seq, + get_last_tool_metadata, + iteration_scope, + set_current_iteration, + set_current_tool_call_seq, + set_last_tool_metadata, + ) + + # Pre-scope: defaults. + assert get_current_iteration() == 0 + assert get_current_tool_call_seq() == 0 + assert get_current_provider_tool_call_id() is None + assert get_last_tool_metadata() == {} + + with iteration_scope(): + set_current_iteration(7) + set_current_tool_call_seq(3, "toolu_abc") + set_last_tool_metadata({"k": "v"}) + assert get_current_iteration() == 7 + assert get_current_tool_call_seq() == 3 + assert get_current_provider_tool_call_id() == "toolu_abc" + assert get_last_tool_metadata() == {"k": "v"} + + # Post-scope: every ContextVar reset to pre-scope state. + assert get_current_iteration() == 0 + assert get_current_tool_call_seq() == 0 + assert get_current_provider_tool_call_id() is None + assert get_last_tool_metadata() == {} + + def test_resets_on_exception(self): + """Exception inside the block still triggers the reset path.""" + import pytest + + from src.utils.types import ( + get_current_iteration, + iteration_scope, + set_current_iteration, + ) + + with pytest.raises(RuntimeError, match="boom"), iteration_scope(): + set_current_iteration(5) + raise RuntimeError("boom") + + assert get_current_iteration() == 0 + + def test_sequential_scopes_do_not_leak(self): + """Two back-to-back scopes (mimicking sequential tool loops) — the + second sees a clean baseline, not stale values from the first.""" + from src.utils.types import ( + get_current_iteration, + iteration_scope, + set_current_iteration, + ) + + with iteration_scope(): + set_current_iteration(9) + + with iteration_scope(): + # Inside scope #2: iteration starts at 0 (the scope reset it), + # not at 9 from the prior scope. + assert get_current_iteration() == 0 diff --git a/tests/llm/test_telemetry_agent_tool_call.py b/tests/llm/test_telemetry_agent_tool_call.py new file mode 100644 index 00000000..9bc8aa4d --- /dev/null +++ b/tests/llm/test_telemetry_agent_tool_call.py @@ -0,0 +1,355 @@ +# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false, reportUnusedFunction=false +"""tests for AgentToolCallCompletedEvent emission. + +Targets: +- `_emit_agent_tool_call_completed` in `src/utils/agent_tools.py` builds a + well-formed event from ToolContext + per-call metadata. +- ToolResult dataclass behaves like a string for `in` / `str()` so tests of + the existing handler contract keep working. +- The `tool_call_seq` ContextVar disambiguates resource ids when the same + tool is called twice in one iteration. +- event opts into the high-volume sampler. +- Search handlers publish search-specific metadata + (top_k / used_embedding / query_tokens / results_count) so analytics can + filter by retrieval intent. +""" + +from __future__ import annotations + +from collections.abc import Generator +from typing import Any, final +from unittest.mock import patch + +import pytest + +from src.telemetry.events import AgentToolCallCompletedEvent, BaseEvent +from src.utils.agent_tools import _emit_agent_tool_call_completed +from src.utils.types import ( + ToolResult, + get_current_provider_tool_call_id, + get_current_tool_call_seq, + set_current_tool_call_seq, +) + + +@pytest.fixture(autouse=True) +def _reset_tool_call_contextvars() -> Generator[None]: + """Restore the tool-call ContextVars after each test. + + `set_current_tool_call_seq` mutates module-level state; without this + fixture, tests that read `get_current_tool_call_seq()` expecting the + default 0 become order-dependent on whichever test ran before. + """ + prev_seq = get_current_tool_call_seq() + prev_provider_id = get_current_provider_tool_call_id() + try: + yield + finally: + set_current_tool_call_seq(prev_seq, prev_provider_id) + + +@final +class _StubToolContext: + """Duck-typed ToolContext stand-in — emitter only reads identifiers.""" + + workspace_name: str + run_id: str | None + agent_type: str | None + parent_category: str | None + + def __init__( + self, + *, + workspace_name: str = "ws", + run_id: str | None = "run-1", + agent_type: str | None = "dialectic", + parent_category: str | None = "dialectic", + ): + self.workspace_name = workspace_name + self.run_id = run_id + self.agent_type = agent_type + self.parent_category = parent_category + + +class TestToolResult: + def test_str_returns_content(self): + result = ToolResult(content="hello", metadata={"x": 1}) + assert str(result) == "hello" + + def test_contains_delegates_to_content(self): + """ToolResult must satisfy existing 'substring in result' assertions.""" + result = ToolResult(content="Created 3 observations", metadata={}) + assert "Created 3" in result + assert "missing" not in result + + def test_metadata_defaults_to_empty(self): + result = ToolResult(content="x") + assert result.metadata == {} + + +class TestToolCallSeqContextVar: + def test_seq_and_provider_id_round_trip(self): + set_current_tool_call_seq(2, "toolu_abc") + assert get_current_tool_call_seq() == 2 + assert get_current_provider_tool_call_id() == "toolu_abc" + + def test_provider_id_can_be_none(self): + set_current_tool_call_seq(0, None) + assert get_current_tool_call_seq() == 0 + assert get_current_provider_tool_call_id() is None + + +class TestEmitAgentToolCallCompleted: + def test_emits_event_with_full_context(self): + emitted: list[BaseEvent] = [] + ctx = _StubToolContext( + run_id="run-7", agent_type="deduction", parent_category="dream" + ) + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_tool_call_completed( + ctx=ctx, + tool_name="search_memory", + duration_ms=42.5, + result_str="Found 5 observations", + metadata={ + "top_k": 20, + "used_embedding": True, + "embedding_query_count": 1, + "query_tokens": 7, + "results_count": 5, + }, + is_error=False, + iteration=3, + tool_call_seq=1, + provider_tool_call_id="toolu_xyz", + ) + + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, AgentToolCallCompletedEvent) + assert ev.run_id == "run-7" + assert ev.parent_category == "dream" + assert ev.agent_type == "deduction" + assert ev.workspace_name == "ws" + assert ev.iteration == 3 + assert ev.tool_call_seq == 1 + assert ev.provider_tool_call_id == "toolu_xyz" + assert ev.tool_name == "search_memory" + assert ev.duration_ms == 42.5 + assert ev.is_error is False + assert ev.result_chars == len("Found 5 observations") + # Search-specific fields surface from metadata. + assert ev.top_k == 20 + assert ev.used_embedding is True + assert ev.embedding_query_count == 1 + assert ev.query_tokens == 7 + assert ev.results_count == 5 + + def test_resource_id_disambiguates_same_tool_in_iteration(self): + """Resource id = {run_id}:{iteration}:{tool_call_seq}. Two calls to + the same tool in one iteration must produce DIFFERENT ids — otherwise + deterministic id generation would collide and dedupe would drop one + of the events.""" + ev_a = AgentToolCallCompletedEvent( + run_id="run-1", + iteration=2, + tool_call_seq=0, + parent_category="dialectic", + agent_type="dialectic", + workspace_name="ws", + tool_name="search_memory", + duration_ms=1.0, + result_chars=10, + result_tokens_estimate=3, + ) + ev_b = AgentToolCallCompletedEvent( + run_id="run-1", + iteration=2, + tool_call_seq=1, + parent_category="dialectic", + agent_type="dialectic", + workspace_name="ws", + tool_name="search_memory", + duration_ms=1.0, + result_chars=10, + result_tokens_estimate=3, + ) + assert ev_a.get_resource_id() != ev_b.get_resource_id() + # Same timestamp + different resource_id → different deterministic ids. + ev_b.timestamp = ev_a.timestamp + assert ev_a.generate_id() != ev_b.generate_id() + + def test_skips_when_run_id_missing(self): + emitted: list[BaseEvent] = [] + ctx = _StubToolContext(run_id=None) + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_tool_call_completed( + ctx=ctx, + tool_name="search_memory", + duration_ms=0.0, + result_str="", + metadata={}, + is_error=False, + iteration=1, + tool_call_seq=0, + provider_tool_call_id=None, + ) + assert emitted == [] + + def test_skips_when_agent_metadata_incomplete(self): + emitted: list[BaseEvent] = [] + ctx = _StubToolContext(agent_type=None) + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_tool_call_completed( + ctx=ctx, + tool_name="search_memory", + duration_ms=0.0, + result_str="", + metadata={}, + is_error=False, + iteration=1, + tool_call_seq=0, + provider_tool_call_id=None, + ) + assert emitted == [] + + def test_swallows_emit_failures(self): + """Telemetry must never bleed into the tool path.""" + + def explode(*_args: Any, **_kwargs: Any) -> None: + raise RuntimeError("emitter wedged") + + ctx = _StubToolContext() + with patch("src.telemetry.events.emit", side_effect=explode): + # Must not raise. + _emit_agent_tool_call_completed( + ctx=ctx, + tool_name="search_memory", + duration_ms=0.0, + result_str="", + metadata={}, + is_error=False, + iteration=1, + tool_call_seq=0, + provider_tool_call_id=None, + ) + + def test_truncation_metadata_round_trips(self): + emitted: list[BaseEvent] = [] + ctx = _StubToolContext() + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_tool_call_completed( + ctx=ctx, + tool_name="get_recent_history", + duration_ms=1.0, + result_str="abc", + metadata={ + "was_truncated": True, + "result_chars_before_truncation": 9000, + }, + is_error=False, + iteration=1, + tool_call_seq=0, + provider_tool_call_id=None, + ) + + ev = emitted[0] + assert isinstance(ev, AgentToolCallCompletedEvent) + assert ev.was_truncated is True + assert ev.result_chars_before_truncation == 9000 + # Truncation delta = before - after = 9000 - 3 = 8997. Calibration can + # compute that downstream; we just verify both fields land. + assert ev.result_chars == 3 + + +def test_volume_class_is_high_volume(): + """event is high-volume — sampled alongside llm.call.completed.""" + assert AgentToolCallCompletedEvent.volume_class() == "high_volume" + + +class TestMaybeTruncatedResult: + """`_maybe_truncated_result` should wrap in ToolResult only when the + helper actually clamps the output, and the wrapping must surface + the fields the AgentToolCallCompletedEvent reads from metadata. + + These tests guard against the regression where the truncation signal + used to be discarded by `_truncate_tool_output` returning bare str — + leaving `was_truncated` / `result_chars_before_truncation` as dead + fields on the event. + """ + + def test_under_cap_returns_bare_string(self): + """No-cap path keeps the bare-str contract — no metadata wrapping.""" + from src.utils.agent_tools import _maybe_truncated_result + + out = _maybe_truncated_result("hello") + assert out == "hello" + assert not isinstance(out, ToolResult) + + def test_over_cap_returns_tool_result_with_metadata(self): + """Truncated path wraps in ToolResult carrying the original size.""" + from src.config import settings + from src.utils.agent_tools import _maybe_truncated_result + + original = "x" * 5000 + with patch.object(settings.LLM, "MAX_TOOL_OUTPUT_CHARS", 100): + out = _maybe_truncated_result(original) + assert isinstance(out, ToolResult) + assert out.metadata["was_truncated"] is True + assert out.metadata["result_chars_before_truncation"] == 5000 + # Content carries the truncation marker so the LLM knows it was clamped. + assert "OUTPUT TRUNCATED" in out.content + + def test_end_to_end_truncation_event(self): + """Wired path: truncated handler output reaches the event with + was_truncated=True. This is the regression check for the + previously-dead `was_truncated` / `result_chars_before_truncation` + fields on AgentToolCallCompletedEvent. + """ + from src.config import settings + from src.utils.agent_tools import _maybe_truncated_result + + emitted: list[BaseEvent] = [] + ctx = _StubToolContext() + + original = "y" * 10_000 + with patch.object(settings.LLM, "MAX_TOOL_OUTPUT_CHARS", 200): + wrapped = _maybe_truncated_result(original) + assert isinstance(wrapped, ToolResult) + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_agent_tool_call_completed( + ctx=ctx, + tool_name="get_recent_history", + duration_ms=1.0, + result_str=wrapped.content, + metadata=wrapped.metadata, + is_error=False, + iteration=1, + tool_call_seq=0, + provider_tool_call_id=None, + ) + + ev = emitted[0] + assert isinstance(ev, AgentToolCallCompletedEvent) + assert ev.was_truncated is True + assert ev.result_chars_before_truncation == 10_000 diff --git a/tests/llm/test_telemetry_llm_call.py b/tests/llm/test_telemetry_llm_call.py new file mode 100644 index 00000000..817aa41e --- /dev/null +++ b/tests/llm/test_telemetry_llm_call.py @@ -0,0 +1,686 @@ +# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false +"""tests for LLMCallCompletedEvent emission and the high-volume sampler. + +Targets: +- `src/llm/executor.py::honcho_llm_call_inner` emits one event per call, + on both success and failure (try/finally). +- `LLMTelemetryContext` round-trips workspace/run_id/iteration/call_purpose + onto the event without mutating the caller-supplied context. +- The sampler at `src/telemetry/emitter.py::_should_sample` keeps every event + of a run together (same run_id → same decision) and lets ground_truth + events through unconditionally. +""" + +from __future__ import annotations + +from typing import Any +from unittest.mock import AsyncMock, patch + +import pytest + +from src.llm.backend import CompletionResult as BackendCompletionResult +from src.llm.executor import _emit_llm_call_completed +from src.llm.runtime import AttemptPlan +from src.llm.types import LLMTelemetryContext +from src.telemetry.events import ( + BaseEvent, + CallPurpose, + DialecticCompletedEvent, + LLMCallCompletedEvent, +) + + +def _make_plan( + *, attempt: int = 1, retry_attempts: int = 3, is_fallback: bool = False +) -> AttemptPlan: + """Minimal AttemptPlan; client/selected_config are unused by the emitter helper.""" + return AttemptPlan( + provider="anthropic", + model="claude-sonnet-4-5", + client=object(), + thinking_budget_tokens=None, + reasoning_effort=None, + selected_config=object(), + attempt=attempt, + retry_attempts=retry_attempts, + is_fallback=is_fallback, + ) + + +class TestEmitLLMCallCompleted: + def test_emits_success_event(self): + emitted: list[BaseEvent] = [] + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_llm_call_completed( + plan=_make_plan(), + telemetry=LLMTelemetryContext( + workspace_name="ws1", + call_purpose=CallPurpose.DIALECTIC_ANSWER.value, + parent_category="dialectic", + run_id="run-xyz", + iteration=2, + ), + provider="anthropic", + model="claude-sonnet-4-5", + max_tokens=2048, + duration_ms=300.0, + has_tools=True, + was_stream=False, + outcome="success", + result=BackendCompletionResult( + content="hi", + input_tokens=10, + output_tokens=5, + cache_read_input_tokens=2, + cache_creation_input_tokens=1, + finish_reason="stop", + ), + error=None, + ) + + assert len(emitted) == 1 + event = emitted[0] + assert isinstance(event, LLMCallCompletedEvent) + assert event.outcome == "success" + assert event.is_final_attempt is False + assert event.workspace_name == "ws1" + assert event.run_id == "run-xyz" + assert event.iteration == 2 + assert event.call_purpose == CallPurpose.DIALECTIC_ANSWER + assert event.provider_input_tokens == 10 + assert event.provider_output_tokens == 5 + assert event.cache_read_tokens == 2 + assert event.cache_creation_tokens == 1 + assert event.finish_reason == "stop" + assert event.has_tools is True + + def test_emits_error_event_with_class_name(self): + emitted: list[BaseEvent] = [] + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_llm_call_completed( + plan=_make_plan(attempt=3, retry_attempts=3, is_fallback=True), + telemetry=None, + provider="openai", + model="gpt-4", + max_tokens=512, + duration_ms=15.0, + has_tools=False, + was_stream=False, + outcome="error", + result=None, + error=RuntimeError("nope"), + ) + + assert len(emitted) == 1 + event = emitted[0] + assert isinstance(event, LLMCallCompletedEvent) + assert event.outcome == "error" + # On the last attempt, is_final_attempt must be True (replaces the + # synthetic "retry_exhausted" outcome from earlier drafts). + assert event.is_final_attempt is True + assert event.error_class == "RuntimeError" + assert event.was_fallback is True + # No result → token fields are 0. + assert event.provider_input_tokens == 0 + assert event.provider_output_tokens == 0 + + def test_unknown_call_purpose_silently_dropped(self): + """Unknown call_purpose strings should not raise; event still emits.""" + emitted: list[BaseEvent] = [] + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_llm_call_completed( + plan=_make_plan(), + telemetry=LLMTelemetryContext(call_purpose="not.a.real.purpose"), + provider="anthropic", + model="claude", + max_tokens=1, + duration_ms=0.0, + has_tools=False, + was_stream=False, + outcome="success", + result=BackendCompletionResult(), + error=None, + ) + + assert len(emitted) == 1 + event = emitted[0] + assert isinstance(event, LLMCallCompletedEvent) + # Unknown purpose drops to None rather than raising. + assert event.call_purpose is None + + def test_provider_label_inferred_from_openrouter_model_prefix(self): + emitted: list[BaseEvent] = [] + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + _emit_llm_call_completed( + plan=_make_plan(), + telemetry=None, + provider="openai", + model="anthropic/claude-3-5-sonnet", + max_tokens=1, + duration_ms=0.0, + has_tools=False, + was_stream=False, + outcome="success", + result=BackendCompletionResult(), + error=None, + ) + + event = emitted[0] + assert isinstance(event, LLMCallCompletedEvent) + assert event.provider_label == "anthropic" + + def test_telemetry_failures_swallowed(self): + """A broken emitter must NOT propagate exceptions out of the LLM path.""" + + def explode(*_args: Any, **_kwargs: Any) -> None: + raise RuntimeError("emitter wedged") + + with patch("src.telemetry.events.emit", side_effect=explode): + # Must not raise. + _emit_llm_call_completed( + plan=_make_plan(), + telemetry=None, + provider="anthropic", + model="claude", + max_tokens=1, + duration_ms=0.0, + has_tools=False, + was_stream=False, + outcome="success", + result=BackendCompletionResult(), + error=None, + ) + + +class TestSampler: + """Tests for the deterministic high-volume sampler.""" + + def test_rate_one_passes_everything(self): + from src.telemetry.emitter import _should_sample + + event = LLMCallCompletedEvent( + transport="anthropic", + model="m", + effective_max_output_tokens=1, + outcome="success", + is_final_attempt=True, + attempt=1, + retry_attempts=1, + was_fallback=False, + duration_ms=0.0, + run_id="anything", + ) + assert _should_sample(event, 1.0) is True + + def test_rate_zero_drops_everything(self): + from src.telemetry.emitter import _should_sample + + event = LLMCallCompletedEvent( + transport="anthropic", + model="m", + effective_max_output_tokens=1, + outcome="success", + is_final_attempt=True, + attempt=1, + retry_attempts=1, + was_fallback=False, + duration_ms=0.0, + run_id="anything", + ) + assert _should_sample(event, 0.0) is False + + def test_same_run_id_gets_same_decision(self): + """Two events with the same run_id must hash to the same bucket so an + entire trace is either kept or dropped — never half-sampled.""" + from src.telemetry.emitter import _should_sample + + def make(iteration: int) -> LLMCallCompletedEvent: + return LLMCallCompletedEvent( + transport="anthropic", + model="m", + effective_max_output_tokens=1, + outcome="success", + is_final_attempt=False, + attempt=1, + retry_attempts=1, + was_fallback=False, + duration_ms=0.0, + run_id="stable-run-id", + iteration=iteration, + ) + + rate = 0.5 + a = _should_sample(make(1), rate) + b = _should_sample(make(2), rate) + c = _should_sample(make(3), rate) + assert a == b == c + + +class TestExecutorEndToEnd: + """Exercise honcho_llm_call_inner's try/finally on both paths.""" + + @pytest.mark.asyncio + async def test_success_path_emits_one_event(self): + from src.llm import executor + + emitted: list[BaseEvent] = [] + result = BackendCompletionResult( + content="ok", input_tokens=3, output_tokens=2, finish_reason="stop" + ) + + with ( + patch.object(executor, "CLIENTS", {"anthropic": object()}), + patch.object( + executor, + "backend_for_provider", + return_value=object(), + ), + patch.object( + executor, + "execute_completion", + new=AsyncMock(return_value=result), + ), + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + ): + await executor.honcho_llm_call_inner( + "anthropic", + "claude-sonnet-4-5", + "hello", + max_tokens=128, + plan=_make_plan(), + telemetry=LLMTelemetryContext( + workspace_name="ws", + call_purpose=CallPurpose.DERIVER_REPRESENTATION.value, + parent_category="representation", + ), + ) + + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, LLMCallCompletedEvent) + assert ev.outcome == "success" + assert ev.provider_output_tokens == 2 + + @pytest.mark.asyncio + async def test_cancelled_path_emits_cancelled_outcome(self): + """asyncio.CancelledError mid-call surfaces as outcome='cancelled', not + 'error' — client disconnects / shutdowns must not pollute error rates.""" + import asyncio + + from src.llm import executor + + emitted: list[BaseEvent] = [] + + async def _cancel(*_args: Any, **_kwargs: Any) -> Any: + raise asyncio.CancelledError() + + with ( + patch.object(executor, "CLIENTS", {"anthropic": object()}), + patch.object(executor, "backend_for_provider", return_value=object()), + patch.object(executor, "execute_completion", new=_cancel), + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + pytest.raises(asyncio.CancelledError), + ): + await executor.honcho_llm_call_inner( + "anthropic", + "claude-sonnet-4-5", + "hello", + max_tokens=128, + plan=_make_plan(), + telemetry=None, + ) + + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, LLMCallCompletedEvent) + assert ev.outcome == "cancelled" + assert ev.error_class == "CancelledError" + + @pytest.mark.asyncio + async def test_stream_cancelled_emits_cancelled_outcome(self): + """Stream path: mid-iteration CancelledError surfaces as 'cancelled'.""" + import asyncio + from collections.abc import AsyncIterator + + from src.llm import executor + + emitted: list[BaseEvent] = [] + + async def _cancelling_stream() -> AsyncIterator[Any]: + # one chunk then cancel — simulates a client disconnect mid-stream. + yield object() # caller's `async for` consumes this + raise asyncio.CancelledError() + + async def _setup_stream(*_args: Any, **_kwargs: Any) -> AsyncIterator[Any]: + return _cancelling_stream() + + with ( + patch.object(executor, "CLIENTS", {"anthropic": object()}), + patch.object(executor, "backend_for_provider", return_value=object()), + patch.object(executor, "execute_stream", new=_setup_stream), + patch.object( + executor, + "stream_chunk_to_response_chunk", + side_effect=lambda chunk: chunk, + ), + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + ): + stream = await executor.honcho_llm_call_inner( + "anthropic", + "claude-sonnet-4-5", + "hello", + max_tokens=128, + plan=_make_plan(), + telemetry=None, + stream=True, + ) + with pytest.raises(asyncio.CancelledError): + async for _ in stream: + pass + + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, LLMCallCompletedEvent) + assert ev.outcome == "cancelled" + assert ev.was_stream is True + + @pytest.mark.asyncio + async def test_stream_setup_failure_emits_and_propagates(self): + """Stream-setup errors must propagate out of the AWAITED + `honcho_llm_call_inner` call (not deferred until first iteration), + so the outer retry wrapper in tool_loop.stream_final_response sees + them. Regression check for the bug where `_stream()` returned a + generator without awaiting `execute_stream`, hiding setup failures + from tenacity. + """ + from src.llm import executor + + emitted: list[BaseEvent] = [] + + async def _setup_explodes(*_args: Any, **_kwargs: Any) -> Any: + raise RuntimeError("rate limited") + + with ( + patch.object(executor, "CLIENTS", {"anthropic": object()}), + patch.object(executor, "backend_for_provider", return_value=object()), + patch.object(executor, "execute_stream", new=_setup_explodes), + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + pytest.raises(RuntimeError, match="rate limited"), + ): + # The await itself must raise — that's how tenacity sees it. + await executor.honcho_llm_call_inner( + "anthropic", + "claude-sonnet-4-5", + "hello", + max_tokens=128, + plan=_make_plan(), + telemetry=None, + stream=True, + ) + + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, LLMCallCompletedEvent) + assert ev.outcome == "error" + assert ev.was_stream is True + assert ev.error_class == "RuntimeError" + + @pytest.mark.asyncio + async def test_error_path_still_emits_via_finally(self): + from src.llm import executor + + emitted: list[BaseEvent] = [] + + async def _boom(*_args: Any, **_kwargs: Any) -> Any: + raise RuntimeError("backend exploded") + + with ( + patch.object(executor, "CLIENTS", {"anthropic": object()}), + patch.object( + executor, + "backend_for_provider", + return_value=object(), + ), + patch.object(executor, "execute_completion", new=_boom), + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + pytest.raises(RuntimeError), + ): + await executor.honcho_llm_call_inner( + "anthropic", + "claude-sonnet-4-5", + "hello", + max_tokens=128, + plan=_make_plan(attempt=3, retry_attempts=3), + telemetry=None, + ) + + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, LLMCallCompletedEvent) + assert ev.outcome == "error" + assert ev.is_final_attempt is True + assert ev.error_class == "RuntimeError" + + +def test_ground_truth_event_skips_sampler(): + """DialecticCompletedEvent declares _volume_class='ground_truth' (default + on BaseEvent) so the sampler should never gate it, even at rate 0.0. + + Drives the real emitter under a zero-rate config and asserts the + ground-truth event still lands in the buffer — the actual bypass + behavior in emit() — not just the helper-function semantics in + _should_sample. Regression guard for a future refactor that pushes + ground_truth events through the sampling decision. + """ + from src.telemetry.emitter import TelemetryEmitter, _should_sample + + # The volume_class declaration is the gate emit() consults. + assert DialecticCompletedEvent.volume_class() == "ground_truth" + + event = DialecticCompletedEvent( + run_id="r", + workspace_name="ws", + peer_name="p", + reasoning_level="medium", + total_duration_ms=10.0, + input_tokens=1, + output_tokens=1, + ) + # Sampler is deterministic 1.0 → True for any event (sanity). + assert _should_sample(event, 1.0) is True + + # Drive the emitter directly under rate=0.0. enabled=True requires a + # non-None endpoint; we use a placeholder URL — the emitter buffers but + # we never flush, so no HTTP traffic is generated. The buffer growing + # proves the ground_truth event bypassed the sampler. + emitter = TelemetryEmitter(endpoint="http://test/events", enabled=True) + with patch("src.config.settings") as mock_settings: + mock_settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE = 0.0 + mock_settings.TELEMETRY.NAMESPACE = "test" + emitter.emit(event) + assert emitter.buffer_size == 1 # ground_truth survived the sampler + + # Sanity check: a high-volume event under rate=0.0 DOES get dropped, + # proving the test setup actually exercises the sampling code path. + from src.telemetry.events import LLMCallCompletedEvent + + sampled_event = LLMCallCompletedEvent( + transport="anthropic", + model="m", + effective_max_output_tokens=1, + finish_reason="stop", + outcome="success", + is_final_attempt=True, + attempt=1, + retry_attempts=1, + was_fallback=False, + duration_ms=1.0, + has_tools=False, + was_stream=False, + ) + assert sampled_event.volume_class() == "high_volume" + with patch("src.config.settings") as mock_settings: + mock_settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE = 0.0 + mock_settings.TELEMETRY.NAMESPACE = "test" + emitter.emit(sampled_event) + # Buffer still 1 — high_volume event was sampled out. + assert emitter.buffer_size == 1 + + +class TestStreamFinalResponseRetryAttempt: + """`stream_final_response` (src/llm/tool_loop.py) must bump the + per-attempt index on the plan it passes to `honcho_llm_call_inner`. + Previously every retried stream-setup emit reported the same `attempt` + value because the pinned `winning_plan` was reused unchanged. Fix 13 + plumbs a per-retry plan via `dataclasses.replace`. + """ + + @pytest.mark.asyncio + async def test_attempt_index_bumps_across_retries(self): + from collections.abc import AsyncIterator + + from src.llm import executor, tool_loop + + emitted: list[BaseEvent] = [] + + # execute_stream raises on the first two calls, succeeds on the third. + call_count = 0 + + async def _flaky_setup(*_args: Any, **_kwargs: Any) -> AsyncIterator[Any]: + nonlocal call_count + call_count += 1 + if call_count < 3: + raise RuntimeError("transient") + + async def _ok_stream() -> AsyncIterator[Any]: + # Empty async generator — the unreachable yield is required + # to keep this function an async generator (no `async def + # ... -> AsyncIterator: return` shortcut exists in Python). + return + yield # pyright: ignore[reportUnreachable] + + return _ok_stream() + + # selected_config=None lets effective_config_for_call synthesize a + # minimal ModelConfig — avoids needing a real ModelConfig in this test. + winning_plan = AttemptPlan( + provider="anthropic", + model="claude-sonnet-4-5", + client=object(), + thinking_budget_tokens=None, + reasoning_effort=None, + selected_config=None, + attempt=1, + retry_attempts=3, + is_fallback=False, + ) + + with ( + patch.object(executor, "CLIENTS", {"anthropic": object()}), + patch.object(executor, "backend_for_provider", return_value=object()), + patch.object(executor, "execute_stream", new=_flaky_setup), + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + ): + stream = tool_loop.stream_final_response( + winning_plan=winning_plan, + prompt="hi", + max_tokens=64, + conversation_messages=[{"role": "user", "content": "x"}], + response_model=None, + json_mode=False, + temperature=None, + stop_seqs=None, + verbosity=None, + enable_retry=True, + retry_attempts=3, + before_retry_callback=lambda _r: None, + telemetry=None, + ) + # Drain (empty) so the wrapper's finally fires for the success attempt. + async for _chunk in stream: + pass + + # 3 emissions: attempts 1 & 2 errored, attempt 3 succeeded. + llm_events = [e for e in emitted if isinstance(e, LLMCallCompletedEvent)] + assert [e.attempt for e in llm_events] == [1, 2, 3] + # Final attempt flag: only True on the last retry (attempt 3 of 3). + assert [e.is_final_attempt for e in llm_events] == [False, False, True] + # First two errored, last succeeded. + assert [e.outcome for e in llm_events] == ["error", "error", "success"] + + +class TestStreamingResponseTokenWriteBack: + """`StreamingResponseWithMetadata` must accumulate the final-stream's + output_tokens (reported in usage chunks by OpenAI/Anthropic) into its + `output_tokens` attribute as the stream drains, so DialecticCompletedEvent + sees tool-loop totals + final-stream totals — not tool-loop totals alone. + """ + + @pytest.mark.asyncio + async def test_output_tokens_folds_in_final_stream_usage(self): + from src.llm.types import ( + HonchoLLMCallStreamChunk, + StreamingResponseWithMetadata, + ) + + async def _fake_stream() -> Any: + # Content-only chunks, then a final usage chunk with cumulative + # output_tokens=137 — matches OpenAI's include_usage pattern. + yield HonchoLLMCallStreamChunk(content="hel", output_tokens=None) + yield HonchoLLMCallStreamChunk(content="lo", output_tokens=None) + yield HonchoLLMCallStreamChunk(content="", is_done=True, output_tokens=137) + + wrapper = StreamingResponseWithMetadata( + stream=_fake_stream(), + tool_calls_made=[], + input_tokens=200, + output_tokens=50, # tool-loop running output total + cache_creation_input_tokens=0, + cache_read_input_tokens=0, + ) + # Before drain, the wrapper holds only the tool-loop total. + assert wrapper.output_tokens == 50 + + chunks: list[HonchoLLMCallStreamChunk] = [] + async for chunk in wrapper: + chunks.append(chunk) + + # After drain, the final-stream's 137 output tokens fold in. + assert wrapper.output_tokens == 50 + 137 + # And we yielded every chunk to the caller — the wrapper is a + # passthrough, not a sink. + assert len(chunks) == 3 diff --git a/tests/llm/test_tool_loop_truncation.py b/tests/llm/test_tool_loop_truncation.py new file mode 100644 index 00000000..97c3170c --- /dev/null +++ b/tests/llm/test_tool_loop_truncation.py @@ -0,0 +1,197 @@ +# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false +"""Regression tests for `hit_input_token_cap` propagation through +`execute_tool_loop`. + +The toolless path (`src/llm/api.py:325-340`) detects the cap hit up-front +by comparing input tokens against `max_input_tokens`. Before this fix, +the tool-loop path called `truncate_messages_to_fit` per iteration but +never propagated the flag — Dialectic (the main tool-loop consumer) +under-reported `hit_input_token_cap` on dialectic/representation events. + +The rule is intentionally token-based, not message-count-based, so the +deriver's single-prompt path (where `truncate_messages_to_fit` keeps the +last unit even when oversized) still surfaces a real cap hit. +""" + +from __future__ import annotations + +from typing import Any +from unittest.mock import patch + +import pytest + +from src.llm import tool_loop +from src.llm.runtime import AttemptPlan +from src.llm.tool_loop import execute_tool_loop +from src.llm.types import HonchoLLMCallResponse + + +def _make_plan() -> AttemptPlan: + # `selected_config=None` works for these tests since `_call_with_messages` + # passes it straight through to the mocked `honcho_llm_call_inner`. + return AttemptPlan( + provider="anthropic", + model="claude-sonnet-4-5", + client=object(), + thinking_budget_tokens=None, + reasoning_effort=None, + selected_config=None, + attempt=1, + retry_attempts=1, + is_fallback=False, + ) + + +async def _terminating_call(*_args: Any, **_kwargs: Any) -> HonchoLLMCallResponse[Any]: + # No tool calls — execute_tool_loop terminates after iteration 1. + return HonchoLLMCallResponse( + content="done", + input_tokens=10, + output_tokens=5, + cache_creation_input_tokens=0, + cache_read_input_tokens=0, + finish_reasons=["stop"], + tool_calls_made=[], + ) + + +@pytest.mark.asyncio +async def test_hit_input_token_cap_fires_when_input_exceeds_cap(): + """When the input message list exceeds `max_input_tokens` by token + count, the response carries `hit_input_token_cap=True`. Regression + check for the rule switch from message-count to token-based. + """ + + # Pretend the conversation totals 200 tokens; cap is 100. + with ( + patch.object(tool_loop, "honcho_llm_call_inner", new=_terminating_call), + patch("src.llm.conversation.count_message_tokens", return_value=200), + patch( + "src.llm.conversation.truncate_messages_to_fit", + side_effect=lambda msgs, _cap: msgs, + ), + ): + result = await execute_tool_loop( + prompt="hi", + max_tokens=64, + messages=[{"role": "user", "content": "huge"}], + tools=[ + { + "name": "noop", + "description": "no-op", + "input_schema": {"type": "object"}, + } + ], + tool_choice="auto", + tool_executor=lambda _name, _input: "", + max_tool_iterations=5, + response_model=None, + json_mode=False, + temperature=None, + stop_seqs=None, + verbosity=None, + enable_retry=False, + retry_attempts=1, + max_input_tokens=100, + get_attempt_plan=_make_plan, + before_retry_callback=lambda _r: None, + stream_final=False, + telemetry=None, + ) + + assert isinstance(result, HonchoLLMCallResponse) + assert result.hit_input_token_cap is True + + +@pytest.mark.asyncio +async def test_hit_input_token_cap_false_when_under_cap(): + """Input tokens under cap → flag stays False, no false positive.""" + + with ( + patch.object(tool_loop, "honcho_llm_call_inner", new=_terminating_call), + patch("src.llm.conversation.count_message_tokens", return_value=50), + patch( + "src.llm.conversation.truncate_messages_to_fit", + side_effect=lambda msgs, _cap: msgs, + ), + ): + result = await execute_tool_loop( + prompt="hi", + max_tokens=64, + messages=[{"role": "user", "content": "small"}], + tools=[ + { + "name": "noop", + "description": "no-op", + "input_schema": {"type": "object"}, + } + ], + tool_choice="auto", + tool_executor=lambda _name, _input: "", + max_tool_iterations=5, + response_model=None, + json_mode=False, + temperature=None, + stop_seqs=None, + verbosity=None, + enable_retry=False, + retry_attempts=1, + max_input_tokens=100, + get_attempt_plan=_make_plan, + before_retry_callback=lambda _r: None, + stream_final=False, + telemetry=None, + ) + + assert isinstance(result, HonchoLLMCallResponse) + assert result.hit_input_token_cap is False + + +@pytest.mark.asyncio +async def test_hit_input_token_cap_fires_even_when_truncate_cant_shrink(): + """Critical regression: the single-message over-cap case (deriver's + prompt-only call) used to silently return hit=False because + `truncate_messages_to_fit` keeps the last unit even when oversized, + making the old message-count-based check stay at False. The new + token-based rule correctly catches this case. + """ + + with ( + patch.object(tool_loop, "honcho_llm_call_inner", new=_terminating_call), + patch("src.llm.conversation.count_message_tokens", return_value=99_999), + # Truncate is a no-op (matches real behavior for single-message inputs). + patch( + "src.llm.conversation.truncate_messages_to_fit", + side_effect=lambda msgs, _cap: msgs, + ), + ): + result = await execute_tool_loop( + prompt="hi", + max_tokens=64, + messages=[{"role": "user", "content": "x" * 1_000_000}], + tools=[ + { + "name": "noop", + "description": "no-op", + "input_schema": {"type": "object"}, + } + ], + tool_choice="auto", + tool_executor=lambda _name, _input: "", + max_tool_iterations=5, + response_model=None, + json_mode=False, + temperature=None, + stop_seqs=None, + verbosity=None, + enable_retry=False, + retry_attempts=1, + max_input_tokens=1_000, + get_attempt_plan=_make_plan, + before_retry_callback=lambda _r: None, + stream_final=False, + telemetry=None, + ) + + assert isinstance(result, HonchoLLMCallResponse) + assert result.hit_input_token_cap is True diff --git a/tests/scripts/test_configure_embeddings.py b/tests/scripts/test_configure_embeddings.py index c35414f9..408f0a6b 100644 --- a/tests/scripts/test_configure_embeddings.py +++ b/tests/scripts/test_configure_embeddings.py @@ -1,4 +1,4 @@ -"""Phase 3: configure_embeddings script tests.""" +"""Tests for the configure_embeddings script.""" from __future__ import annotations diff --git a/tests/startup/test_embedding_validator.py b/tests/startup/test_embedding_validator.py index ef31b24f..962e07f0 100644 --- a/tests/startup/test_embedding_validator.py +++ b/tests/startup/test_embedding_validator.py @@ -1,4 +1,4 @@ -"""Phase 2: startup embedding-schema validator + VECTOR_STORE_DIMENSIONS deprecation.""" +"""Startup embedding-schema validator + VECTOR_STORE_DIMENSIONS deprecation.""" from __future__ import annotations @@ -178,9 +178,9 @@ def test_vector_store_dimensions_explicit_set_warns( def test_non_1536_pgvector_without_migrated_no_longer_raises_at_config_time() -> None: - """Phase 2 removed the dim-vs-MIGRATED guard. Constructing AppSettings + """The dim-vs-MIGRATED guard has been removed. Constructing AppSettings with non-1536 + default pgvector + MIGRATED=false should now succeed - (the runtime schema validator at startup is the new safety net).""" + (the runtime schema validator at startup is the safety net).""" env = { **os.environ, "PYTHON_DOTENV_DISABLED": "1", diff --git a/tests/telemetry/conftest.py b/tests/telemetry/conftest.py index a52194e2..f871e22d 100644 --- a/tests/telemetry/conftest.py +++ b/tests/telemetry/conftest.py @@ -2,7 +2,7 @@ """Fixtures for telemetry unit tests. This module provides: -- Sample event fixtures for all 12 event types +- Sample event fixtures for all telemetry event types - Mock settings fixtures for controlling telemetry configuration - Mock HTTP client fixtures for testing the emitter without network calls """ @@ -19,10 +19,16 @@ from src.telemetry.events.agent import ( AgentToolPeerCardUpdatedEvent, AgentToolSummaryCreatedEvent, ) +from src.telemetry.events.api import ( + FileUploadedEvent, + GetContextEvent, + MessageCreatedEvent, +) from src.telemetry.events.base import BaseEvent from src.telemetry.events.deletion import DeletionCompletedEvent from src.telemetry.events.dialectic import DialecticCompletedEvent from src.telemetry.events.dream import DreamRunEvent, DreamSpecialistEvent +from src.telemetry.events.llm import CallPurpose, LLMCallCompletedEvent from src.telemetry.events.reconciliation import ( CleanupStaleItemsCompletedEvent, SyncVectorsCompletedEvent, @@ -64,7 +70,101 @@ def sample_representation_event( llm_call_ms=1200.0, total_duration_ms=1300.0, input_tokens=5000, + total_input_tokens=7500, output_tokens=500, + # additive fields + queued_message_count=3, + prompt_message_count=10, + prompt_message_tokens=7000, + extra_context_message_count=7, + extra_context_tokens=2000, + prompt_scaffold_tokens=500, + batch_max_tokens=20_000, + max_input_tokens=23_000, + was_flush_enabled=False, + hit_batch_token_cap=False, + hit_input_token_cap=False, + observer_count=1, + ) + + +@pytest.fixture +def sample_message_created_event(fixed_timestamp: datetime) -> MessageCreatedEvent: + """Create a sample MessageCreatedEvent for testing.""" + return MessageCreatedEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + session_name="test_session", + message_count=2, + total_tokens=250, + last_message_id="msg_abc123_fixture_____", + ) + + +@pytest.fixture +def sample_file_uploaded_event(fixed_timestamp: datetime) -> FileUploadedEvent: + """Create a sample FileUploadedEvent for testing.""" + return FileUploadedEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + session_name="test_session", + peer_name="user_peer", + file_id="file_123", + filename="notes.txt", + content_type="text/plain", + file_size_bytes=1024, + message_count=2, + total_tokens=250, + ) + + +@pytest.fixture +def sample_llm_call_event(fixed_timestamp: datetime) -> LLMCallCompletedEvent: + """Sample LLMCallCompletedEvent ().""" + return LLMCallCompletedEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + call_purpose=CallPurpose.DIALECTIC_ANSWER, + parent_category="dialectic", + transport="anthropic", + provider_label=None, + model="claude-sonnet-4-5", + effective_max_output_tokens=4096, + provider_input_tokens=1234, + provider_output_tokens=567, + cache_read_tokens=100, + cache_creation_tokens=50, + finish_reason="stop", + outcome="success", + is_final_attempt=False, + attempt=1, + retry_attempts=3, + was_fallback=False, + duration_ms=1100.5, + has_tools=True, + tool_call_count=2, + run_id="abc12345", + iteration=1, + ) + + +@pytest.fixture +def sample_get_context_event(fixed_timestamp: datetime) -> GetContextEvent: + """Create a sample GetContextEvent for testing.""" + return GetContextEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + context_scope="session", + session_name="test_session", + tokens_requested=4000, + message_count=10, + has_summary=True, + has_representation=False, + has_peer_card=False, + search_query_provided=False, + include_summary=True, + peer_perspective_provided=False, + total_duration_ms=25.0, ) @@ -87,6 +187,13 @@ def sample_dream_run_event(fixed_timestamp: datetime) -> DreamRunEvent: total_input_tokens=25000, total_output_tokens=3000, total_duration_ms=45000.0, + # additive fields + dream_type="omni", + enabled_types_count=1, + trigger_reason="document_threshold", + delay_reason="idle_timeout", + documents_since_last_dream_at_schedule=55, + document_threshold=50, ) @@ -106,6 +213,11 @@ def sample_dream_specialist_event(fixed_timestamp: datetime) -> DreamSpecialistE output_tokens=2000, duration_ms=25000.0, success=True, + # additive rollups + created_observation_count=7, + deleted_observation_count=2, + peer_card_updated=True, + search_tool_calls_count=4, ) @@ -225,6 +337,10 @@ def sample_summary_created_event( summary_type="short", input_tokens=4000, output_tokens=300, + # additive token breakdown + previous_summary_tokens=200, + message_tokens=3500, + prompt_scaffold_tokens=300, ) @@ -276,6 +392,9 @@ def sample_cleanup_event(fixed_timestamp: datetime) -> CleanupStaleItemsComplete @pytest.fixture def all_sample_events( sample_representation_event: RepresentationCompletedEvent, + sample_message_created_event: MessageCreatedEvent, + sample_file_uploaded_event: FileUploadedEvent, + sample_get_context_event: GetContextEvent, sample_dream_run_event: DreamRunEvent, sample_dream_specialist_event: DreamSpecialistEvent, sample_dialectic_event: DialecticCompletedEvent, @@ -287,10 +406,14 @@ def all_sample_events( sample_deletion_event: DeletionCompletedEvent, sample_sync_vectors_event: SyncVectorsCompletedEvent, sample_cleanup_event: CleanupStaleItemsCompletedEvent, + sample_llm_call_event: LLMCallCompletedEvent, ) -> list[BaseEvent]: """Return all sample events as a list for parametrized tests.""" return [ sample_representation_event, + sample_message_created_event, + sample_file_uploaded_event, + sample_get_context_event, sample_dream_run_event, sample_dream_specialist_event, sample_dialectic_event, @@ -302,6 +425,7 @@ def all_sample_events( sample_deletion_event, sample_sync_vectors_event, sample_cleanup_event, + sample_llm_call_event, ] diff --git a/tests/telemetry/test_embedding_call_event.py b/tests/telemetry/test_embedding_call_event.py new file mode 100644 index 00000000..ee96e749 --- /dev/null +++ b/tests/telemetry/test_embedding_call_event.py @@ -0,0 +1,333 @@ +# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false +"""tests for EmbeddingCallCompletedEvent emission. + +Targets: +- New `EmbeddingCallCompletedEvent` (embedding.call.completed) at schema v1 + with high_volume sampling. +- `EmbeddingCallPurpose` closed enum. +- The `embedding_call_purpose` context manager round-trips the slug onto + the event via the ContextVar, without changing call signatures. +- `_emit_embedding_call` wrapper emits on success AND on exception, and + propagates the underlying error unchanged. +""" + +from __future__ import annotations + +from typing import Any +from unittest.mock import patch + +import pytest + +from src.embedding_client import _emit_embedding_call +from src.telemetry.events import ( + BaseEvent, + EmbeddingCallCompletedEvent, + EmbeddingCallPurpose, +) +from src.utils.types import ( + embedding_call_purpose, + get_embedding_call_purpose, +) + + +class TestEventShape: + def test_event_type_and_version(self): + assert EmbeddingCallCompletedEvent.event_type() == "embedding.call.completed" + assert EmbeddingCallCompletedEvent.schema_version() == 1 + assert EmbeddingCallCompletedEvent.category() == "llm" + + def test_volume_class_is_high_volume(self): + """event participates in HIGH_VOLUME_SAMPLE_RATE alongside + llm.call.completed — search-heavy paths can flood the buffer.""" + assert EmbeddingCallCompletedEvent.volume_class() == "high_volume" + + def test_resource_id_disambiguates(self): + ev_a = EmbeddingCallCompletedEvent( + provider="openai", + model="text-embedding-3-small", + input_count=5, + duration_ms=10.0, + outcome="success", + call_purpose=EmbeddingCallPurpose.SEARCH_MEMORY, + ) + ev_b = EmbeddingCallCompletedEvent( + provider="openai", + model="text-embedding-3-small", + input_count=5, + duration_ms=10.0, + outcome="success", + call_purpose=EmbeddingCallPurpose.SEARCH_MESSAGES, + ) + # Different purpose → different resource id. + assert ev_a.get_resource_id() != ev_b.get_resource_id() + + +class TestEmbeddingCallPurposeEnum: + def test_known_values(self): + # The closed taxonomy. Adding a value here requires a coordinated + # update with downstream analytics that filter on call_purpose. + assert EmbeddingCallPurpose.SEARCH_MEMORY.value == "search_memory" + assert EmbeddingCallPurpose.SEARCH_MESSAGES.value == "search_messages" + assert EmbeddingCallPurpose.CREATE_OBSERVATIONS.value == "create_observations" + assert EmbeddingCallPurpose.VECTOR_SYNC.value == "vector_sync" + assert EmbeddingCallPurpose.SUMMARY.value == "summary" + assert EmbeddingCallPurpose.MESSAGE_CREATE.value == "message_create" + + +class TestContextManager: + def test_sets_and_clears(self): + assert get_embedding_call_purpose() is None + with embedding_call_purpose("search_memory"): + assert get_embedding_call_purpose() == "search_memory" + # ContextVar must reset on exit. + assert get_embedding_call_purpose() is None + + def test_nested_context_managers_restore_outer(self): + """Nested usage shouldn't lose the outer purpose on inner-exit.""" + with embedding_call_purpose("search_memory"): + assert get_embedding_call_purpose() == "search_memory" + with embedding_call_purpose("create_observations"): + assert get_embedding_call_purpose() == "create_observations" + # After inner exits, outer purpose must be restored — not None. + assert get_embedding_call_purpose() == "search_memory" + + def test_exception_in_block_still_resets(self): + try: + with embedding_call_purpose("search_memory"): + raise RuntimeError("boom") + except RuntimeError: + pass + assert get_embedding_call_purpose() is None + + +class TestEmitEmbeddingCallWrapper: + @pytest.mark.asyncio + async def test_success_emits_event_with_purpose(self): + emitted: list[BaseEvent] = [] + + async def _fake_call() -> list[float]: + return [0.1, 0.2, 0.3] + + with ( + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + embedding_call_purpose("search_memory"), + ): + result = await _emit_embedding_call( + provider="openai", + model="text-embedding-3-small", + texts=["query"], + input_tokens_estimate=3, + fn=_fake_call, + ) + + assert result == [0.1, 0.2, 0.3] + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, EmbeddingCallCompletedEvent) + assert ev.outcome == "success" + assert ev.provider == "openai" + assert ev.model == "text-embedding-3-small" + assert ev.input_count == 1 + assert ev.input_tokens_estimate == 3 + assert ev.call_purpose == EmbeddingCallPurpose.SEARCH_MEMORY + assert ev.error_class is None + + @pytest.mark.asyncio + async def test_exception_emits_error_event_and_propagates(self): + emitted: list[BaseEvent] = [] + + async def _boom() -> list[float]: + raise RuntimeError("provider down") + + with ( + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + pytest.raises(RuntimeError, match="provider down"), + ): + await _emit_embedding_call( + provider="gemini", + model="text-embedding-005", + texts=["a", "b"], + input_tokens_estimate=10, + fn=_boom, + ) + + # Event emitted on the error path too (try/finally). + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, EmbeddingCallCompletedEvent) + assert ev.outcome == "error" + assert ev.error_class == "RuntimeError" + assert ev.input_count == 2 + + @pytest.mark.asyncio + async def test_cancellation_emits_cancelled_outcome(self): + """asyncio.CancelledError surfaces as outcome='cancelled', not 'error' + — client disconnects / shutdowns must not pollute error rates.""" + import asyncio + + emitted: list[BaseEvent] = [] + + async def _cancel() -> list[float]: + raise asyncio.CancelledError() + + with ( + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + pytest.raises(asyncio.CancelledError), + ): + await _emit_embedding_call( + provider="openai", + model="text-embedding-3-small", + texts=["a"], + input_tokens_estimate=4, + fn=_cancel, + ) + + assert len(emitted) == 1 + ev = emitted[0] + assert isinstance(ev, EmbeddingCallCompletedEvent) + assert ev.outcome == "cancelled" + assert ev.error_class == "CancelledError" + + @pytest.mark.asyncio + async def test_unknown_purpose_drops_to_none(self): + """Unknown call_purpose ContextVar strings shouldn't crash the + emitter — they fall through to call_purpose=None on the event.""" + emitted: list[BaseEvent] = [] + + async def _fake_call() -> list[float]: + return [0.1] + + with ( + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + embedding_call_purpose("not.a.real.purpose"), + ): + await _emit_embedding_call( + provider="openai", + model="x", + texts=["q"], + input_tokens_estimate=1, + fn=_fake_call, + ) + + ev = emitted[0] + assert isinstance(ev, EmbeddingCallCompletedEvent) + assert ev.call_purpose is None + + @pytest.mark.asyncio + async def test_telemetry_failure_swallowed(self): + """Telemetry path must not propagate exceptions into the caller.""" + + def explode(*_args: Any, **_kwargs: Any) -> None: + raise RuntimeError("emitter wedged") + + async def _fake_call() -> str: + return "ok" + + with patch("src.telemetry.events.emit", side_effect=explode): + # Must NOT raise from the wrapper even though emit() throws. + result = await _emit_embedding_call( + provider="openai", + model="x", + texts=["q"], + input_tokens_estimate=1, + fn=_fake_call, + ) + assert result == "ok" + + +class TestIsFinalAttempt: + """`is_final_attempt` must reflect real retry state — one-shot callers + report True (no further attempt), retry-loop callers thread the real + index. Previously hardcoded to False; that conflated one-shot success, + mid-retry failure, and exhausted retry on dashboards. + """ + + @pytest.mark.asyncio + async def test_oneshot_default_is_true(self): + emitted: list[BaseEvent] = [] + + async def _fake_call() -> list[float]: + return [0.1] + + with patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ): + await _emit_embedding_call( + provider="openai", + model="x", + texts=["q"], + input_tokens_estimate=1, + fn=_fake_call, + ) + + ev = emitted[0] + assert isinstance(ev, EmbeddingCallCompletedEvent) + assert ev.is_final_attempt is True + + @pytest.mark.asyncio + async def test_mid_retry_is_false(self): + emitted: list[BaseEvent] = [] + + async def _boom() -> list[float]: + raise RuntimeError("transient") + + with ( + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + pytest.raises(RuntimeError), + ): + await _emit_embedding_call( + provider="openai", + model="x", + texts=["q"], + input_tokens_estimate=1, + fn=_boom, + is_final_attempt=False, + ) + + ev = emitted[0] + assert isinstance(ev, EmbeddingCallCompletedEvent) + assert ev.is_final_attempt is False + + @pytest.mark.asyncio + async def test_exhausted_retry_is_true(self): + emitted: list[BaseEvent] = [] + + async def _boom() -> list[float]: + raise RuntimeError("permanent") + + with ( + patch( + "src.telemetry.events.emit", + side_effect=lambda event: emitted.append(event), + ), + pytest.raises(RuntimeError), + ): + await _emit_embedding_call( + provider="openai", + model="x", + texts=["q"], + input_tokens_estimate=1, + fn=_boom, + is_final_attempt=True, + ) + + ev = emitted[0] + assert isinstance(ev, EmbeddingCallCompletedEvent) + assert ev.is_final_attempt is True + assert ev.outcome == "error" diff --git a/tests/telemetry/test_emit_function.py b/tests/telemetry/test_emit_function.py index 89ddbeee..c4e2c8ad 100644 --- a/tests/telemetry/test_emit_function.py +++ b/tests/telemetry/test_emit_function.py @@ -32,6 +32,7 @@ def create_test_event() -> RepresentationCompletedEvent: llm_call_ms=100.0, total_duration_ms=110.0, input_tokens=100, + total_input_tokens=150, output_tokens=50, ) diff --git a/tests/telemetry/test_emitter.py b/tests/telemetry/test_emitter.py index 5d1e67a1..6e05b651 100644 --- a/tests/telemetry/test_emitter.py +++ b/tests/telemetry/test_emitter.py @@ -44,6 +44,7 @@ def create_test_event(message_id: str = "msg_001") -> RepresentationCompletedEve llm_call_ms=100.0, total_duration_ms=110.0, input_tokens=100, + total_input_tokens=150, output_tokens=50, ) @@ -980,3 +981,59 @@ class TestCloudEventFormat: cloud_event = json.loads(captured_content) # Without namespace, source should be /honcho/{category} assert cloud_event["source"] == "/honcho/representation" + + +class TestHonchoVersionInjection: + """Tests for honcho_version body injection.""" + + @pytest.mark.asyncio + async def test_honcho_version_present_in_body(self): + """honcho_version is unconditionally injected into event.data from the + HONCHO_VERSION constant (sourced from pyproject.toml).""" + from src._version import HONCHO_VERSION + + emitter = TelemetryEmitter(endpoint="http://test:8001/events") + + captured_content = None + + async def capture_post(url, content=None, headers=None): + nonlocal captured_content + captured_content = content + response = MagicMock() + response.status_code = 200 + response.raise_for_status = MagicMock() + return response + + mock_client = AsyncMock() + mock_client.post = capture_post + mock_client.aclose = AsyncMock() + emitter._client = mock_client + emitter._running = True + + with patch("src.config.settings") as mock_settings: + mock_settings.TELEMETRY.NAMESPACE = "test" + event = create_test_event() + emitter.emit(event) + + await emitter.flush() + + assert captured_content is not None + cloud_event = json.loads(captured_content) + assert cloud_event["data"]["honcho_version"] == HONCHO_VERSION + + def test_emit_does_not_mutate_event_instance(self): + """contract: emit() injects into the serialized body, never the + event instance. Tests asserting on the event object stay deterministic.""" + emitter = TelemetryEmitter(endpoint="http://test:8001/events") + emitter._running = True + + with patch("src.config.settings") as mock_settings: + mock_settings.TELEMETRY.NAMESPACE = "test" + event = create_test_event() + before = event.model_dump() + emitter.emit(event) + after = event.model_dump() + + # The event instance must be unchanged by emit(). + assert before == after + assert "honcho_version" not in after diff --git a/tests/telemetry/test_events.py b/tests/telemetry/test_events.py index c60c3923..ef6f4ffb 100644 --- a/tests/telemetry/test_events.py +++ b/tests/telemetry/test_events.py @@ -1,7 +1,7 @@ # pyright: reportUnknownParameterType=false, reportMissingParameterType=false, reportUnusedParameter=false """Unit tests for telemetry event classes. -Tests all 12 event types for: +Tests all telemetry event types for: - Correct instantiation with required fields - event_type(), schema_version(), category() class methods - get_resource_id() returns expected format @@ -21,10 +21,16 @@ from src.telemetry.events.agent import ( AgentToolPeerCardUpdatedEvent, AgentToolSummaryCreatedEvent, ) +from src.telemetry.events.api import ( + FileUploadedEvent, + GetContextEvent, + MessageCreatedEvent, +) from src.telemetry.events.base import BaseEvent, generate_event_id from src.telemetry.events.deletion import DeletionCompletedEvent from src.telemetry.events.dialectic import DialecticCompletedEvent from src.telemetry.events.dream import DreamRunEvent, DreamSpecialistEvent +from src.telemetry.events.llm import CallPurpose, LLMCallCompletedEvent from src.telemetry.events.reconciliation import ( CleanupStaleItemsCompletedEvent, SyncVectorsCompletedEvent, @@ -91,6 +97,27 @@ class TestGenerateEventId: # Base64url encoded 16 bytes = 22 chars (without padding) assert len(event_id) == 4 + 22 # "evt_" + 22 chars + def test_honcho_version_changes_id(self, fixed_timestamp: datetime): + """Same event from different deploys must produce distinct IDs so + downstream dedupe doesn't silently merge events whose payload shape + may have shifted between versions.""" + id_a = generate_event_id( + "test.event", fixed_timestamp, "resource_1", honcho_version="2.0.0" + ) + id_b = generate_event_id( + "test.event", fixed_timestamp, "resource_1", honcho_version="2.0.1" + ) + assert id_a != id_b + + def test_honcho_version_none_matches_empty(self, fixed_timestamp: datetime): + """None and unset version segments are equivalent — backwards- + compatible with callers that don't pass the new kwarg yet.""" + id_default = generate_event_id("test.event", fixed_timestamp, "resource_1") + id_explicit_none = generate_event_id( + "test.event", fixed_timestamp, "resource_1", honcho_version=None + ) + assert id_default == id_explicit_none + # ============================================================================= # Tests for BaseEvent class @@ -116,6 +143,7 @@ class TestBaseEvent: llm_call_ms=100.0, total_duration_ms=110.0, input_tokens=100, + total_input_tokens=150, output_tokens=50, ) @@ -142,10 +170,6 @@ class TestRepresentationCompletedEvent: """event_type() returns correct value.""" assert RepresentationCompletedEvent.event_type() == "representation.completed" - def test_schema_version(self): - """schema_version() returns correct value.""" - assert RepresentationCompletedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert RepresentationCompletedEvent.category() == "representation" @@ -181,6 +205,280 @@ class TestRepresentationCompletedEvent: assert data["explicit_conclusion_count"] == 5 +# ============================================================================= +# Tests for LLMCallCompletedEvent () +# ============================================================================= + + +class TestLLMCallCompletedEvent: + """Tests for the LLMCallCompletedEvent.""" + + def test_event_type(self): + assert LLMCallCompletedEvent.event_type() == "llm.call.completed" + + def test_category(self): + assert LLMCallCompletedEvent.category() == "llm" + + def test_volume_class(self): + # event must be high_volume so the sampler picks it up. + assert LLMCallCompletedEvent.volume_class() == "high_volume" + + def test_get_resource_id_includes_attempt( + self, sample_llm_call_event: LLMCallCompletedEvent + ): + # Resource id must include attempt so multiple retry attempts in one + # iteration get distinct deterministic ids. + assert ( + sample_llm_call_event.get_resource_id() + == "abc12345:1:1:anthropic:claude-sonnet-4-5" + ) + + def test_call_purpose_enum_values(self): + # The closed taxonomy used by callers. + assert CallPurpose.DERIVER_REPRESENTATION.value == "deriver.representation" + assert CallPurpose.DIALECTIC_ANSWER.value == "dialectic.answer" + assert CallPurpose.DREAM_DEDUCTION.value == "dream.deduction" + assert CallPurpose.DREAM_INDUCTION.value == "dream.induction" + assert CallPurpose.SUMMARY_SHORT.value == "summary.short" + assert CallPurpose.SUMMARY_LONG.value == "summary.long" + + def test_error_outcome_with_error_class(self, fixed_timestamp: datetime): + event = LLMCallCompletedEvent( + timestamp=fixed_timestamp, + transport="openai", + model="gpt-4", + effective_max_output_tokens=512, + outcome="error", + is_final_attempt=True, + error_class="RateLimitError", + attempt=3, + retry_attempts=3, + was_fallback=True, + duration_ms=200.0, + ) + assert event.outcome == "error" + assert event.error_class == "RateLimitError" + assert event.is_final_attempt is True + # Token fields default to 0 when no result was produced. + assert event.provider_input_tokens == 0 + assert event.provider_output_tokens == 0 + + def test_stream_placeholder_has_zero_tokens(self, fixed_timestamp: datetime): + event = LLMCallCompletedEvent( + timestamp=fixed_timestamp, + transport="anthropic", + model="claude-sonnet-4-5", + effective_max_output_tokens=2048, + outcome="success", + is_final_attempt=False, + attempt=1, + retry_attempts=3, + was_fallback=False, + duration_ms=0.0, + was_stream=True, + ) + assert event.was_stream is True + assert event.provider_input_tokens == 0 + assert event.provider_output_tokens == 0 + + +# ============================================================================= +# Tests for MessageCreatedEvent +# ============================================================================= + + +class TestMessageCreatedEvent: + """Tests for MessageCreatedEvent.""" + + def test_event_type(self): + """event_type() returns correct value.""" + assert MessageCreatedEvent.event_type() == "message.created" + + def test_category(self): + """category() returns correct value.""" + assert MessageCreatedEvent.category() == "api" + + def test_get_resource_id(self, sample_message_created_event: MessageCreatedEvent): + """get_resource_id() keys on workspace:session:source:last_message_id.""" + assert ( + sample_message_created_event.get_resource_id() + == "test_workspace:test_session:api:msg_abc123_fixture_____" + ) + + def test_source_defaults_to_api(self, fixed_timestamp: datetime): + """source defaults to api.""" + event = MessageCreatedEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + session_name="test_session", + message_count=1, + total_tokens=100, + last_message_id="msg_default_source_____", + ) + assert event.source == "api" + + def test_distinct_batches_get_distinct_ids(self, fixed_timestamp: datetime): + """Two batches of the same size in the same session+source must produce + different event ids — the previous (v1) key collided here.""" + e1 = MessageCreatedEvent( + timestamp=fixed_timestamp, + workspace_name="ws", + session_name="sess", + message_count=5, + total_tokens=500, + source="api", + last_message_id="msg_first_batch________", + ) + e2 = MessageCreatedEvent( + timestamp=fixed_timestamp, + workspace_name="ws", + session_name="sess", + message_count=5, + total_tokens=500, + source="api", + last_message_id="msg_second_batch_______", + ) + assert e1.generate_id() != e2.generate_id() + assert e1.get_resource_id() != e2.get_resource_id() + + +# ============================================================================= +# Tests for FileUploadedEvent +# ============================================================================= + + +class TestFileUploadedEvent: + """Tests for FileUploadedEvent.""" + + def test_event_type(self): + """event_type() returns correct value.""" + assert FileUploadedEvent.event_type() == "file.uploaded" + + def test_category(self): + """category() returns correct value.""" + assert FileUploadedEvent.category() == "api" + + def test_get_resource_id(self, sample_file_uploaded_event: FileUploadedEvent): + """get_resource_id() returns workspace:session:file format.""" + assert ( + sample_file_uploaded_event.get_resource_id() + == "test_workspace:test_session:file_123" + ) + + def test_optional_file_fields(self, fixed_timestamp: datetime): + """filename, content_type, and file_size_bytes are optional.""" + event = FileUploadedEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + session_name="test_session", + peer_name="user_peer", + file_id="file_123", + message_count=1, + total_tokens=100, + ) + assert event.filename is None + assert event.content_type is None + assert event.file_size_bytes is None + + +# ============================================================================= +# Tests for GetContextEvent +# ============================================================================= + + +class TestGetContextEvent: + """Tests for GetContextEvent.""" + + def test_event_type(self): + """event_type() returns correct value.""" + assert GetContextEvent.event_type() == "context.retrieved" + + def test_category(self): + """category() returns correct value.""" + assert GetContextEvent.category() == "api" + + def test_get_resource_id_session(self, sample_get_context_event: GetContextEvent): + """session context resource ID includes workspace and session. + + Uses empty-string sentinel for unset peer/target so that a peer + literally named "none" can't collide with the absent-peer case. + """ + assert ( + sample_get_context_event.get_resource_id() + == "test_workspace:session:test_session::" + ) + + def test_get_resource_id_disambiguates_peer_named_none( + self, fixed_timestamp: datetime + ): + """Regression: a peer literally named "none" must NOT collide with + absent-peer resource ids. Empty-string sentinel guards this. + """ + absent = GetContextEvent( + timestamp=fixed_timestamp, + workspace_name="ws", + context_scope="peer", + total_duration_ms=1.0, + ) + literal_none = GetContextEvent( + timestamp=fixed_timestamp, + workspace_name="ws", + context_scope="peer", + peer_name="none", + target_name="none", + total_duration_ms=1.0, + ) + assert absent.get_resource_id() != literal_none.get_resource_id() + + def test_get_resource_id_peer(self, fixed_timestamp: datetime): + """peer context resource ID includes observer and observed peers.""" + event = GetContextEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + context_scope="peer", + peer_name="observer", + target_name="observed", + total_duration_ms=10.0, + ) + assert event.get_resource_id() == "test_workspace:peer:observer:observed" + + def test_context_defaults(self, fixed_timestamp: datetime): + """Context booleans and counts have conservative defaults.""" + event = GetContextEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + context_scope="session", + session_name="test_session", + total_duration_ms=10.0, + ) + assert event.message_count == 0 + assert event.has_summary is False + assert event.has_representation is False + assert event.include_summary is None + assert event.tokens_requested is None + assert event.peer_perspective_provided is False + + def test_session_context_can_record_raw_request_options( + self, fixed_timestamp: datetime + ): + """Raw request options can be recorded independently of resolved values.""" + event = GetContextEvent( + timestamp=fixed_timestamp, + workspace_name="test_workspace", + context_scope="session", + session_name="test_session", + peer_name="observer", + target_name="observed", + tokens_requested=8000, + include_summary=False, + peer_perspective_provided=True, + total_duration_ms=10.0, + ) + assert event.tokens_requested == 8000 + assert event.include_summary is False + assert event.peer_perspective_provided is True + + # ============================================================================= # Tests for DreamRunEvent # ============================================================================= @@ -193,10 +491,6 @@ class TestDreamRunEvent: """event_type() returns correct value.""" assert DreamRunEvent.event_type() == "dream.run" - def test_schema_version(self): - """schema_version() returns correct value.""" - assert DreamRunEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert DreamRunEvent.category() == "dream" @@ -229,10 +523,6 @@ class TestDreamSpecialistEvent: """event_type() returns correct value.""" assert DreamSpecialistEvent.event_type() == "dream.specialist" - def test_schema_version(self): - """schema_version() returns correct value.""" - assert DreamSpecialistEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert DreamSpecialistEvent.category() == "dream" @@ -262,10 +552,6 @@ class TestDialecticCompletedEvent: """event_type() returns correct value.""" assert DialecticCompletedEvent.event_type() == "dialectic.completed" - def test_schema_version(self): - """schema_version() returns correct value.""" - assert DialecticCompletedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert DialecticCompletedEvent.category() == "dialectic" @@ -324,10 +610,6 @@ class TestAgentIterationEvent: """event_type() returns correct value.""" assert AgentIterationEvent.event_type() == "agent.iteration" - def test_schema_version(self): - """schema_version() returns correct value.""" - assert AgentIterationEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert AgentIterationEvent.category() == "agent" @@ -380,10 +662,6 @@ class TestAgentToolConclusionsCreatedEvent: == "agent.tool.conclusions.created" ) - def test_schema_version(self): - """schema_version() returns correct value.""" - assert AgentToolConclusionsCreatedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert AgentToolConclusionsCreatedEvent.category() == "agent" @@ -420,10 +698,6 @@ class TestAgentToolConclusionsDeletedEvent: == "agent.tool.conclusions.deleted" ) - def test_schema_version(self): - """schema_version() returns correct value.""" - assert AgentToolConclusionsDeletedEvent.schema_version() == 2 - def test_category(self): """category() returns correct value.""" assert AgentToolConclusionsDeletedEvent.category() == "agent" @@ -452,10 +726,6 @@ class TestAgentToolPeerCardUpdatedEvent: AgentToolPeerCardUpdatedEvent.event_type() == "agent.tool.peer_card.updated" ) - def test_schema_version(self): - """schema_version() returns correct value.""" - assert AgentToolPeerCardUpdatedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert AgentToolPeerCardUpdatedEvent.category() == "agent" @@ -482,10 +752,6 @@ class TestAgentToolSummaryCreatedEvent: """event_type() returns correct value.""" assert AgentToolSummaryCreatedEvent.event_type() == "agent.tool.summary.created" - def test_schema_version(self): - """schema_version() returns correct value.""" - assert AgentToolSummaryCreatedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert AgentToolSummaryCreatedEvent.category() == "agent" @@ -532,10 +798,6 @@ class TestDeletionCompletedEvent: """event_type() returns correct value.""" assert DeletionCompletedEvent.event_type() == "deletion.completed" - def test_schema_version(self): - """schema_version() returns correct value.""" - assert DeletionCompletedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert DeletionCompletedEvent.category() == "deletion" @@ -601,10 +863,6 @@ class TestSyncVectorsCompletedEvent: == "reconciliation.sync_vectors.completed" ) - def test_schema_version(self): - """schema_version() returns correct value.""" - assert SyncVectorsCompletedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert SyncVectorsCompletedEvent.category() == "reconciliation" @@ -642,10 +900,6 @@ class TestCleanupStaleItemsCompletedEvent: == "reconciliation.cleanup_stale_items.completed" ) - def test_schema_version(self): - """schema_version() returns correct value.""" - assert CleanupStaleItemsCompletedEvent.schema_version() == 1 - def test_category(self): """category() returns correct value.""" assert CleanupStaleItemsCompletedEvent.category() == "reconciliation" @@ -665,6 +919,25 @@ class TestCleanupStaleItemsCompletedEvent: assert event.documents_cleaned == 0 assert event.queue_items_cleaned == 0 + def test_queue_items_cleaned_round_trips_through_pydantic( + self, fixed_timestamp: datetime + ): + """Regression: `queue_items_cleaned` is a real field, not just + plumbing. Previously the consumer emit site dropped the captured + `deleted_count` and the field always defaulted to 0 on the wire. + """ + event = CleanupStaleItemsCompletedEvent( + timestamp=fixed_timestamp, + total_duration_ms=500.0, + queue_items_cleaned=42, + ) + assert event.queue_items_cleaned == 42 + # Serialize → deserialize to ensure the field crosses the wire. + data = event.model_dump(mode="json") + assert data["queue_items_cleaned"] == 42 + round_tripped = CleanupStaleItemsCompletedEvent.model_validate(data) + assert round_tripped.queue_items_cleaned == 42 + # ============================================================================= # Parametrized tests across all event types diff --git a/tests/telemetry/test_representation_v2_fields.py b/tests/telemetry/test_representation_v2_fields.py new file mode 100644 index 00000000..669da96b --- /dev/null +++ b/tests/telemetry/test_representation_v2_fields.py @@ -0,0 +1,194 @@ +# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false +"""tests for RepresentationCompletedEvent additive fields + truncation. + +Targets: +- Schema stays at v2 (additive, no bump). Existing `input_tokens` semantics + unchanged. +- fields are defaultable (no breakage for callers that ignore them) + and round-trip through Pydantic serialization. +- `HonchoLLMCallResponse.hit_input_token_cap` defaults to False but can be + flipped by the tool-less cap-detection path in `src/llm/api.py`. +""" + +from __future__ import annotations + +from src.llm.types import HonchoLLMCallResponse +from src.telemetry.events.representation import RepresentationCompletedEvent + + +class TestRepresentationV2AdditiveFields: + def test_schema_stays_at_v2(self): + """is additive — schema_version must NOT bump to 3.""" + assert RepresentationCompletedEvent.schema_version() == 2 + + def test_new_fields_are_optional(self): + """Existing callers must keep working without supplying any new + fields. All new fields default.""" + event = RepresentationCompletedEvent( + workspace_name="ws", + session_name="s", + observed="user", + queue_items_processed=1, + earliest_message_id="m1", + latest_message_id="m1", + message_count=1, + explicit_conclusion_count=0, + context_preparation_ms=10.0, + llm_call_ms=100.0, + total_duration_ms=110.0, + input_tokens=100, + total_input_tokens=200, + output_tokens=50, + ) + # All fields land with defaults. + assert event.queued_message_count == 0 + assert event.prompt_message_count == 0 + assert event.prompt_message_tokens == 0 + assert event.extra_context_message_count == 0 + assert event.extra_context_tokens == 0 + assert event.prompt_scaffold_tokens == 0 + assert event.batch_max_tokens == 0 + assert event.max_input_tokens == 0 + assert event.was_flush_enabled is False + assert event.hit_batch_token_cap is False + assert event.hit_input_token_cap is False + assert event.observer_count == 0 + + def test_input_tokens_semantics_preserved(self): + """The downstream metering key must remain 'queued-message tokens'. + + Added many fields, but `input_tokens` MUST stay as the downstream + metering key for representation.completed. Don't rename or + repurpose without coordinating with consumers. + """ + event = RepresentationCompletedEvent( + workspace_name="ws", + session_name="s", + observed="user", + queue_items_processed=2, + earliest_message_id="m1", + latest_message_id="m5", + message_count=5, + explicit_conclusion_count=3, + context_preparation_ms=10.0, + llm_call_ms=100.0, + total_duration_ms=110.0, + input_tokens=300, # ← queued message tokens; the billing key + total_input_tokens=900, # provider-side total + output_tokens=50, + queued_message_count=2, + prompt_message_count=5, + prompt_message_tokens=800, + extra_context_message_count=3, + extra_context_tokens=500, + prompt_scaffold_tokens=100, + ) + # input_tokens and queued_message_count should describe the same set + # (queue items being reasoned about) — assert the conceptual link. + assert event.input_tokens == 300 + assert event.queued_message_count == 2 + # And the breakdown adds up: extra + scaffold + queued ≈ total provider input + # (it's an approximation — provider includes formatting overhead). + assert ( + event.extra_context_tokens + + event.input_tokens + + event.prompt_scaffold_tokens + == 900 + ) + assert event.total_input_tokens == 900 + + def test_cap_hit_flags(self): + event = RepresentationCompletedEvent( + workspace_name="ws", + session_name="s", + observed="user", + queue_items_processed=10, + earliest_message_id="m1", + latest_message_id="m10", + message_count=10, + explicit_conclusion_count=5, + context_preparation_ms=10.0, + llm_call_ms=100.0, + total_duration_ms=110.0, + input_tokens=20_000, + total_input_tokens=23_000, + output_tokens=500, + batch_max_tokens=20_000, + max_input_tokens=23_000, + was_flush_enabled=True, + hit_batch_token_cap=True, + hit_input_token_cap=True, + observer_count=2, + ) + assert event.was_flush_enabled is True + assert event.hit_batch_token_cap is True + assert event.hit_input_token_cap is True + assert event.batch_max_tokens == 20_000 + assert event.max_input_tokens == 23_000 + assert event.observer_count == 2 + + def test_model_dump_includes_new_fields(self): + event = RepresentationCompletedEvent( + workspace_name="ws", + session_name="s", + observed="user", + queue_items_processed=1, + earliest_message_id="m1", + latest_message_id="m1", + message_count=1, + explicit_conclusion_count=0, + context_preparation_ms=10.0, + llm_call_ms=100.0, + total_duration_ms=110.0, + input_tokens=100, + total_input_tokens=150, + output_tokens=50, + hit_batch_token_cap=True, + ) + data = event.model_dump(mode="json") + for field in ( + "queued_message_count", + "prompt_message_count", + "prompt_message_tokens", + "extra_context_message_count", + "extra_context_tokens", + "prompt_scaffold_tokens", + "batch_max_tokens", + "max_input_tokens", + "was_flush_enabled", + "hit_batch_token_cap", + "hit_input_token_cap", + "observer_count", + ): + assert field in data, f"missing field: {field}" + assert data["hit_batch_token_cap"] is True + + +class TestHitInputTokenCapFlag: + """`HonchoLLMCallResponse.hit_input_token_cap` is the bridge between the + tool-less cap-detection path in src/llm/api.py and the deriver's + `hit_input_token_cap` field on RepresentationCompletedEvent. + + The flag is token-based — it fires whenever the original input exceeded + `max_input_tokens`, whether or not message truncation could actually + shrink the input below cap (the deriver's single-prompt case can't). + """ + + def test_defaults_to_false(self): + response = HonchoLLMCallResponse( + content="hi", + input_tokens=10, + output_tokens=5, + finish_reasons=["stop"], + ) + assert response.hit_input_token_cap is False + + def test_can_be_flipped(self): + response = HonchoLLMCallResponse( + content="hi", + input_tokens=10, + output_tokens=5, + finish_reasons=["stop"], + ) + response.hit_input_token_cap = True + assert response.hit_input_token_cap is True diff --git a/tests/telemetry/test_summary_v2_fields.py b/tests/telemetry/test_summary_v2_fields.py new file mode 100644 index 00000000..83c10b9f --- /dev/null +++ b/tests/telemetry/test_summary_v2_fields.py @@ -0,0 +1,122 @@ +# pyright: reportPrivateUsage=false +"""tests for AgentToolSummaryCreatedEvent additive token breakdown. + +Targets: +- Schema bumps to v2. +- `input_tokens` semantic preserved (provider-side input). +- New breakdown fields (`previous_summary_tokens`, `message_tokens`, + `prompt_scaffold_tokens`) default cleanly and round-trip. +- The conceptual relationship: message_tokens + previous_summary_tokens + describes the *user-data* portion of the input; prompt_scaffold_tokens is + the static instruction portion. Together they approximate the provider + input_tokens (modulo formatting overhead). +""" + +from __future__ import annotations + +from src.telemetry.events.agent import AgentToolSummaryCreatedEvent + + +class TestAdditiveFields: + def test_new_fields_default_to_zero(self): + """Callers that omit the breakdown fields must construct valid events.""" + event = AgentToolSummaryCreatedEvent( + run_id="r", + iteration=0, + parent_category="deriver", + agent_type="summarizer", + workspace_name="ws", + session_name="s", + message_id="m1", + message_count=10, + message_seq_in_session=10, + summary_type="short", + input_tokens=1000, + output_tokens=100, + ) + assert event.previous_summary_tokens == 0 + assert event.message_tokens == 0 + assert event.prompt_scaffold_tokens == 0 + + def test_input_tokens_semantic_preserved(self): + """`input_tokens` continues to be the provider-side LLM input count. + We deliberately do NOT add a redundant `provider_input_tokens` — + the existing field already serves that purpose, and a duplicate + would silently fork downstream queries.""" + event = AgentToolSummaryCreatedEvent( + run_id="r", + iteration=0, + parent_category="deriver", + agent_type="summarizer", + workspace_name="ws", + session_name="s", + message_id="m1", + message_count=10, + message_seq_in_session=10, + summary_type="long", + input_tokens=5000, # provider-reported total + output_tokens=400, + previous_summary_tokens=500, + message_tokens=4000, + prompt_scaffold_tokens=400, + ) + # message + prev_summary + scaffold ≈ input_tokens (small drift from + # provider-side formatting overhead is expected). + breakdown_sum = ( + event.message_tokens + + event.previous_summary_tokens + + event.prompt_scaffold_tokens + ) + assert breakdown_sum <= event.input_tokens + 200 # allow small overhead + assert event.input_tokens == 5000 + + def test_first_summary_has_zero_previous_summary_tokens(self): + """When there's no prior summary for the session, the breakdown + carries `previous_summary_tokens=0` so analytics can distinguish + first-summary calls from rollup calls.""" + event = AgentToolSummaryCreatedEvent( + run_id="r", + iteration=0, + parent_category="deriver", + agent_type="summarizer", + workspace_name="ws", + session_name="s", + message_id="m1", + message_count=5, + message_seq_in_session=5, + summary_type="short", + input_tokens=1500, + output_tokens=150, + previous_summary_tokens=0, # first summary for this session + message_tokens=1200, + prompt_scaffold_tokens=300, + ) + assert event.previous_summary_tokens == 0 + assert event.message_tokens > 0 + + def test_model_dump_includes_breakdown_fields(self): + event = AgentToolSummaryCreatedEvent( + run_id="r", + iteration=0, + parent_category="deriver", + agent_type="summarizer", + workspace_name="ws", + session_name="s", + message_id="m1", + message_count=10, + message_seq_in_session=10, + summary_type="short", + input_tokens=1000, + output_tokens=100, + previous_summary_tokens=100, + message_tokens=700, + prompt_scaffold_tokens=200, + ) + data = event.model_dump(mode="json") + for field in ( + "previous_summary_tokens", + "message_tokens", + "prompt_scaffold_tokens", + ): + assert field in data, f"missing field: {field}" + assert data["message_tokens"] == 700 diff --git a/tests/test_models_vector_dim.py b/tests/test_models_vector_dim.py index 539e359d..b5f4aea0 100644 --- a/tests/test_models_vector_dim.py +++ b/tests/test_models_vector_dim.py @@ -1,4 +1,4 @@ -"""Phase 1: verify src/models.py honors EMBEDDING_VECTOR_DIMENSIONS at import time.""" +"""Verify src/models.py honors EMBEDDING_VECTOR_DIMENSIONS at import time.""" from __future__ import annotations diff --git a/tests/utils/test_agent_tools.py b/tests/utils/test_agent_tools.py index 95406587..b300417d 100644 --- a/tests/utils/test_agent_tools.py +++ b/tests/utils/test_agent_tools.py @@ -253,7 +253,8 @@ class TestCreateObservations: result = await _handle_create_observations(ctx, {"observations": []}) assert "ERROR" in result - assert "empty" in result.lower() + # Handlers may return ToolResult (); str() returns .content. + assert "empty" in str(result).lower() async def test_batch_embedding_failure_falls_back_to_individual_embeds( self, @@ -414,8 +415,10 @@ class TestCreateObservations: result = await create_observations( observations=[ - schemas.ObservationInput(content=" ", level="explicit"), - schemas.ObservationInput(content=" trimmed observation ", level="explicit"), + schemas.ObservationInput(content=" ", level="explicit"), + schemas.ObservationInput( + content=" trimmed observation ", level="explicit" + ), ], observer=peer1.name, observed=peer2.name, @@ -753,8 +756,12 @@ class TestSearchMessages: result = await _handle_search_messages(ctx, {"query": "test message"}) - # Should return some result (may be empty if semantic search doesn't match) - assert isinstance(result, str) + # handler may return ToolResult (with search metadata) or + # a plain str. Both carry the result text; just check it's + # introspectable as string content. + from src.utils.types import ToolResult + + assert isinstance(result, str | ToolResult) @pytest.mark.asyncio @@ -895,7 +902,7 @@ class TestGetRecentHistory: result = await _handle_get_recent_history(ctx, {}) assert "Conversation history" in result - assert "messages" in result.lower() + assert "messages" in str(result).lower() async def test_without_session_uses_observed( self, @@ -1043,7 +1050,7 @@ class TestUpdatePeerCard: workspace, peer1, peer2, _, _, _ = tool_test_data ctx = make_tool_context() - oversized = ["Name: John", " Name: John ", "", " "] + oversized = ["Name: John", " Name: John ", "", " "] oversized.extend([f"Fact {i}" for i in range(MAX_PEER_CARD_FACTS + 5)]) await _handle_update_peer_card(ctx, {"content": oversized}) @@ -1078,7 +1085,7 @@ class TestUpdatePeerCard: # Now attempt to update with None — should be a no-op result = await _handle_update_peer_card(ctx, {"content": None}) - assert "empty" in result.lower() + assert "empty" in str(result).lower() # Refresh the observer so the identity map picks up the committed update await db_session.refresh(peer1) @@ -1107,7 +1114,7 @@ class TestUpdatePeerCard: # Now attempt to update with empty list — should be a no-op result = await _handle_update_peer_card(ctx, {"content": []}) - assert "empty" in result.lower() + assert "empty" in str(result).lower() # Refresh the observer so the identity map picks up the committed update await db_session.refresh(peer1) From 10f72a7d0d937fc0b5419270142d88f5b1f3ae90 Mon Sep 17 00:00:00 2001 From: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> Date: Thu, 21 May 2026 12:31:53 -0400 Subject: [PATCH 3/7] fix(sdk): add peer field to session creation methods (#705) --- .pre-commit-config.yaml | 2 +- sdks/python/src/honcho/aio.py | 13 ++++++ sdks/python/src/honcho/client.py | 19 ++++++++- sdks/typescript/__tests__/session.test.ts | 36 +++++++++++++++++ sdks/typescript/src/client.ts | 20 ++++++++- tests/sdk/test_session.py | 49 +++++++++++++++++++++++ 6 files changed, 135 insertions(+), 4 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 4ba8ab43..c093bf34 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -60,7 +60,7 @@ repos: language: system files: ^(src/|tests/|sdks/python/|scripts/).*\.py$ require_serial: true - pass_filenames: false + pass_filenames: true # Run main application tests - id: pytest-main diff --git a/sdks/python/src/honcho/aio.py b/sdks/python/src/honcho/aio.py index 0edcc60f..04ca42fb 100644 --- a/sdks/python/src/honcho/aio.py +++ b/sdks/python/src/honcho/aio.py @@ -243,6 +243,14 @@ class HonchoAio(AsyncMetadataConfigMixin): *, metadata: dict[str, object] | None = None, configuration: SessionConfiguration | None = None, + peers: str + | PeerBase + | tuple[str, SessionPeerConfig] + | tuple[PeerBase, SessionPeerConfig] + | list[PeerBase | str] + | list[tuple[PeerBase | str, SessionPeerConfig]] + | list[PeerBase | str | tuple[PeerBase | str, SessionPeerConfig]] + | None = None, ) -> Session: """ Get or create a session with the given ID asynchronously. @@ -251,6 +259,9 @@ class HonchoAio(AsyncMetadataConfigMixin): id: Unique identifier for the session within the workspace. metadata: Optional metadata dictionary to associate with this session. configuration: Optional configuration to set for this session. + peers: Optional peers to attach to the session at creation. Accepts the + same shape as Session.add_peers (peer ID string, Peer object, list + of either, or tuples with SessionPeerConfig). Returns: A Session object with cached values from the API response. @@ -261,6 +272,8 @@ class HonchoAio(AsyncMetadataConfigMixin): body["metadata"] = metadata if configuration is not None: body["configuration"] = configuration.model_dump(exclude_none=True) + if peers is not None: + body["peers"] = normalize_peers_to_dict(peers) data = await self._honcho._async_http_client.post( routes.sessions(self._honcho.workspace_id), body=body diff --git a/sdks/python/src/honcho/client.py b/sdks/python/src/honcho/client.py index 969ff741..1dbd3a8c 100644 --- a/sdks/python/src/honcho/client.py +++ b/sdks/python/src/honcho/client.py @@ -17,6 +17,7 @@ from .api_types import ( PeerResponse, QueueStatusResponse, SessionConfiguration, + SessionPeerConfig, SessionResponse, WorkspaceConfiguration, WorkspaceResponse, @@ -28,7 +29,7 @@ from .mixins import MetadataConfigMixin from .pagination import SyncPage from .peer import Peer from .session import Session -from .utils import resolve_id +from .utils import normalize_peers_to_dict, resolve_id logger = logging.getLogger(__name__) @@ -400,6 +401,17 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul None, description="Optional configuration to set for this session. If set, will get/create session immediately with flags.", ), + peers: str + | PeerBase + | tuple[str, SessionPeerConfig] + | tuple[PeerBase, SessionPeerConfig] + | list[PeerBase | str] + | list[tuple[PeerBase | str, SessionPeerConfig]] + | list[PeerBase | str | tuple[PeerBase | str, SessionPeerConfig]] + | None = Field( + None, + description="Optional peers to attach to the session at creation. Accepts the same shape as Session.add_peers.", + ), ) -> Session: """ Get or create a session with the given ID. @@ -411,6 +423,9 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul id: Unique identifier for the session within the workspace. metadata: Optional metadata dictionary to associate with this session. configuration: Optional configuration to set for this session. + peers: Optional peers to attach to the session at creation. Accepts the + same shape as Session.add_peers (peer ID string, Peer object, list + of either, or tuples with SessionPeerConfig). Returns: A Session object with cached metadata, configuration, created_at, and is_active. @@ -421,6 +436,8 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul body["metadata"] = metadata if configuration is not None: body["configuration"] = configuration.model_dump(exclude_none=True) + if peers is not None: + body["peers"] = normalize_peers_to_dict(peers) data = self._http.post(routes.sessions(self.workspace_id), body=body) session_data = SessionResponse.model_validate(data) diff --git a/sdks/typescript/__tests__/session.test.ts b/sdks/typescript/__tests__/session.test.ts index b9f4cdeb..f7f2f838 100644 --- a/sdks/typescript/__tests__/session.test.ts +++ b/sdks/typescript/__tests__/session.test.ts @@ -95,6 +95,42 @@ describe('Session', () => { expect(session1.id).toBe(session2.id) expect(session2.metadata).toEqual({ version: 2 }) }) + + test('creates session with peers from string array', async () => { + const session = await client.session('session-with-peers-strings', { + peers: ['create-peer-a', 'create-peer-b'], + }) + + const peers = await session.peers() + const ids = peers.map((p) => p.id) + expect(ids).toContain('create-peer-a') + expect(ids).toContain('create-peer-b') + }) + + test('creates session with peers from Peer objects', async () => { + const peerA = await client.peer('create-obj-peer-a') + const peerB = await client.peer('create-obj-peer-b') + const session = await client.session('session-with-peer-objects', { + peers: [peerA, peerB], + }) + + const peers = await session.peers() + const ids = peers.map((p) => p.id) + expect(ids).toContain('create-obj-peer-a') + expect(ids).toContain('create-obj-peer-b') + }) + + test('creates session with peers and per-peer config', async () => { + const session = await client.session('session-with-peer-config', { + peers: [ + ['create-config-peer', { observeMe: true, observeOthers: false }], + ], + }) + + const config = await session.getPeerConfiguration('create-config-peer') + expect(config.observeMe).toBe(true) + expect(config.observeOthers).toBe(false) + }) }) // =========================================================================== diff --git a/sdks/typescript/src/client.ts b/sdks/typescript/src/client.ts index 1270a319..4b510945 100644 --- a/sdks/typescript/src/client.ts +++ b/sdks/typescript/src/client.ts @@ -22,6 +22,8 @@ import { HonchoConfigSchema, LimitSchema, normalizeListOptions, + type PeerAddition, + PeerAdditionToApiSchema, type PeerConfig, PeerConfigSchema, PeerIdSchema, @@ -338,6 +340,10 @@ export class Honcho { id: string metadata?: Record configuration?: SessionConfig + peers?: Record< + string, + { observe_me?: boolean | null; observe_others?: boolean | null } + > } ): Promise { return this._http.post( @@ -347,6 +353,7 @@ export class Honcho { id: params.id, metadata: params.metadata, configuration: sessionConfigToApi(params.configuration), + peers: params.peers, }, } ) @@ -488,10 +495,13 @@ export class Honcho { * @param id - Unique identifier for the session within the workspace. Should be a * stable identifier that can be used consistently to reference the * same conversation - * @param metadata - Optional metadata dictionary to associate with this session. + * @param options.metadata - Optional metadata dictionary to associate with this session. * If set, will get/create session immediately with metadata. - * @param configuration - Optional configuration to set for this session. + * @param options.configuration - Optional configuration to set for this session. * If set, will get/create session immediately with flags. + * @param options.peers - Optional peers to attach to the session at creation. + * Accepts the same shape as `session.addPeers()` (peer ID strings, + * Peer objects, arrays of either, or a record with per-peer config). * @returns Promise resolving to a Session object that can be used to add peers, * send messages, and manage conversation context * @throws Error if the session ID is empty or invalid @@ -501,6 +511,7 @@ export class Honcho { options?: { metadata?: SessionMetadata configuration?: SessionConfig + peers?: PeerAddition } ): Promise { await this._ensureWorkspace() @@ -511,11 +522,16 @@ export class Honcho { const validatedConfiguration = options?.configuration ? SessionConfigSchema.parse(options.configuration) : undefined + const validatedPeers = + options?.peers !== undefined + ? PeerAdditionToApiSchema.parse(options.peers) + : undefined const sessionData = await this._getOrCreateSession(this.workspaceId, { id: validatedId, configuration: validatedConfiguration, metadata: validatedMetadata, + peers: validatedPeers, }) return new Session( validatedId, diff --git a/tests/sdk/test_session.py b/tests/sdk/test_session.py index 3a618891..a15693fa 100644 --- a/tests/sdk/test_session.py +++ b/tests/sdk/test_session.py @@ -200,6 +200,55 @@ async def test_session_peer_config(client_fixture: tuple[Honcho, str]): assert retrieved_config.observe_others +@pytest.mark.asyncio +async def test_session_create_with_peers(client_fixture: tuple[Honcho, str]): + """ + Tests creating a session with peers attached in a single call. + """ + honcho_client, client_type = client_fixture + + if client_type == "async": + session = await honcho_client.aio.session( + id="test-session-create-peers-async", + peers=["create-peer-async-a", "create-peer-async-b"], + ) + assert isinstance(session, Session) + peers = await session.aio.peers() + peer_ids = {p.id for p in peers} + assert "create-peer-async-a" in peer_ids + assert "create-peer-async-b" in peer_ids + + config = SessionPeerConfig(observe_me=True, observe_others=False) + peer = await honcho_client.aio.peer(id="create-peer-async-config") + session_with_config = await honcho_client.aio.session( + id="test-session-create-peers-config-async", + peers=[(peer, config)], + ) + retrieved = await session_with_config.aio.get_peer_configuration(peer) + assert retrieved.observe_me is True + assert retrieved.observe_others is False + else: + session = honcho_client.session( + id="test-session-create-peers", + peers=["create-peer-a", "create-peer-b"], + ) + assert isinstance(session, Session) + peers = session.peers() + peer_ids = {p.id for p in peers} + assert "create-peer-a" in peer_ids + assert "create-peer-b" in peer_ids + + config = SessionPeerConfig(observe_me=True, observe_others=False) + peer = honcho_client.peer(id="create-peer-config") + session_with_config = honcho_client.session( + id="test-session-create-peers-config", + peers=[(peer, config)], + ) + retrieved = session_with_config.get_peer_configuration(peer) + assert retrieved.observe_me is True + assert retrieved.observe_others is False + + @pytest.mark.asyncio async def test_session_messages(client_fixture: tuple[Honcho, str]): """ From 4f579d5c6662c0d8b5394b3c77217436f7438619 Mon Sep 17 00:00:00 2001 From: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> Date: Thu, 21 May 2026 13:25:34 -0400 Subject: [PATCH 4/7] fix: reframe peer card prompts as stable identity markers (#686) * fix: reframe peer card prompts as stable identity markers * fix: remove strict parameter validation for thinking on anthropic and openai transports * fix(dreamer): Add backwards compatability instructions for peer card prompt --- src/dreamer/specialists.py | 118 ++++++++------ src/llm/backends/anthropic.py | 12 +- src/llm/backends/openai.py | 23 +-- src/utils/agent_tools.py | 124 +++++++++++++-- tests/dreamer/test_model_config_usage.py | 57 ++++++- tests/llm/test_backends/test_anthropic.py | 37 +++-- tests/llm/test_backends/test_openai.py | 84 ++++++++-- tests/utils/test_agent_tools.py | 185 ++++++++++++++++++++-- 8 files changed, 531 insertions(+), 109 deletions(-) diff --git a/src/dreamer/specialists.py b/src/dreamer/specialists.py index 4d3ecc97..af67b989 100644 --- a/src/dreamer/specialists.py +++ b/src/dreamer/specialists.py @@ -75,9 +75,12 @@ class BaseSpecialist(ABC): """Base class for agentic specialists.""" name: str = "base" + # Whether this specialist is allowed to write to the peer card. Defaults to True; + # specialists that should never touch the card (e.g., induction) override to False. + can_update_peer_card: bool = True # Subclasses can override to customize the peer card update instruction peer_card_update_instruction: str = ( - "Only update this with durable profile facts via `update_peer_card`." + "Only update this with durable identity markers via `update_peer_card`." ) @abstractmethod @@ -195,8 +198,10 @@ If you update it, send the full deduplicated list and remove stale entries. db, workspace_name, schemas.PeerCreate(name=observed) ) - # Determine if peer card tools should be included - peer_card_enabled = ( + # Determine if peer card tools should be included. Specialists that + # cannot write to the peer card (e.g., induction) skip the fetch and + # the prompt section entirely. + peer_card_enabled = self.can_update_peer_card and ( configuration is None or configuration.peer_card.create ) @@ -434,7 +439,7 @@ class DeductionSpecialist(BaseSpecialist): """ name: str = "deduction" - peer_card_update_instruction: str = "Update this with `update_peer_card` only for stable biographical/profile facts." + peer_card_update_instruction: str = "Update this with `update_peer_card` only for stable identity markers. See the PEER CARD section in the system prompt for the allowed entry kinds and rules." def get_tools(self, *, peer_card_enabled: bool = True) -> list[dict[str, Any]]: if peer_card_enabled: @@ -462,26 +467,61 @@ class DeductionSpecialist(BaseSpecialist): ) -> str: peer_card_section = "" if peer_card_enabled: - peer_card_section = """ + peer_card_section = f""" ## PEER CARD (REQUIRED) -The peer card is a summary of stable biographical facts. You MUST update it when you learn: -- Name, age, location, occupation -- Family members and relationships -- Standing instructions ("call me X", "don't mention Y") -- Core preferences and traits +The peer card is {observed}'s identity store: stable identity markers that distinguish this entity from others and persist across interactions. Behavior, tendencies, transient state, and episodic facts belong in observations, not on the peer card. -Never add temporary event summaries, one-off conclusions, reasoning traces, or contradiction notes. +A peer can be anything with identity that changes over time — a human, an agent, a codebase, a team, an organization. Do not assume {observed} is human. Do not require any field; empty is the correct output when evidence is absent. -Format entries as: -- Plain facts: "Name: Alice", "Works at Google", "Lives in NYC" -- `INSTRUCTION: ...` for standing instructions -- `PREFERENCE: ...` for preferences -- `TRAIT: ...` for personality traits +### Allowed entry kinds -Call `update_peer_card` with the complete updated list when you have new biographical info. -Keep it concise (max 40 entries), deduplicated, and current.""" +Each entry must start with one of these four prefixes (exact case, followed by a space): + +- `IDENTITY: ...` — canonical name, kind, aliases, IDs + - `IDENTITY: Name: Alice` + - `IDENTITY: Kind: Python monorepo` + - `IDENTITY: Version: 4.2` + - `IDENTITY: Aliases: alice@example.com` +- `ATTRIBUTE: ...` — stable durable property of the entity (including explicitly stated standing preferences) + - `ATTRIBUTE: Location: NYC` + - `ATTRIBUTE: Language: Python` + - `ATTRIBUTE: Prefers tea` + - `ATTRIBUTE: Charter: ship Honcho infrastructure` +- `RELATIONSHIP: ...` — durable link to another entity + - `RELATIONSHIP: Spouse: Bob` + - `RELATIONSHIP: Maintainer: vineeth` + - `RELATIONSHIP: Members: vineeth, rajat` +- `INSTRUCTION: ...` — standing rule of engagement that {observed} has explicitly stated (do/don't for the observer). Only when explicit; never inferred from behavior. + - `INSTRUCTION: Call me Vee` + - `INSTRUCTION: Never push to main without review` + +### Rules + +1. **Stable.** If the value plausibly changes within six months absent a deliberate announcement, it does not belong on the card. Prefer leaving the card empty over filling it with volatile content. +2. **Subject is {observed}.** Every entry must be a fact about {observed}, not about another participant in the session. Never write facts about co-occurring peers into the card, no matter how frequently they appear in the messages. +3. **Evidence-grounded.** Only write what {observed} has explicitly stated, or what another participant has explicitly stated about {observed} with {observed}'s assent. No "general knowledge" inferences (`"co-founder"` does not imply an age; mentioning a colleague does not imply a family relationship). +4. **Type-agnostic.** {observed} may not be human. Do not require name/age/location/family/occupation fields. +5. **No behavioral content.** TRAITs, behavioral tendencies, patterns, and inferred preferences belong in observations, not on the peer card. Do not write `TRAIT:` entries or behavioral `PREFERENCE:` entries — they will be rejected. +6. **No evidence bundles.** Each entry is one concise fact. No `e.g.` clauses, no parenthetical example lists, no semicolon-separated value dumps. + +### Migrating an existing peer card + +The CURRENT PEER CARD shown in the user message may contain entries from an older format that do not start with an allowed prefix (e.g. `Name: Alice`, `Lives in NYC`, `TRAIT: Analytical`, `PREFERENCE: Detailed explanations`). When you call `update_peer_card`, you are responsible for re-emitting the entries you want to keep — entries you omit are dropped, and entries without an allowed prefix are silently rejected. + +For each legacy entry: + +- If it is still a valid identity marker, re-emit it under the correct prefix and keep the original content where reasonable. Examples: + - `Name: Alice` → `IDENTITY: Name: Alice` + - `Lives in NYC` → `ATTRIBUTE: Location: NYC` + - `Works at Google` → `ATTRIBUTE: Employer: Google` + - `INSTRUCTION: Call me Vee` → keep as is (already correctly prefixed) +- Drop entries that violate the rules above: behavioral `TRAIT:` lines, inferred behavioral `PREFERENCE:` lines, one-off events, transient state, evidence bundles. Do not re-prefix them — they are not identity markers. + +When in doubt about a specific legacy entry, prefer migrating it (so valid info isn't lost) over dropping it. Splitting one dense legacy entry into multiple correctly-prefixed entries is fine and encouraged (e.g. a semicolon-separated `Tech Stack:` dump can become several `ATTRIBUTE:` lines, one per durable tool/platform). + +Call `update_peer_card` with the complete deduplicated list when there is a durable identity update to record, or when the existing card needs migration. Entries that do not start with one of the four allowed prefixes will be rejected. Keep concise (max 40 entries).""" return f"""You are a deductive reasoning agent analyzing observations about {observed}. @@ -578,20 +618,19 @@ class InductionSpecialist(BaseSpecialist): 1. Explores observations to understand what's there 2. Identifies patterns and generalizations across multiple observations 3. Creates new inductive observations with source linkage - 4. Updates peer card with high-confidence traits and tendencies + + Does not write to the peer card — the peer card stores stable identity markers, + which is deduction's responsibility. Inductive patterns and tendencies stay as + observations. """ name: str = "induction" - peer_card_update_instruction: str = "Only add highly stable profile traits/preferences; do not copy transient conclusions." + # Induction never writes to the peer card; behavioral patterns are observations. + can_update_peer_card: bool = False def get_tools(self, *, peer_card_enabled: bool = True) -> list[dict[str, Any]]: - if peer_card_enabled: - return INDUCTION_SPECIALIST_TOOLS - return [ - t - for t in INDUCTION_SPECIALIST_TOOLS - if t["name"] not in PEER_CARD_TOOL_NAMES - ] + _ = peer_card_enabled + return INDUCTION_SPECIALIST_TOOLS def get_model_config(self) -> ConfiguredModelSettings: return _require_specialist_model_config( @@ -608,21 +647,7 @@ class InductionSpecialist(BaseSpecialist): def build_system_prompt( self, observed: str, *, peer_card_enabled: bool = True ) -> str: - peer_card_section = "" - if peer_card_enabled: - peer_card_section = """ - -## PEER CARD (REQUIRED) - -After identifying patterns, only update the peer card for durable profile-level traits/preferences: -- `TRAIT: Analytical thinker` -- `TRAIT: Tends to reschedule when stressed` -- `PREFERENCE: Prefers detailed explanations` - -Do NOT add temporary patterns, episode-specific conclusions, or reasoning summaries. -Call `update_peer_card` with the complete deduplicated list only when a durable profile update is warranted. -Keep it concise (max 40 entries).""" - + _ = peer_card_enabled return f"""You are an inductive reasoning agent identifying patterns about {observed}. ## YOUR JOB @@ -658,7 +683,6 @@ Create inductive observations when you see patterns: ### Temporal Patterns - "Career goals have remained consistent" - "Living situation changes frequently" -{peer_card_section} ## CREATING OBSERVATIONS @@ -690,11 +714,13 @@ Use `create_observations_inductive`. hints: list[str] | None, peer_card: list[str] | None = None, ) -> str: - peer_card_context = self._build_peer_card_context(peer_card) + # Induction does not consume peer card context — it produces inductive + # observations, not identity-marker updates. + _ = peer_card if hints: hints_str = "\n".join(f"- {q}" for q in hints[:5]) - return f"""{peer_card_context}Explore and find patterns. These areas may be worth investigating: + return f"""Explore and find patterns. These areas may be worth investigating: {hints_str} @@ -702,7 +728,7 @@ But follow the evidence - if you find patterns elsewhere, pursue those. Start with `get_recent_observations`.""" - return f"""{peer_card_context}Explore the observation space and identify patterns. + return """Explore the observation space and identify patterns. Remember: patterns need 2+ sources. Look for tendencies, preferences, and behavioral regularities. diff --git a/src/llm/backends/anthropic.py b/src/llm/backends/anthropic.py index cdf775be..17138583 100644 --- a/src/llm/backends/anthropic.py +++ b/src/llm/backends/anthropic.py @@ -34,11 +34,7 @@ class AnthropicBackend: max_output_tokens: int | None = None, extra_params: dict[str, Any] | None = None, ) -> CompletionResult: - del max_output_tokens - if thinking_effort is not None: - raise ValueError( - "Anthropic backend does not support thinking_effort; use thinking_budget_tokens instead" - ) + del max_output_tokens, thinking_effort request_messages, system_messages = self._extract_system(messages) params: dict[str, Any] = { @@ -123,11 +119,7 @@ class AnthropicBackend: extra_params: dict[str, Any] | None = None, ) -> AsyncIterator[StreamChunk]: is_json_mode = self._json_mode(extra_params) - del max_output_tokens - if thinking_effort is not None: - raise ValueError( - "Anthropic backend does not support thinking_effort; use thinking_budget_tokens instead" - ) + del max_output_tokens, thinking_effort request_messages, system_messages = self._extract_system(messages) params: dict[str, Any] = { diff --git a/src/llm/backends/openai.py b/src/llm/backends/openai.py index 1e01e78a..b2d82d91 100644 --- a/src/llm/backends/openai.py +++ b/src/llm/backends/openai.py @@ -128,11 +128,6 @@ class OpenAIBackend: max_output_tokens: int | None = None, extra_params: dict[str, Any] | None = None, ) -> CompletionResult: - if thinking_budget_tokens is not None: - raise ValidationException( - "OpenAI backend does not support thinking_budget_tokens; use thinking_effort instead" - ) - params = self._build_params( model=model, messages=messages, @@ -142,6 +137,7 @@ class OpenAIBackend: tools=tools, tool_choice=tool_choice, thinking_effort=thinking_effort, + thinking_budget_tokens=thinking_budget_tokens, extra_params=extra_params, ) @@ -221,11 +217,6 @@ class OpenAIBackend: max_output_tokens: int | None = None, extra_params: dict[str, Any] | None = None, ) -> AsyncIterator[StreamChunk]: - if thinking_budget_tokens is not None: - raise ValidationException( - "OpenAI backend does not support thinking_budget_tokens; use thinking_effort instead" - ) - params = self._build_params( model=model, messages=messages, @@ -235,6 +226,7 @@ class OpenAIBackend: tools=tools, tool_choice=tool_choice, thinking_effort=thinking_effort, + thinking_budget_tokens=thinking_budget_tokens, extra_params=extra_params, ) params["stream"] = True @@ -284,6 +276,7 @@ class OpenAIBackend: tools: list[dict[str, Any]] | None, tool_choice: str | dict[str, Any] | None, thinking_effort: str | None, + thinking_budget_tokens: int | None, extra_params: dict[str, Any] | None, ) -> dict[str, Any]: params: dict[str, Any] = { @@ -304,6 +297,16 @@ class OpenAIBackend: if thinking_effort: params["reasoning_effort"] = thinking_effort + # Token-budget style thinking is not part of the native OpenAI API, but + # OpenAI-compatible proxies (OpenRouter, etc.) accept a `reasoning` object + # on the request body. Pass through via extra_body so it reaches those + # backends; operators on providers that need a different shape (vLLM, + # Fireworks, ...) can override via ModelConfig.provider_params. + if thinking_budget_tokens is not None and thinking_budget_tokens > 0: + params.setdefault("extra_body", {}).setdefault("reasoning", {})[ + "max_tokens" + ] = thinking_budget_tokens + if stop: params["stop"] = stop if tools: diff --git a/src/utils/agent_tools.py b/src/utils/agent_tools.py index 8cdfd6cb..de5b2b09 100644 --- a/src/utils/agent_tools.py +++ b/src/utils/agent_tools.py @@ -33,6 +33,36 @@ logger = logging.getLogger(__name__) # Hard cap to prevent unbounded peer card growth from repeated agent updates. MAX_PEER_CARD_FACTS = 40 +# Identity-marker prefixes allowed on the peer card. Anything else is rejected +# structurally — see `_validate_peer_card_entry`. +PEER_CARD_ALLOWED_PREFIXES: tuple[str, ...] = ( + "IDENTITY:", + "ATTRIBUTE:", + "RELATIONSHIP:", + "INSTRUCTION:", +) + +# Per-entry character cap to block evidence-bundle dumps and runaway lines. +MAX_PEER_CARD_ENTRY_LENGTH = 200 + + +def _validate_peer_card_entry(line: str) -> bool: + """Structural validation for a single peer card entry. + + Returns True when the line starts with one of the allowed prefixes followed + by a space, has a non-empty body after the prefix, and fits within the per- + entry length cap. Subject-substance correctness (is this actually about the + observed peer?) is left to the prompt — this is form-only. + """ + if not line or len(line) > MAX_PEER_CARD_ENTRY_LENGTH: + return False + for prefix in PEER_CARD_ALLOWED_PREFIXES: + prefix_with_space = f"{prefix} " + if line.startswith(prefix_with_space): + body = line[len(prefix_with_space) :].strip() + return bool(body) + return False + def _normalized_observation_input( obs: schemas.ObservationInput, @@ -460,9 +490,16 @@ TOOLS: dict[str, dict[str, Any]] = { "update_peer_card": { "name": "update_peer_card", "description": ( - "Update the peer card with durable profile facts about the observed peer. " - + "Only include stable biographical facts, standing instructions, and long-lived preferences/traits. " - + "Do not include one-off conclusions, temporary events, or duplicate entries." + "Update the peer card with stable identity markers about the observed peer. " + "An identity marker distinguishes the peer from others of its kind and persists across interactions. " + "The peer may be any entity with identity that changes over time (human, agent, codebase, team, organization) — do not assume the peer is human. " + "Each entry must start with one of four prefixes: `IDENTITY:` (canonical name, kind, aliases, IDs), " + "`ATTRIBUTE:` (stable durable property, including explicitly stated standing preferences), " + "`RELATIONSHIP:` (durable link to another entity), or " + "`INSTRUCTION:` (standing rule of engagement the peer has explicitly stated). " + "Do not write `TRAIT:` or behavioral `PREFERENCE:` entries, one-off observations, transient state, " + "inferred facts not directly supported by evidence, evidence bundles / `e.g.` clauses, or entries about co-occurring peers. " + "Entries without an allowed prefix or that exceed the per-entry length cap are rejected." ), "input_schema": { "type": "object", @@ -471,7 +508,9 @@ TOOLS: dict[str, dict[str, Any]] = { "type": "array", "description": ( "Complete deduplicated peer card list (max 40 entries). " - + "Each entry should be a concise standalone profile fact." + "Each entry must start with one of the allowed prefixes " + "(`IDENTITY: `, `ATTRIBUTE: `, `RELATIONSHIP: `, `INSTRUCTION: `) " + "followed by one concise identity marker. Entries without an allowed prefix are rejected." ), "items": {"type": "string"}, }, @@ -796,7 +835,7 @@ DEDUCTION_SPECIALIST_TOOLS: list[dict[str, Any]] = [ # Tools for the induction specialist (dreamer phase 2) # Creates inductive observations from explicit and deductive observations # Includes message access for context and self-directed exploration -# Note: get_peer_card is not included - peer card is injected into the prompt directly +# Induction does not write to the peer card — that is deduction's responsibility. INDUCTION_SPECIALIST_TOOLS: list[dict[str, Any]] = [ # Discovery tools TOOLS["get_recent_observations"], @@ -804,7 +843,6 @@ INDUCTION_SPECIALIST_TOOLS: list[dict[str, Any]] = [ TOOLS["search_messages"], # Action tools TOOLS["create_observations_inductive"], - TOOLS["update_peer_card"], ] @@ -1425,9 +1463,16 @@ async def _handle_update_peer_card( ) return "Peer card content was empty, no update performed." - # Normalize and deduplicate to keep peer cards bounded and stable. + # Normalize, validate structure, and deduplicate to keep peer cards bounded + # and on-spec. normalized_peer_card: list[str] = [] seen: set[str] = set() + rejected_count = 0 + # Keep a small sample of rejected entries to surface back to the model so it + # can self-correct on a retry. Capped to avoid bloating the tool response. + rejected_samples: list[str] = [] + _REJECTED_SAMPLE_CAP = 3 + _REJECTED_SAMPLE_LINE_LIMIT = 120 items = ( cast(list[str], raw_peer_card_content) if isinstance(raw_peer_card_content, list) @@ -1438,6 +1483,16 @@ async def _handle_update_peer_card( if not line: continue + if not _validate_peer_card_entry(line): + rejected_count += 1 + if len(rejected_samples) < _REJECTED_SAMPLE_CAP: + rejected_samples.append(line[:_REJECTED_SAMPLE_LINE_LIMIT]) + logger.info( + "Rejecting peer card entry (no allowed prefix, empty body, or over length cap): %r", + line[:80], + ) + continue + # Case-insensitive dedupe with whitespace normalization. normalized_key = " ".join(line.lower().split()) if normalized_key in seen: @@ -1445,12 +1500,44 @@ async def _handle_update_peer_card( seen.add(normalized_key) normalized_peer_card.append(line) - # Don't clear the peer card if all content normalized to empty. + if rejected_count: + logger.info( + "Peer card update for %s/%s/%s rejected %d structurally invalid entries", + ctx.workspace_name, + ctx.observer, + ctx.observed, + rejected_count, + ) + + def _format_rejection_feedback(scope: str) -> str: + """Build a self-correction hint for the model. `scope` is grammar glue: + either "all" (every entry rejected) or e.g. "3 of 12" (partial).""" + samples_block = "" + if rejected_samples: + sample_lines = "\n".join(f" - {s!r}" for s in rejected_samples) + extra = ( + f" (+{rejected_count - len(rejected_samples)} more)" + if rejected_count > len(rejected_samples) + else "" + ) + samples_block = f" Examples of rejected entries{extra}:\n{sample_lines}" + return ( + f"Rejected {scope} entries for failing structural validation. " + "Each entry must start with one of `IDENTITY: `, `ATTRIBUTE: `, " + "`RELATIONSHIP: `, or `INSTRUCTION: ` and stay under the per-entry " + f"length cap.{samples_block}" + ) + + # Don't clear the peer card if all content normalized to empty or every + # entry was structurally invalid. if not normalized_peer_card: logger.warning( - "Peer card update normalized to empty for %s, keeping existing card", + "Peer card update normalized to empty for %s (rejected=%d), keeping existing card", ctx.workspace_name, + rejected_count, ) + if rejected_count: + return _format_rejection_feedback(f"all {rejected_count}") return "Peer card content was empty after normalization, no update performed." if len(normalized_peer_card) > MAX_PEER_CARD_FACTS: @@ -1494,9 +1581,24 @@ async def _handle_update_peer_card( # can set its `peer_card_updated` flag without name-counting. from src.utils.types import ToolResult + success_content = ( + f"Updated peer card for {ctx.observed} by {ctx.observer} " + f"with {len(normalized_peer_card)} entries." + ) + if rejected_count: + # Partial reject: surface the rejection so the model can re-emit the + # dropped entries (with correct prefixes) on a retry instead of + # silently losing them. + accepted = len(normalized_peer_card) + total = accepted + rejected_count + success_content = f"{success_content} {_format_rejection_feedback(f'{rejected_count} of {total}')}" return ToolResult( - content=f"Updated peer card for {ctx.observed} by {ctx.observer}", - metadata={"peer_card_updated": True, "facts_count": len(normalized_peer_card)}, + content=success_content, + metadata={ + "peer_card_updated": True, + "facts_count": len(normalized_peer_card), + "rejected_count": rejected_count, + }, ) diff --git a/tests/dreamer/test_model_config_usage.py b/tests/dreamer/test_model_config_usage.py index 91d1d141..90892632 100644 --- a/tests/dreamer/test_model_config_usage.py +++ b/tests/dreamer/test_model_config_usage.py @@ -3,10 +3,65 @@ from unittest.mock import AsyncMock, patch import pytest from src.config import settings -from src.dreamer.specialists import DeductionSpecialist +from src.dreamer.specialists import DeductionSpecialist, InductionSpecialist from src.llm import HonchoLLMCallResponse +def test_deduction_prompt_uses_identity_markers_framing() -> None: + """Deduction prompt must frame the peer card as an identity store with the + entity-agnostic prefix taxonomy, not as a human bio sheet.""" + prompt = DeductionSpecialist().build_system_prompt("alice", peer_card_enabled=True) + + assert "identity store" in prompt + assert "stable identity markers" in prompt + for prefix in ("IDENTITY:", "ATTRIBUTE:", "RELATIONSHIP:", "INSTRUCTION:"): + assert prefix in prompt + # Cross-entity examples confirm the prompt is not biased toward humans. + assert "codebase" in prompt + assert "team" in prompt + # Behavioral content must be explicitly excluded. + assert "TRAIT:" in prompt + # The old human-shaped REQUIRED enumeration must be gone. + assert "Family members and relationships" not in prompt + assert "Core preferences and traits" not in prompt + + +def test_deduction_prompt_omits_peer_card_when_disabled() -> None: + prompt = DeductionSpecialist().build_system_prompt("alice", peer_card_enabled=False) + assert "PEER CARD" not in prompt + assert "IDENTITY:" not in prompt + + +def test_induction_prompt_has_no_peer_card_section() -> None: + """Induction no longer writes to the peer card; its prompt must not reference it.""" + prompt = InductionSpecialist().build_system_prompt("alice", peer_card_enabled=True) + assert "PEER CARD" not in prompt + assert "update_peer_card" not in prompt + + +def test_induction_specialist_cannot_update_peer_card() -> None: + """Induction must have can_update_peer_card=False and no update_peer_card tool.""" + specialist = InductionSpecialist() + assert specialist.can_update_peer_card is False + + tool_names = {t["name"] for t in specialist.get_tools()} + assert "update_peer_card" not in tool_names + # Sanity: induction still has the discovery and create tools it actually needs. + assert "create_observations_inductive" in tool_names + assert "search_memory" in tool_names + + +def test_deduction_specialist_can_update_peer_card() -> None: + specialist = DeductionSpecialist() + assert specialist.can_update_peer_card is True + + tool_names = {t["name"] for t in specialist.get_tools(peer_card_enabled=True)} + assert "update_peer_card" in tool_names + + disabled_names = {t["name"] for t in specialist.get_tools(peer_card_enabled=False)} + assert "update_peer_card" not in disabled_names + + @pytest.mark.asyncio async def test_deduction_specialist_uses_nested_model_config( monkeypatch: pytest.MonkeyPatch, diff --git a/tests/llm/test_backends/test_anthropic.py b/tests/llm/test_backends/test_anthropic.py index c8f2bbdf..52de0fa2 100644 --- a/tests/llm/test_backends/test_anthropic.py +++ b/tests/llm/test_backends/test_anthropic.py @@ -123,13 +123,32 @@ async def test_anthropic_backend_skips_assistant_prefill_for_claude_4_models() - @pytest.mark.asyncio -async def test_anthropic_backend_rejects_thinking_effort() -> None: - backend = AnthropicBackend(Mock()) - - with pytest.raises(ValueError, match="does not support thinking_effort"): - await backend.complete( - model="claude-haiku-4-5", - messages=[{"role": "user", "content": "Hello"}], - max_tokens=100, - thinking_effort="high", +async def test_anthropic_backend_ignores_thinking_effort() -> None: + client = Mock() + client.messages.create = AsyncMock( + return_value=SimpleNamespace( + content=[TextBlock(type="text", text="ok")], + usage=SimpleNamespace( + input_tokens=10, + output_tokens=5, + cache_creation_input_tokens=0, + cache_read_input_tokens=0, + ), + stop_reason="end_turn", ) + ) + + backend = AnthropicBackend(client) + await backend.complete( + model="claude-haiku-4-5", + messages=[{"role": "user", "content": "Hello"}], + max_tokens=100, + thinking_effort="high", + ) + + await_args = client.messages.create.await_args + if await_args is None: + raise AssertionError("Expected Anthropic client call") + call = await_args.kwargs + assert "thinking" not in call + assert "reasoning_effort" not in call diff --git a/tests/llm/test_backends/test_openai.py b/tests/llm/test_backends/test_openai.py index 81838202..695b12cd 100644 --- a/tests/llm/test_backends/test_openai.py +++ b/tests/llm/test_backends/test_openai.py @@ -3,7 +3,6 @@ from unittest.mock import AsyncMock, Mock import pytest -from src.exceptions import ValidationException from src.llm.backends.openai import OpenAIBackend @@ -153,18 +152,79 @@ async def test_openai_backend_does_not_treat_proxy_models_with_gpt5_substring_as @pytest.mark.asyncio -async def test_openai_backend_rejects_thinking_budget_tokens() -> None: - backend = OpenAIBackend(Mock()) - - with pytest.raises( - ValidationException, match="does not support thinking_budget_tokens" - ): - await backend.complete( - model="gpt-5-mini", - messages=[{"role": "user", "content": "Hello"}], - max_tokens=100, - thinking_budget_tokens=256, +async def test_openai_backend_passes_thinking_budget_via_extra_body() -> None: + client = Mock() + client.chat.completions.create = AsyncMock( + return_value=SimpleNamespace( + choices=[ + SimpleNamespace( + finish_reason="stop", + message=SimpleNamespace( + content="ok", + tool_calls=[], + reasoning_details=[], + ), + ) + ], + usage=SimpleNamespace( + prompt_tokens=10, + completion_tokens=5, + prompt_tokens_details=None, + ), ) + ) + + backend = OpenAIBackend(client) + await backend.complete( + model="x-ai/grok-4.1-fast", + messages=[{"role": "user", "content": "Hello"}], + max_tokens=100, + thinking_budget_tokens=256, + ) + + await_args = client.chat.completions.create.await_args + if await_args is None: + raise AssertionError("Expected OpenAI create call") + call = await_args.kwargs + assert call["extra_body"] == {"reasoning": {"max_tokens": 256}} + + +@pytest.mark.asyncio +async def test_openai_backend_skips_extra_body_when_thinking_budget_zero() -> None: + client = Mock() + client.chat.completions.create = AsyncMock( + return_value=SimpleNamespace( + choices=[ + SimpleNamespace( + finish_reason="stop", + message=SimpleNamespace( + content="ok", + tool_calls=[], + reasoning_details=[], + ), + ) + ], + usage=SimpleNamespace( + prompt_tokens=10, + completion_tokens=5, + prompt_tokens_details=None, + ), + ) + ) + + backend = OpenAIBackend(client) + await backend.complete( + model="x-ai/grok-4.1-fast", + messages=[{"role": "user", "content": "Hello"}], + max_tokens=100, + thinking_budget_tokens=0, + ) + + await_args = client.chat.completions.create.await_args + if await_args is None: + raise AssertionError("Expected OpenAI create call") + call = await_args.kwargs + assert "extra_body" not in call @pytest.mark.asyncio diff --git a/tests/utils/test_agent_tools.py b/tests/utils/test_agent_tools.py index b300417d..8ddab1bc 100644 --- a/tests/utils/test_agent_tools.py +++ b/tests/utils/test_agent_tools.py @@ -14,7 +14,9 @@ from sqlalchemy.ext.asyncio import AsyncSession from src import crud, models, schemas from src.config import settings from src.utils.agent_tools import ( + MAX_PEER_CARD_ENTRY_LENGTH, MAX_PEER_CARD_FACTS, + PEER_CARD_ALLOWED_PREFIXES, ObservationsCreatedResult, ToolContext, _handle_create_observations, # pyright: ignore[reportPrivateUsage] @@ -32,6 +34,7 @@ from src.utils.agent_tools import ( _handle_search_messages, # pyright: ignore[reportPrivateUsage] _handle_search_messages_temporal, # pyright: ignore[reportPrivateUsage] _handle_update_peer_card, # pyright: ignore[reportPrivateUsage] + _validate_peer_card_entry, # pyright: ignore[reportPrivateUsage] create_observations, create_tool_executor, extract_preferences, @@ -415,7 +418,7 @@ class TestCreateObservations: result = await create_observations( observations=[ - schemas.ObservationInput(content=" ", level="explicit"), + schemas.ObservationInput(content=" ", level="explicit"), schemas.ObservationInput( content=" trimmed observation ", level="explicit" ), @@ -1023,7 +1026,14 @@ class TestUpdatePeerCard: ctx = make_tool_context() result = await _handle_update_peer_card( - ctx, {"content": ["Name: John", "Location: NYC", "Occupation: Engineer"]} + ctx, + { + "content": [ + "IDENTITY: Name: John", + "ATTRIBUTE: Location: NYC", + "ATTRIBUTE: Occupation: Engineer", + ] + }, ) assert "Updated peer card" in result @@ -1038,7 +1048,7 @@ class TestUpdatePeerCard: observed=peer2.name, ) assert peer_card is not None - assert "Name: John" in peer_card + assert "IDENTITY: Name: John" in peer_card async def test_deduplicates_and_caps_peer_card( self, @@ -1050,8 +1060,15 @@ class TestUpdatePeerCard: workspace, peer1, peer2, _, _, _ = tool_test_data ctx = make_tool_context() - oversized = ["Name: John", " Name: John ", "", " "] - oversized.extend([f"Fact {i}" for i in range(MAX_PEER_CARD_FACTS + 5)]) + oversized = [ + "IDENTITY: Name: John", + " IDENTITY: Name: John ", + "", + " ", + ] + oversized.extend( + [f"IDENTITY: Aliases: alias-{i}" for i in range(MAX_PEER_CARD_FACTS + 5)] + ) await _handle_update_peer_card(ctx, {"content": oversized}) @@ -1066,7 +1083,7 @@ class TestUpdatePeerCard: assert peer_card is not None assert len(peer_card) == MAX_PEER_CARD_FACTS assert all(line.strip() for line in peer_card) - assert peer_card.count("Name: John") == 1 + assert peer_card.count("IDENTITY: Name: John") == 1 async def test_none_content_preserves_existing_card( self, @@ -1080,7 +1097,8 @@ class TestUpdatePeerCard: # First, create a valid peer card await _handle_update_peer_card( - ctx, {"content": ["Name: Alice", "Location: NYC"]} + ctx, + {"content": ["IDENTITY: Name: Alice", "ATTRIBUTE: Location: NYC"]}, ) # Now attempt to update with None — should be a no-op @@ -1097,7 +1115,7 @@ class TestUpdatePeerCard: observed=peer2.name, ) assert peer_card is not None - assert "Name: Alice" in peer_card + assert "IDENTITY: Name: Alice" in peer_card async def test_empty_list_preserves_existing_card( self, @@ -1110,7 +1128,10 @@ class TestUpdatePeerCard: ctx = make_tool_context() # First, create a valid peer card - await _handle_update_peer_card(ctx, {"content": ["Name: Bob", "Age: 30"]}) + await _handle_update_peer_card( + ctx, + {"content": ["IDENTITY: Name: Bob", "ATTRIBUTE: Age: 30"]}, + ) # Now attempt to update with empty list — should be a no-op result = await _handle_update_peer_card(ctx, {"content": []}) @@ -1126,7 +1147,151 @@ class TestUpdatePeerCard: observed=peer2.name, ) assert peer_card is not None - assert "Name: Bob" in peer_card + assert "IDENTITY: Name: Bob" in peer_card + + async def test_rejects_entries_without_allowed_prefix( + self, + db_session: AsyncSession, + tool_test_data: Any, + make_tool_context: Callable[..., ToolContext], + ): + """Entries without an allowed prefix are dropped; valid entries pass through.""" + from src.utils.types import ToolResult + + workspace, peer1, peer2, _, _, _ = tool_test_data + ctx = make_tool_context() + + result = await _handle_update_peer_card( + ctx, + { + "content": [ + "IDENTITY: Name: Carol", + "Age: 39+", # rejected: no prefix + "Daughter: Keyan", # rejected: no prefix + "TRAIT: Methodical", # rejected: TRAIT not allowed + "PREFERENCE: Tea", # rejected: bare PREFERENCE not allowed + "ATTRIBUTE: Location: Germantown, TN", + ] + }, + ) + + # Partial-reject success path must surface the rejection in the tool + # response so the model can re-emit the dropped entries (with correct + # prefixes) on a retry instead of silently losing them. + assert isinstance(result, ToolResult) + content_lower = str(result).lower() + assert "updated peer card" in content_lower + assert "rejected 4 of 6" in content_lower + # At least one rejected sample should appear so the model knows what + # to fix. + assert "age: 39+" in content_lower or "trait: methodical" in content_lower + assert result.metadata is not None + assert result.metadata["peer_card_updated"] is True + assert result.metadata["facts_count"] == 2 + assert result.metadata["rejected_count"] == 4 + + await db_session.refresh(peer1) + peer_card = await crud.get_peer_card( + db_session, + workspace_name=workspace.name, + observer=peer1.name, + observed=peer2.name, + ) + assert peer_card is not None + assert peer_card == [ + "IDENTITY: Name: Carol", + "ATTRIBUTE: Location: Germantown, TN", + ] + + async def test_all_entries_rejected_preserves_existing_card( + self, + db_session: AsyncSession, + tool_test_data: Any, + make_tool_context: Callable[..., ToolContext], + ): + """When every entry fails validation, the existing card is preserved.""" + workspace, peer1, peer2, _, _, _ = tool_test_data + ctx = make_tool_context() + + await _handle_update_peer_card(ctx, {"content": ["IDENTITY: Name: Dana"]}) + + result = await _handle_update_peer_card( + ctx, + { + "content": [ + "TRAIT: Detail-oriented", + "PREFERENCE: Coffee", + "Random unprefixed line", + ] + }, + ) + assert "rejected" in str(result).lower() + + await db_session.refresh(peer1) + peer_card = await crud.get_peer_card( + db_session, + workspace_name=workspace.name, + observer=peer1.name, + observed=peer2.name, + ) + assert peer_card == ["IDENTITY: Name: Dana"] + + +class TestPeerCardEntryValidator: + """Unit tests for the pure structural validator.""" + + @pytest.mark.parametrize( + "entry", + [ + "IDENTITY: Name: Alice", + "ATTRIBUTE: Location: NYC", + "ATTRIBUTE: Prefers tea", + "RELATIONSHIP: Spouse: Bob", + "RELATIONSHIP: Maintainer: vineeth", + "INSTRUCTION: Call me Vee", + "INSTRUCTION: Never push to main without review", + ], + ) + def test_accepts_well_formed_entries(self, entry: str): + assert _validate_peer_card_entry(entry) is True + + @pytest.mark.parametrize( + "entry", + [ + "", + " ", + "Name: Alice", # missing prefix + "Age: 39+", # missing prefix + "Daughter: Keyan", # missing prefix + "TRAIT: Methodical", # disallowed kind + "PREFERENCE: Tea", # disallowed kind + "identity: name: alice", # wrong case + "IDENTITY:Name: Alice", # missing space after colon + "IDENTITY: ", # empty body + "IDENTITY: ", # whitespace-only body + ], + ) + def test_rejects_malformed_entries(self, entry: str): + assert _validate_peer_card_entry(entry) is False + + def test_rejects_over_length_cap(self): + long_value = "x" * (MAX_PEER_CARD_ENTRY_LENGTH + 1) + assert _validate_peer_card_entry(f"IDENTITY: Name: {long_value}") is False + + def test_accepts_at_length_cap(self): + # Build an entry exactly at the cap. + prefix = "IDENTITY: " + body = "x" * (MAX_PEER_CARD_ENTRY_LENGTH - len(prefix)) + assert _validate_peer_card_entry(prefix + body) is True + + def test_allowed_prefixes_constant_is_complete(self): + # Guard against silent drift between the prompt and the validator. + assert PEER_CARD_ALLOWED_PREFIXES == ( + "IDENTITY:", + "ATTRIBUTE:", + "RELATIONSHIP:", + "INSTRUCTION:", + ) @pytest.mark.asyncio From 0cf63c10daf66b76e0f813f1cabdeddae1400dbc Mon Sep 17 00:00:00 2001 From: adavyas <121313528+adavyas@users.noreply.github.com> Date: Thu, 21 May 2026 10:40:47 -0700 Subject: [PATCH 5/7] feat(api): restore reverse pagination (#685) * feat(api): restore v3 reverse pagination * docs: add reverse pagination docstrings * docs: document session reverse parameter * fix: add fallback column for ties * refactor: tighten reverse query typing * chore: pre-commit styling * chore(tests): Add additional validation tests and update changelogs --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> --- CHANGELOG.md | 1 + docs/v3/openapi.json | 76 ++++++++ sdks/python/CHANGELOG.md | 6 + sdks/python/src/honcho/aio.py | 35 +++- sdks/python/src/honcho/client.py | 36 +++- sdks/typescript/CHANGELOG.md | 6 + sdks/typescript/src/client.ts | 9 +- sdks/typescript/src/types/api.ts | 5 + src/crud/peer.py | 17 +- src/crud/session.py | 15 +- src/crud/workspace.py | 14 +- src/routers/peers.py | 11 +- src/routers/sessions.py | 10 +- src/routers/workspaces.py | 5 +- tests/routes/test_peers.py | 298 ++++++++++++++++++++++++++++++- tests/routes/test_sessions.py | 158 ++++++++++++++++ tests/routes/test_workspaces.py | 130 ++++++++++++++ 17 files changed, 815 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ab9302c6..901265ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - Stream-final retries pin to the `AttemptPlan` that succeeded rather than re-running provider selection through the outer `current_attempt` ContextVar (which could roll streaming back to primary after the tool loop had already switched to fallback) - OpenAI structured-output calls continue to use `chat.completions.parse()` with strict schema enforcement, while tool-calling paths use `chat.completions.create()` without `strict:True` for broader proxy compatibility (OpenRouter, vLLM, Ollama) - Gemini `cached_content` reuse keys now include `system_instruction` and `tool_config` so cache hits don't cross configurations that differ only in those fields +- `reverse` query parameter is now honored on the v3 workspace list (`POST /v3/workspaces/list`), peer list (`POST /v3/workspaces/{workspace_id}/peers/list`), workspace-scoped session list (`POST /v3/workspaces/{workspace_id}/sessions/list`), and peer-scoped session list (`POST /v3/workspaces/{workspace_id}/peers/{peer_id}/sessions`). Honcho SDKs at 2.1.0+ were already sending `reverse=true` for these routes but the server silently ignored it. Ties on `created_at` now fall back to the internal nanoid `id` so ordering remains stable across pages. ### Removed diff --git a/docs/v3/openapi.json b/docs/v3/openapi.json index a2e46b2c..161cafc5 100644 --- a/docs/v3/openapi.json +++ b/docs/v3/openapi.json @@ -99,6 +99,25 @@ }, "description": "Page number" }, + { + "name": "reverse", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "description": "Whether to reverse the order of results", + "default": false, + "title": "Reverse" + }, + "description": "Whether to reverse the order of results" + }, { "name": "size", "in": "query", @@ -493,6 +512,25 @@ "title": "Workspace Id" } }, + { + "name": "reverse", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "description": "Whether to reverse the order of results", + "default": false, + "title": "Reverse" + }, + "description": "Whether to reverse the order of results" + }, { "name": "page", "in": "query", @@ -719,6 +757,25 @@ "title": "Peer Id" } }, + { + "name": "reverse", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "description": "Whether to reverse the order of results", + "default": false, + "title": "Reverse" + }, + "description": "Whether to reverse the order of results" + }, { "name": "page", "in": "query", @@ -1358,6 +1415,25 @@ "title": "Workspace Id" } }, + { + "name": "reverse", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "description": "Whether to reverse the order of results", + "default": false, + "title": "Reverse" + }, + "description": "Whether to reverse the order of results" + }, { "name": "page", "in": "query", diff --git a/sdks/python/CHANGELOG.md b/sdks/python/CHANGELOG.md index 36576f05..a7056000 100644 --- a/sdks/python/CHANGELOG.md +++ b/sdks/python/CHANGELOG.md @@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). +## [Unreleased] + +### Added + +- `page`, `size`, and `reverse` pagination parameters on `Honcho.workspaces()` and `HonchoAio.workspaces()`, closing the gap from 2.1.0 which added these to `peers()`, `sessions()`, `messages()`, and `conclusions.list()` but not to `workspaces()`. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter. + ## [2.1.1] - 2026-04-01 ### Fixed diff --git a/sdks/python/src/honcho/aio.py b/sdks/python/src/honcho/aio.py index 04ca42fb..9ff27a12 100644 --- a/sdks/python/src/honcho/aio.py +++ b/sdks/python/src/honcho/aio.py @@ -92,24 +92,30 @@ class HonchoAio(AsyncMetadataConfigMixin): _honcho: "Honcho" def __init__(self, honcho: "Honcho") -> None: + """Create an async view backed by a sync Honcho client.""" self._honcho = honcho # AsyncMetadataConfigMixin implementation def _get_async_http_client(self): + """Return the async HTTP client used by metadata helpers.""" return self._honcho._async_http_client def _get_fetch_route(self) -> str: + """Return the workspace fetch route for metadata helpers.""" return routes.workspaces() def _get_update_route(self) -> str: + """Return the workspace update route for metadata helpers.""" return routes.workspace(self._honcho.workspace_id) def _get_fetch_body(self) -> dict[str, Any]: + """Return the request body used to fetch this workspace.""" return {"id": self._honcho.workspace_id} def _parse_response( self, data: dict[str, Any] ) -> tuple[dict[str, object], dict[str, object]]: + """Parse workspace metadata and configuration from an API response.""" workspace = WorkspaceResponse.model_validate(data) # Return configuration as dict for mixin compatibility return workspace.metadata or {}, workspace.configuration.model_dump( @@ -117,18 +123,22 @@ class HonchoAio(AsyncMetadataConfigMixin): ) def _set_metadata(self, metadata: dict[str, object]) -> None: + """Update cached workspace metadata on the parent client.""" self._honcho._metadata = metadata def _set_configuration(self, configuration: dict[str, object]) -> None: + """Update cached workspace configuration on the parent client.""" # Convert dict to typed configuration self._honcho._configuration = WorkspaceConfiguration.model_validate( configuration ) def _get_metadata(self) -> dict[str, object]: + """Return cached workspace metadata from the parent client.""" return self._honcho._metadata or {} def _get_configuration(self) -> dict[str, object]: + """Return cached workspace configuration from the parent client.""" if self._honcho._configuration is None: return {} return self._honcho._configuration.model_dump(exclude_none=True) @@ -216,6 +226,7 @@ class HonchoAio(AsyncMetadataConfigMixin): ) def transform(peer: PeerResponse) -> Peer: + """Convert a peer API response into a Peer SDK object.""" return Peer( peer.id, self._honcho, @@ -225,6 +236,7 @@ class HonchoAio(AsyncMetadataConfigMixin): ) async def fetch_next(next_page: int) -> AsyncPage[PeerResponse, Peer]: + """Fetch the next page while preserving filters and ordering.""" next_query: dict[str, Any] = {"page": next_page, "size": size} if reverse: next_query["reverse"] = "true" @@ -318,6 +330,7 @@ class HonchoAio(AsyncMetadataConfigMixin): ) def transform(session: SessionResponse) -> Session: + """Convert a session API response into a Session SDK object.""" return Session( session.id, self._honcho, @@ -328,6 +341,7 @@ class HonchoAio(AsyncMetadataConfigMixin): ) async def fetch_next(next_page: int) -> AsyncPage[SessionResponse, Session]: + """Fetch the next page while preserving filters and ordering.""" next_query: dict[str, Any] = {"page": next_page, "size": size} if reverse: next_query["reverse"] = "true" @@ -341,22 +355,37 @@ class HonchoAio(AsyncMetadataConfigMixin): return AsyncPage(data, SessionResponse, transform, fetch_next) async def workspaces( - self, filters: dict[str, object] | None = None + self, + filters: dict[str, object] | None = None, + *, + page: int = 1, + size: int = 50, + reverse: bool = False, ) -> AsyncPage[WorkspaceResponse, str]: """Get all workspace IDs asynchronously.""" + query: dict[str, Any] = {"page": page, "size": size} + if reverse: + query["reverse"] = "true" + data = await self._honcho._async_http_client.post( routes.workspaces_list(), body={"filters": filters} if filters else None, + query=query, ) def transform(workspace: WorkspaceResponse) -> str: + """Convert a workspace API response into its workspace ID.""" return workspace.id - async def fetch_next(page: int) -> AsyncPage[WorkspaceResponse, str]: + async def fetch_next(next_page: int) -> AsyncPage[WorkspaceResponse, str]: + """Fetch the next page while preserving filters and ordering.""" + next_query: dict[str, Any] = {"page": next_page, "size": size} + if reverse: + next_query["reverse"] = "true" next_data = await self._honcho._async_http_client.post( routes.workspaces_list(), body={"filters": filters} if filters else None, - query={"page": page}, + query=next_query, ) return AsyncPage(next_data, WorkspaceResponse, transform, fetch_next) diff --git a/sdks/python/src/honcho/client.py b/sdks/python/src/honcho/client.py index 1dbd3a8c..06a2887c 100644 --- a/sdks/python/src/honcho/client.py +++ b/sdks/python/src/honcho/client.py @@ -83,20 +83,25 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul # MetadataConfigMixin implementation def _get_http_client(self): + """Return the sync HTTP client used by metadata helpers.""" return self._http def _get_fetch_route(self) -> str: + """Return the workspace fetch route for metadata helpers.""" return routes.workspaces() def _get_update_route(self) -> str: + """Return the workspace update route for metadata helpers.""" return routes.workspace(self.workspace_id) def _get_fetch_body(self) -> dict[str, Any]: + """Return the request body used to fetch this workspace.""" return {"id": self.workspace_id} def _parse_response( self, data: dict[str, Any] ) -> tuple[dict[str, object], dict[str, object]]: + """Parse workspace metadata and configuration from an API response.""" workspace = WorkspaceResponse.model_validate(data) # Return configuration as dict for mixin compatibility return workspace.metadata or {}, workspace.configuration.model_dump( @@ -365,6 +370,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul ) def transform(peer: PeerResponse) -> Peer: + """Convert a peer API response into a Peer SDK object.""" return Peer( peer.id, self, @@ -374,6 +380,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul ) def fetch_next(next_page: int) -> SyncPage[PeerResponse, Peer]: + """Fetch the next page while preserving filters and ordering.""" next_query: dict[str, Any] = {"page": next_page, "size": size} if reverse: next_query["reverse"] = "true" @@ -483,6 +490,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul ) def transform(session: SessionResponse) -> Session: + """Convert a session API response into a Session SDK object.""" return Session( session.id, self, @@ -493,6 +501,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul ) def fetch_next(next_page: int) -> SyncPage[SessionResponse, Session]: + """Fetch the next page while preserving filters and ordering.""" next_query: dict[str, Any] = {"page": next_page, "size": size} if reverse: next_query["reverse"] = "true" @@ -506,7 +515,12 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul return SyncPage(data, SessionResponse, transform, fetch_next) def workspaces( - self, filters: dict[str, object] | None = None + self, + filters: dict[str, object] | None = None, + *, + page: int = 1, + size: int = 50, + reverse: bool = False, ) -> SyncPage[WorkspaceResponse, str]: """ Get all workspace IDs from the Honcho instance. @@ -514,22 +528,38 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul Makes an API call to retrieve all workspace IDs that the authenticated user has access to. + Args: + filters: Optional filter criteria. + page: Page number (1-indexed). Default: 1. + size: Number of items per page. Default: 50. + reverse: If True, reverses the default ordering. Default: False. + Returns: A paginated SyncPage of workspace ID strings """ + query: dict[str, Any] = {"page": page, "size": size} + if reverse: + query["reverse"] = "true" + data = self._http.post( routes.workspaces_list(), body={"filters": filters} if filters else None, + query=query, ) def transform(workspace: WorkspaceResponse) -> str: + """Convert a workspace API response into its workspace ID.""" return workspace.id - def fetch_next(page: int) -> SyncPage[WorkspaceResponse, str]: + def fetch_next(next_page: int) -> SyncPage[WorkspaceResponse, str]: + """Fetch the next page while preserving filters and ordering.""" + next_query: dict[str, Any] = {"page": next_page, "size": size} + if reverse: + next_query["reverse"] = "true" next_data = self._http.post( routes.workspaces_list(), body={"filters": filters} if filters else None, - query={"page": page}, + query=next_query, ) return SyncPage(next_data, WorkspaceResponse, transform, fetch_next) diff --git a/sdks/typescript/CHANGELOG.md b/sdks/typescript/CHANGELOG.md index 008d2f45..e5ab73ac 100644 --- a/sdks/typescript/CHANGELOG.md +++ b/sdks/typescript/CHANGELOG.md @@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). +## [Unreleased] + +### Fixed + +- `Honcho.workspaces()` now actually forwards the `reverse` option to the server. The 2.1.0 changelog listed `workspaces()` among the list methods that gained `reverse`, but `client.ts` was missing the field on the params type and request builder, so the option was silently dropped. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter. + ## [2.1.1] - 2026-04-01 ### Fixed diff --git a/sdks/typescript/src/client.ts b/sdks/typescript/src/client.ts index 4b510945..db8e237a 100644 --- a/sdks/typescript/src/client.ts +++ b/sdks/typescript/src/client.ts @@ -232,6 +232,7 @@ export class Honcho { filters?: Record page?: number size?: number + reverse?: boolean }): Promise> { return this._http.post>( `/${API_VERSION}/workspaces/list`, @@ -242,6 +243,7 @@ export class Honcho { query: { page: params?.page, size: params?.size, + reverse: params?.reverse ? 'true' : undefined, }, } ) @@ -707,7 +709,7 @@ export class Honcho { * user has access to. * * @param options - Either a legacy raw filter object or an options object with - * `filters`, `page`, and `size`. See + * `filters`, `page`, `size`, and `reverse`. See * [search filters documentation](https://honcho.dev/docs/v3/documentation/core-concepts/features/using-filters). * @returns Promise resolving to a Page of workspace ID strings. Returns an empty * page if no workspaces are accessible or none exist @@ -719,20 +721,24 @@ export class Honcho { filters?: Filters page?: number size?: number + reverse?: boolean } ): Promise> { const normalizedOptions = normalizeListOptions(options, [ 'filters', 'page', 'size', + 'reverse', ]) const validatedFilter = normalizedOptions.filters ? FilterSchema.parse(normalizedOptions.filters) : undefined + const reverse = normalizedOptions.reverse const workspacesPage = await this._listWorkspaces({ filters: validatedFilter, page: normalizedOptions.page, size: normalizedOptions.size, + reverse, }) const fetchNextPage = async ( @@ -743,6 +749,7 @@ export class Honcho { filters: validatedFilter, page, size, + reverse, }) } diff --git a/sdks/typescript/src/types/api.ts b/sdks/typescript/src/types/api.ts index 1622c99c..65c6d88c 100644 --- a/sdks/typescript/src/types/api.ts +++ b/sdks/typescript/src/types/api.ts @@ -35,6 +35,7 @@ export interface WorkspaceListParams { filters?: Record page?: number size?: number + reverse?: boolean } // ============================================================================= @@ -64,6 +65,7 @@ export interface PeerListParams { filters?: Record page?: number size?: number + reverse?: boolean } export interface PeerChatParams { @@ -141,6 +143,7 @@ export interface SessionListParams { filters?: Record page?: number size?: number + reverse?: boolean } export interface SessionCloneParams { @@ -226,6 +229,7 @@ export interface MessageListParams { filters?: Record page?: number size?: number + reverse?: boolean } export interface MessageSearchParams { @@ -262,6 +266,7 @@ export interface ConclusionListParams { filters?: Record page?: number size?: number + reverse?: boolean } export interface ConclusionQueryParams { diff --git a/src/crud/peer.py b/src/crud/peer.py index 4c144b9b..21792b0f 100644 --- a/src/crud/peer.py +++ b/src/crud/peer.py @@ -1,3 +1,5 @@ +"""CRUD helpers for peer records and peer-scoped session queries.""" + from logging import getLogger from typing import Any @@ -210,13 +212,17 @@ async def get_peer( async def get_peers( workspace_name: str, - filters: dict[str, str] | None = None, + filters: dict[str, Any] | None = None, + reverse: bool = False, ) -> Select[tuple[models.Peer]]: + """Build a filtered peer list query ordered by creation time.""" stmt = select(models.Peer).where(models.Peer.workspace_name == workspace_name) stmt = apply_filter(stmt, models.Peer, filters) - return stmt.order_by(models.Peer.created_at) + if reverse: + return stmt.order_by(models.Peer.created_at.desc(), models.Peer.id.desc()) + return stmt.order_by(models.Peer.created_at.asc(), models.Peer.id.asc()) async def update_peer( @@ -285,6 +291,7 @@ async def get_sessions_for_peer( workspace_name: str, peer_name: str, filters: dict[str, Any] | None = None, + reverse: bool = False, ) -> Select[tuple[models.Session]]: """ Get all sessions for a peer through the session_peers relationship. @@ -293,6 +300,7 @@ async def get_sessions_for_peer( workspace_name: Name of the workspace peer_name: Name of the peer filters: Filter sessions by metadata + reverse: Whether to reverse the default creation order Returns: SQLAlchemy Select statement @@ -310,6 +318,9 @@ async def get_sessions_for_peer( stmt = apply_filter(stmt, models.Session, filters) - stmt: Select[tuple[models.Session]] = stmt.order_by(models.Session.created_at) + if reverse: + stmt = stmt.order_by(models.Session.created_at.desc(), models.Session.id.desc()) + else: + stmt = stmt.order_by(models.Session.created_at.asc(), models.Session.id.asc()) return stmt diff --git a/src/crud/session.py b/src/crud/session.py index 9580c16d..712188bc 100644 --- a/src/crud/session.py +++ b/src/crud/session.py @@ -1,3 +1,5 @@ +"""CRUD helpers for sessions and session-related relationship data.""" + from dataclasses import dataclass from logging import getLogger from typing import Any @@ -114,9 +116,18 @@ def count_observers_in_config( async def get_sessions( workspace_name: str, filters: dict[str, Any] | None = None, + reverse: bool = False, ) -> Select[tuple[models.Session]]: """ Get all active sessions in a workspace. + + Args: + workspace_name: Name of the workspace + filters: Optional filters to apply to the query + reverse: If True, order by created_at descending; if False, ascending + + Returns: + Select statement for Session objects """ stmt = ( select(models.Session) @@ -126,7 +137,9 @@ async def get_sessions( stmt = apply_filter(stmt, models.Session, filters) - return stmt.order_by(models.Session.created_at) + if reverse: + return stmt.order_by(models.Session.created_at.desc(), models.Session.id.desc()) + return stmt.order_by(models.Session.created_at.asc(), models.Session.id.asc()) async def get_or_create_session( diff --git a/src/crud/workspace.py b/src/crud/workspace.py index 3df2bb46..52662c79 100644 --- a/src/crud/workspace.py +++ b/src/crud/workspace.py @@ -1,3 +1,5 @@ +"""CRUD helpers for workspace records and workspace deletion checks.""" + from dataclasses import dataclass from logging import getLogger from typing import Any @@ -154,17 +156,25 @@ async def get_or_create_workspace( async def get_all_workspaces( filters: dict[str, Any] | None = None, + reverse: bool = False, ) -> Select[tuple[models.Workspace]]: """ Get all workspaces. Args: - db: Database session filters: Filter the workspaces by a dictionary of metadata + reverse: Whether to reverse the default creation order """ stmt = select(models.Workspace) stmt = apply_filter(stmt, models.Workspace, filters) - stmt: Select[tuple[models.Workspace]] = stmt.order_by(models.Workspace.created_at) + if reverse: + stmt = stmt.order_by( + models.Workspace.created_at.desc(), models.Workspace.id.desc() + ) + else: + stmt = stmt.order_by( + models.Workspace.created_at.asc(), models.Workspace.id.asc() + ) return stmt diff --git a/src/routers/peers.py b/src/routers/peers.py index 99014455..efa19a78 100644 --- a/src/routers/peers.py +++ b/src/routers/peers.py @@ -1,3 +1,5 @@ +"""FastAPI routes for peer resources and peer-scoped operations.""" + import json import logging from collections.abc import AsyncIterator @@ -40,6 +42,7 @@ async def get_peers( options: schemas.PeerGet | None = Body( None, description="Filtering options for the peers list" ), + reverse: bool = Query(False, description="Whether to reverse the order of results"), db: AsyncSession = db, ): """Get all Peers for a Workspace, paginated with optional filters.""" @@ -51,7 +54,11 @@ async def get_peers( return await apaginate( db, - await crud.get_peers(workspace_name=workspace_id, filters=filter_param), + await crud.get_peers( + workspace_name=workspace_id, + filters=filter_param, + reverse=reverse, + ), ) @@ -126,6 +133,7 @@ async def get_sessions_for_peer( options: schemas.SessionGet | None = Body( None, description="Filtering options for the sessions list" ), + reverse: bool = Query(False, description="Whether to reverse the order of results"), db: AsyncSession = db, ): """Get all Sessions for a Peer, paginated with optional filters.""" @@ -142,6 +150,7 @@ async def get_sessions_for_peer( workspace_name=workspace_id, peer_name=peer_id, filters=filter_param, + reverse=reverse, ), ) diff --git a/src/routers/sessions.py b/src/routers/sessions.py index 98f669df..98a8714a 100644 --- a/src/routers/sessions.py +++ b/src/routers/sessions.py @@ -1,3 +1,5 @@ +"""FastAPI routes for session resources and session-scoped operations.""" + import logging from contextlib import suppress from time import perf_counter @@ -248,6 +250,7 @@ async def get_sessions( options: schemas.SessionGet | None = Body( None, description="Filtering and pagination options for the sessions list" ), + reverse: bool = Query(False, description="Whether to reverse the order of results"), db: AsyncSession = db, ): """Get all Sessions for a Workspace, paginated with optional filters.""" @@ -259,7 +262,12 @@ async def get_sessions( filter_param = None return await apaginate( - db, await crud.get_sessions(workspace_name=workspace_id, filters=filter_param) + db, + await crud.get_sessions( + workspace_name=workspace_id, + filters=filter_param, + reverse=reverse, + ), ) diff --git a/src/routers/workspaces.py b/src/routers/workspaces.py index 6671afbc..a2298480 100644 --- a/src/routers/workspaces.py +++ b/src/routers/workspaces.py @@ -1,3 +1,5 @@ +"""FastAPI routes for workspace resources and workspace-scoped operations.""" + import logging from fastapi import APIRouter, Body, Depends, HTTPException, Path, Query, Response @@ -64,6 +66,7 @@ async def get_all_workspaces( options: schemas.WorkspaceGet | None = Body( None, description="Filtering and pagination options for the workspaces list" ), + reverse: bool = Query(False, description="Whether to reverse the order of results"), db: AsyncSession = db, ): """Get all Workspaces, paginated with optional filters.""" @@ -75,7 +78,7 @@ async def get_all_workspaces( return await apaginate( db, - await crud.get_all_workspaces(filters=filter_param), + await crud.get_all_workspaces(filters=filter_param, reverse=reverse), ) diff --git a/tests/routes/test_peers.py b/tests/routes/test_peers.py index 9626ea10..dd00eff9 100644 --- a/tests/routes/test_peers.py +++ b/tests/routes/test_peers.py @@ -1,3 +1,4 @@ +import datetime from typing import Any import pytest @@ -5,7 +6,7 @@ from fastapi.testclient import TestClient from nanoid import generate as generate_nanoid from sqlalchemy.ext.asyncio import AsyncSession -from src import crud +from src import crud, models from src.models import Peer, Workspace @@ -169,6 +170,148 @@ def test_get_peers_with_null_filter( assert isinstance(data["items"], list) +def test_get_peers_with_reverse( + client: TestClient, sample_data: tuple[Workspace, Peer] +): + """Test peer listing with reverse creation-time ordering.""" + test_workspace, _ = sample_data + reverse_group = f"reverse-peers-{generate_nanoid()}" + first_name = f"reverse-peer-a-{generate_nanoid()}" + second_name = f"reverse-peer-b-{generate_nanoid()}" + + first_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers", + json={"name": first_name, "metadata": {"reverse_group": reverse_group}}, + ) + assert first_response.status_code in [200, 201] + + second_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers", + json={"name": second_name, "metadata": {"reverse_group": reverse_group}}, + ) + assert second_response.status_code in [200, 201] + + normal_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/list", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert reverse_response.status_code == 200 + + assert [item["id"] for item in normal_response.json()["items"]] == [ + first_name, + second_name, + ] + assert [item["id"] for item in reverse_response.json()["items"]] == [ + second_name, + first_name, + ] + + +@pytest.mark.asyncio +async def test_get_peers_reverse_uses_id_tiebreaker( + client: TestClient, + db_session: AsyncSession, + sample_data: tuple[Workspace, Peer], +): + """Peers with identical created_at fall back to ordering by id (nanoid PK).""" + test_workspace, _ = sample_data + reverse_group = f"tiebreaker-peers-{generate_nanoid()}" + shared_created_at = datetime.datetime( + 2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc + ) + + low_id = "A" * 21 + high_id = "z" * 21 + low_name = f"tie-low-peer-{generate_nanoid()}" + high_name = f"tie-high-peer-{generate_nanoid()}" + + db_session.add( + models.Peer( + id=low_id, + name=low_name, + workspace_name=test_workspace.name, + created_at=shared_created_at, + h_metadata={"reverse_group": reverse_group}, + ) + ) + db_session.add( + models.Peer( + id=high_id, + name=high_name, + workspace_name=test_workspace.name, + created_at=shared_created_at, + h_metadata={"reverse_group": reverse_group}, + ) + ) + await db_session.commit() + + normal_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/list", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert reverse_response.status_code == 200 + + # When created_at ties, ordering falls back to the nanoid id: low_id < high_id + # lexicographically, so low sorts first ascending and last descending. + assert [item["id"] for item in normal_response.json()["items"]] == [ + low_name, + high_name, + ] + assert [item["id"] for item in reverse_response.json()["items"]] == [ + high_name, + low_name, + ] + + +def test_get_peers_reverse_with_pagination( + client: TestClient, sample_data: tuple[Workspace, Peer] +): + """Paged reverse listing returns newest-first across consecutive pages.""" + test_workspace, _ = sample_data + reverse_group = f"paged-reverse-peers-{generate_nanoid()}" + peer_names = [f"paged-reverse-peer-{i}-{generate_nanoid()}" for i in range(3)] + + for peer_name in peer_names: + response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers", + json={"name": peer_name, "metadata": {"reverse_group": reverse_group}}, + ) + assert response.status_code in [200, 201] + + page_one = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true&page=1&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_one.status_code == 200 + page_two = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true&page=2&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_two.status_code == 200 + page_three = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true&page=3&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_three.status_code == 200 + + assert page_one.json()["total"] == 3 + assert [item["id"] for item in page_one.json()["items"]] == [peer_names[2]] + assert [item["id"] for item in page_two.json()["items"]] == [peer_names[1]] + assert [item["id"] for item in page_three.json()["items"]] == [peer_names[0]] + + def test_update_peer(client: TestClient, sample_data: tuple[Workspace, Peer]): test_workspace, test_peer = sample_data response = client.put( @@ -308,6 +451,159 @@ def test_get_sessions_for_peer_with_empty_filter( assert isinstance(data["items"], list) +def test_get_sessions_for_peer_with_reverse( + client: TestClient, sample_data: tuple[Workspace, Peer] +): + """Test peer session listing with reverse creation-time ordering.""" + test_workspace, test_peer = sample_data + reverse_group = f"reverse-peer-sessions-{generate_nanoid()}" + first_session = f"reverse-peer-session-a-{generate_nanoid()}" + second_session = f"reverse-peer-session-b-{generate_nanoid()}" + + first_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions", + json={ + "id": first_session, + "peer_names": {test_peer.name: {}}, + "metadata": {"reverse_group": reverse_group}, + }, + ) + assert first_response.status_code in [200, 201] + + second_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions", + json={ + "id": second_session, + "peer_names": {test_peer.name: {}}, + "metadata": {"reverse_group": reverse_group}, + }, + ) + assert second_response.status_code in [200, 201] + + normal_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert reverse_response.status_code == 200 + + assert [item["id"] for item in normal_response.json()["items"]] == [ + first_session, + second_session, + ] + assert [item["id"] for item in reverse_response.json()["items"]] == [ + second_session, + first_session, + ] + + +@pytest.mark.asyncio +async def test_get_sessions_for_peer_reverse_uses_id_tiebreaker( + client: TestClient, + db_session: AsyncSession, + sample_data: tuple[Workspace, Peer], +): + """Peer-scoped sessions with identical created_at fall back to ordering by id.""" + test_workspace, test_peer = sample_data + reverse_group = f"tiebreaker-peer-sessions-{generate_nanoid()}" + shared_created_at = datetime.datetime( + 2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc + ) + + low_id = "A" * 21 + high_id = "z" * 21 + low_name = f"tie-low-peer-session-{generate_nanoid()}" + high_name = f"tie-high-peer-session-{generate_nanoid()}" + + for session_id, session_name in ((low_id, low_name), (high_id, high_name)): + db_session.add( + models.Session( + id=session_id, + name=session_name, + workspace_name=test_workspace.name, + created_at=shared_created_at, + h_metadata={"reverse_group": reverse_group}, + ) + ) + db_session.add( + models.SessionPeer( + workspace_name=test_workspace.name, + session_name=session_name, + peer_name=test_peer.name, + ) + ) + await db_session.commit() + + normal_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert reverse_response.status_code == 200 + + assert [item["id"] for item in normal_response.json()["items"]] == [ + low_name, + high_name, + ] + assert [item["id"] for item in reverse_response.json()["items"]] == [ + high_name, + low_name, + ] + + +def test_get_sessions_for_peer_reverse_with_pagination( + client: TestClient, sample_data: tuple[Workspace, Peer] +): + """Paged reverse listing of a peer's sessions returns newest-first across pages.""" + test_workspace, test_peer = sample_data + reverse_group = f"paged-reverse-peer-sessions-{generate_nanoid()}" + session_names = [ + f"paged-reverse-peer-session-{i}-{generate_nanoid()}" for i in range(3) + ] + + for session_name in session_names: + response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions", + json={ + "id": session_name, + "peer_names": {test_peer.name: {}}, + "metadata": {"reverse_group": reverse_group}, + }, + ) + assert response.status_code in [200, 201] + + page_one = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true&page=1&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_one.status_code == 200 + page_two = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true&page=2&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_two.status_code == 200 + page_three = client.post( + f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true&page=3&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_three.status_code == 200 + + assert page_one.json()["total"] == 3 + assert [item["id"] for item in page_one.json()["items"]] == [session_names[2]] + assert [item["id"] for item in page_two.json()["items"]] == [session_names[1]] + assert [item["id"] for item in page_three.json()["items"]] == [session_names[0]] + + def test_chat( client: TestClient, sample_data: tuple[Workspace, Peer], diff --git a/tests/routes/test_sessions.py b/tests/routes/test_sessions.py index c7025253..181336e1 100644 --- a/tests/routes/test_sessions.py +++ b/tests/routes/test_sessions.py @@ -1,8 +1,12 @@ +import datetime from typing import Any +import pytest from fastapi.testclient import TestClient from nanoid import generate as generate_nanoid +from sqlalchemy.ext.asyncio import AsyncSession +from src import models from src.models import Peer, Workspace @@ -223,6 +227,160 @@ def test_get_sessions_with_empty_filter( assert isinstance(data["items"], list) +def test_get_sessions_with_reverse( + client: TestClient, sample_data: tuple[Workspace, Peer] +): + """Test session listing with reverse creation-time ordering.""" + test_workspace, test_peer = sample_data + reverse_group = f"reverse-sessions-{generate_nanoid()}" + first_session = f"reverse-session-a-{generate_nanoid()}" + second_session = f"reverse-session-b-{generate_nanoid()}" + + first_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions", + json={ + "id": first_session, + "peer_names": {test_peer.name: {}}, + "metadata": {"reverse_group": reverse_group}, + }, + ) + assert first_response.status_code in [200, 201] + + second_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions", + json={ + "id": second_session, + "peer_names": {test_peer.name: {}}, + "metadata": {"reverse_group": reverse_group}, + }, + ) + assert second_response.status_code in [200, 201] + + normal_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions/list", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert reverse_response.status_code == 200 + + assert [item["id"] for item in normal_response.json()["items"]] == [ + first_session, + second_session, + ] + assert [item["id"] for item in reverse_response.json()["items"]] == [ + second_session, + first_session, + ] + + +@pytest.mark.asyncio +async def test_get_sessions_reverse_uses_id_tiebreaker( + client: TestClient, + db_session: AsyncSession, + sample_data: tuple[Workspace, Peer], +): + """Sessions with identical created_at fall back to ordering by id (nanoid PK).""" + test_workspace, _ = sample_data + reverse_group = f"tiebreaker-sessions-{generate_nanoid()}" + shared_created_at = datetime.datetime( + 2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc + ) + + low_id = "A" * 21 + high_id = "z" * 21 + low_name = f"tie-low-{generate_nanoid()}" + high_name = f"tie-high-{generate_nanoid()}" + + db_session.add( + models.Session( + id=low_id, + name=low_name, + workspace_name=test_workspace.name, + created_at=shared_created_at, + h_metadata={"reverse_group": reverse_group}, + ) + ) + db_session.add( + models.Session( + id=high_id, + name=high_name, + workspace_name=test_workspace.name, + created_at=shared_created_at, + h_metadata={"reverse_group": reverse_group}, + ) + ) + await db_session.commit() + + normal_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions/list", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert reverse_response.status_code == 200 + + # When created_at ties, ordering falls back to the nanoid id: low_id < high_id + # lexicographically, so low sorts first ascending and last descending. + assert [item["id"] for item in normal_response.json()["items"]] == [ + low_name, + high_name, + ] + assert [item["id"] for item in reverse_response.json()["items"]] == [ + high_name, + low_name, + ] + + +def test_get_sessions_reverse_with_pagination( + client: TestClient, sample_data: tuple[Workspace, Peer] +): + """Paged reverse listing returns newest-first across consecutive pages.""" + test_workspace, test_peer = sample_data + reverse_group = f"paged-reverse-sessions-{generate_nanoid()}" + session_names = [f"paged-reverse-session-{i}-{generate_nanoid()}" for i in range(3)] + + for session_name in session_names: + response = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions", + json={ + "id": session_name, + "peer_names": {test_peer.name: {}}, + "metadata": {"reverse_group": reverse_group}, + }, + ) + assert response.status_code in [200, 201] + + page_one = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true&page=1&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_one.status_code == 200 + page_two = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true&page=2&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_two.status_code == 200 + page_three = client.post( + f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true&page=3&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_three.status_code == 200 + + assert page_one.json()["total"] == 3 + assert [item["id"] for item in page_one.json()["items"]] == [session_names[2]] + assert [item["id"] for item in page_two.json()["items"]] == [session_names[1]] + assert [item["id"] for item in page_three.json()["items"]] == [session_names[0]] + + def test_update_delete_metadata( client: TestClient, sample_data: tuple[Workspace, Peer] ): diff --git a/tests/routes/test_workspaces.py b/tests/routes/test_workspaces.py index 031e5a90..0350b729 100644 --- a/tests/routes/test_workspaces.py +++ b/tests/routes/test_workspaces.py @@ -1,3 +1,4 @@ +import datetime from typing import Any from unittest.mock import AsyncMock, patch @@ -125,6 +126,135 @@ async def test_get_all_workspaces_with_null_filter(client: TestClient): assert isinstance(data["items"], list) +@pytest.mark.asyncio +async def test_get_all_workspaces_with_reverse(client: TestClient): + """Test workspace listing with reverse creation-time ordering.""" + first_name = f"reverse-workspace-{generate_nanoid()}" + second_name = f"reverse-workspace-{generate_nanoid()}" + + first_response = client.post( + "/v3/workspaces", + json={"name": first_name, "metadata": {"reverse_group": first_name}}, + ) + assert first_response.status_code in [200, 201] + + second_response = client.post( + "/v3/workspaces", + json={"name": second_name, "metadata": {"reverse_group": first_name}}, + ) + assert second_response.status_code in [200, 201] + + normal_response = client.post( + "/v3/workspaces/list", + json={"filters": {"metadata": {"reverse_group": first_name}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + "/v3/workspaces/list?reverse=true", + json={"filters": {"metadata": {"reverse_group": first_name}}}, + ) + assert reverse_response.status_code == 200 + + assert [item["id"] for item in normal_response.json()["items"]] == [ + first_name, + second_name, + ] + assert [item["id"] for item in reverse_response.json()["items"]] == [ + second_name, + first_name, + ] + + +@pytest.mark.asyncio +async def test_get_all_workspaces_reverse_uses_id_tiebreaker( + client: TestClient, db_session: AsyncSession +): + """Workspaces with identical created_at fall back to ordering by id (nanoid PK).""" + reverse_group = f"tiebreaker-{generate_nanoid()}" + shared_created_at = datetime.datetime( + 2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc + ) + + low_id = "A" * 21 + high_id = "z" * 21 + low_name = f"tie-low-{generate_nanoid()}" + high_name = f"tie-high-{generate_nanoid()}" + + db_session.add( + models.Workspace( + id=low_id, + name=low_name, + created_at=shared_created_at, + h_metadata={"reverse_group": reverse_group}, + ) + ) + db_session.add( + models.Workspace( + id=high_id, + name=high_name, + created_at=shared_created_at, + h_metadata={"reverse_group": reverse_group}, + ) + ) + await db_session.commit() + + normal_response = client.post( + "/v3/workspaces/list", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert normal_response.status_code == 200 + + reverse_response = client.post( + "/v3/workspaces/list?reverse=true", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert reverse_response.status_code == 200 + + normal_items = [item["id"] for item in normal_response.json()["items"]] + reverse_items = [item["id"] for item in reverse_response.json()["items"]] + + # When created_at ties, ordering falls back to the nanoid id: low_id < high_id + # lexicographically, so the workspace with id="AAA..." sorts first ascending. + assert normal_items == [low_name, high_name] + assert reverse_items == [high_name, low_name] + + +@pytest.mark.asyncio +async def test_get_all_workspaces_reverse_with_pagination(client: TestClient): + """Paged reverse listing returns newest-first across consecutive pages.""" + reverse_group = f"paged-reverse-{generate_nanoid()}" + names = [f"paged-reverse-{i}-{generate_nanoid()}" for i in range(3)] + + for name in names: + response = client.post( + "/v3/workspaces", + json={"name": name, "metadata": {"reverse_group": reverse_group}}, + ) + assert response.status_code in [200, 201] + + page_one = client.post( + "/v3/workspaces/list?reverse=true&page=1&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_one.status_code == 200 + page_two = client.post( + "/v3/workspaces/list?reverse=true&page=2&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_two.status_code == 200 + page_three = client.post( + "/v3/workspaces/list?reverse=true&page=3&size=1", + json={"filters": {"metadata": {"reverse_group": reverse_group}}}, + ) + assert page_three.status_code == 200 + + assert page_one.json()["total"] == 3 + assert [item["id"] for item in page_one.json()["items"]] == [names[2]] + assert [item["id"] for item in page_two.json()["items"]] == [names[1]] + assert [item["id"] for item in page_three.json()["items"]] == [names[0]] + + def test_update_workspace(client: TestClient, sample_data: tuple[Workspace, Peer]): test_workspace, _ = sample_data _new_name = str(generate_nanoid()) From 7470866d12845ed4b56bf3449d058e65df96b1c1 Mon Sep 17 00:00:00 2001 From: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> Date: Thu, 21 May 2026 14:32:41 -0400 Subject: [PATCH 6/7] chore(docs): Update changelogs and increment version (#713) --- CHANGELOG.md | 77 +- CLAUDE.md | 7 +- README.md | 46 +- docs/changelog/compatibility-guide.mdx | 7 +- docs/changelog/introduction.mdx | 81 +- docs/docs.json | 2 +- docs/v3/openapi.json | 2322 +++++------------------- pyproject.toml | 2 +- sdks/python/CHANGELOG.md | 7 +- sdks/python/pyproject.toml | 2 +- sdks/typescript/CHANGELOG.md | 10 +- sdks/typescript/package.json | 2 +- uv.lock | 6 +- 13 files changed, 643 insertions(+), 1928 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 901265ef..60085e5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,39 +5,74 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). -## [Unreleased] +## [3.0.7] - 2026-05-21 ### Added -- New `src/llm/` package as the single owner of provider runtime: clients, backends, history adapters, tool loop, request builder, credentials, and caching policy -- `AttemptPlan` dataclass captures per-retry provider selection (client, model, reasoning_effort, thinking_budget_tokens, selected_config) and pins it across stream-final retries so streaming doesn't bounce back to primary after the tool loop has settled on fallback -- Gemini JSON-schema sanitizer for `function_declarations` — strips keywords Gemini's validator rejects (`additionalProperties`, `allOf`, etc.) while preserving semantics for all other backends -- Dreamer specialists derive `effective_max_tokens` from `model_config.max_output_tokens` with a per-specialist default fallback -- Regression tests covering fallback-config thinking-param reach, provider_params → extra_params boundary, OpenAI reasoning-model parameter routing, Gemini blocked finish_reason handling, and fail-fast `max_tool_iterations` validation +- New `src/llm/` module as the single owner of provider runtime: clients, backends, history adapters, tool loop, request builder, credentials, and caching policy (#459) +- `AttemptPlan` dataclass captures per-retry provider selection (client, model, reasoning_effort, thinking_budget_tokens, selected_config) and pins it across stream-final retries so streaming doesn't bounce back to primary after the tool loop has settled on fallback (#459) +- Gemini JSON-schema sanitizer for `function_declarations` — strips keywords Gemini's validator rejects (`additionalProperties`, `allOf`, etc.) while preserving semantics for all other backends (#459) +- Dreamer specialists derive `effective_max_tokens` from `model_config.max_output_tokens` with a per-specialist default fallback (#459) +- New cloudevent `LLMCallCompletedEvent` (`llm.call.completed`) fires once per provider hit with full cost-attribution context: transport/provider_label, model, token counts with cache breakdown, finish_reason, outcome, `is_final_attempt`, retry/fallback state, duration, tool-call shape, streaming flag, and agent correlation (`run_id` + iteration). Includes a `CallPurpose` closed enum (`deriver.representation`, `dialectic.answer`, `dream.deduction|induction`, `summary.short|long`) (#637) +- `RepresentationCompletedEvent` now carries `total_input_tokens` for full-trace cost attribution (#637) +- Per-emitter `honcho_version` injection on all CloudEvents plus emitter health metrics (#637) +- `TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE` (default 1.0) — deterministic per-`run_id` sampler so an entire agent trace is kept or dropped together; aggregate envelopes bypass the sampler (#637) +- Deriver custom instructions: per-workspace/peer guidance threaded into the deriver prompt with a `MAX_CUSTOM_INSTRUCTIONS_TOKENS` budget (default 2000); deriver `MAX_INPUT_TOKENS` raised 23000 → 25000 to make room (#609) +- Configurable embedding dimensions: `EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE` (`auto`/`always`/`never`) controls whether the OpenAI `dimensions=` parameter is forwarded; `auto` (default) sends it when the operator explicitly set `EMBEDDING_VECTOR_DIMENSIONS` and the model is not on the known-rejecting allowlist (#678) +- New `honcho-cli` package — Python CLI for inspecting and managing peers, sessions, and configuration against a Honcho deployment (#424) +- `HONCHO_API_URL` env var support in the MCP Worker, enabling self-hosted Honcho deployments to point the Worker at their own instance instead of `https://api.honcho.dev` (#575) +- API ID `max_length` increased from 100 to 512 across `WorkspaceCreate`, `PeerCreate`, and `SessionCreate` to align the API contract with the underlying DB schema (#684) +- Regression tests covering fallback-config thinking-param reach, provider_params → extra_params boundary, OpenAI reasoning-model parameter routing, Gemini blocked finish_reason handling, and fail-fast `max_tool_iterations` validation (#459) ### Changed -- All LLM orchestration moved out of `src/utils/clients.py` into `src/llm/` with modules split by responsibility (api, executor, tool_loop, runtime, registry, conversation, request_builder, credentials, caching, backends, history_adapters) -- Default `ModelConfig` factories (deriver, summary, dreamer specialists, dialectic levels) normalized to `openai/gpt-5.4-mini` with no extra parameters set by default; operators add transport/thinking overrides explicitly -- OpenAI reasoning-model routing widened via `_uses_max_completion_tokens` heuristic covering `gpt-5.x` and `o1/o3/o4` — these models receive `max_completion_tokens` instead of `max_tokens` -- Override client factories switched from unbounded `@cache` to `@lru_cache(maxsize=128)` for predictable memory growth on long-running processes -- `get_backend` now delegates to `client_for_model_config`, so the live-test path and production path share one missing-API-key validation -- Blocked Gemini responses (`SAFETY`, `RECITATION`, `PROHIBITED_CONTENT`, `BLOCKLIST`) raise `LLMError` in the streaming path too (previously only the non-streaming path), ensuring retry/fallback logic fires uniformly -- Transport-change env overrides now strip transport-specific thinking params (thinking_budget_tokens vs. reasoning_effort) during config merge, including at the dialectic-level merge, so switching from Anthropic → OpenAI doesn't leave orphaned Anthropic-only params that the OpenAI backend would reject -- `max_tool_iterations` out-of-range inputs now raise `ValidationException` instead of being silently clamped -- Troubleshooting docs updated to reflect nested-env-var form for per-component thinking-budget overrides +- All LLM orchestration moved out of `src/utils/clients.py` into `src/llm/` with modules split by responsibility (api, executor, tool_loop, runtime, registry, conversation, request_builder, credentials, caching, backends, history_adapters) (#459) +- Default `ModelConfig` factories (deriver, summary, dreamer specialists, dialectic levels) normalized to `openai/gpt-5.4-mini` with no extra parameters set by default; operators add transport/thinking overrides explicitly (#459) +- OpenAI reasoning-model routing widened via `_uses_max_completion_tokens` heuristic covering `gpt-5.x` and `o1/o3/o4` — these models receive `max_completion_tokens` instead of `max_tokens` (#459) +- Override client factories switched from unbounded `@cache` to `@lru_cache(maxsize=128)` for predictable memory growth on long-running processes (#459) +- `get_backend` now delegates to `client_for_model_config`, so the live-test path and production path share one missing-API-key validation (#459) +- Blocked Gemini responses (`SAFETY`, `RECITATION`, `PROHIBITED_CONTENT`, `BLOCKLIST`) raise `LLMError` in the streaming path too (previously only the non-streaming path), ensuring retry/fallback logic fires uniformly (#459) +- Transport-change env overrides now strip transport-specific thinking params (thinking_budget_tokens vs. reasoning_effort) during config merge, including at the dialectic-level merge, so switching from Anthropic → OpenAI doesn't leave orphaned Anthropic-only params that the OpenAI backend would reject (#459) +- `max_tool_iterations` out-of-range inputs now raise `ValidationException` instead of being silently clamped (#459) +- Public API schemas (`WorkspaceCreate`, `PeerCreate`, `SessionCreate`) and SDK validation (`api_types.py`, `validation.ts`) accept IDs up to 512 chars (was 100) (#684) +- Peer card prompts reframed as stable identity markers (replaces the prior "biographical/profile facts" language). Induction specialist is now opted out of peer card writes (`can_update_peer_card = False`) so only deduction touches the card (#686) +- Vector store queries no longer fetch embedding vectors — only document metadata is returned, reducing payload size and DB load (pgvector, lancedb, turbopuffer) (#682) +- Langfuse trace metadata now includes `namespace`, `model`, and `provider` so traces can be filtered by deployment slice (#565) +- Deriver: model-aware tokenizer (replaces the previously hardcoded encoding) and explicit guard on empty message content (#647) +- Dialectic level defaults now merge correctly with per-level overrides in `src/config` (DEV-1733) (#656) +- Default dialectic tool choice switched from forced/required to `auto` (#630) +- Vector sync given a substantial retry budget to tolerate transient embedding provider outages (#604) +- `AgentToolConclusionsDeletedEvent` payload now carries `levels` for parity with the rest of the conclusion event surface (#612) +- Turbopuffer vector store: `InternalServerError` caught and surfaced as a warning rather than a hard failure; unused `upsert_with_retry` and `VectorUpsertResult` removed; explicit silent and explicit-error paths for vector DB server errors (#561) +- Troubleshooting docs updated to reflect nested-env-var form for per-component thinking-budget overrides (#459) +- README refresh (#681) +- CLAUDE.md refreshed against the current `src/` layout (#680) ### Fixed -- Fallback `ModelConfig` temperature and `thinking_budget_tokens` reach the backend on the final retry — previously the primary's values were pre-populated into caller kwargs early and clobbered fallback values via `effective_config_for_call(update=...)` -- Stream-final retries pin to the `AttemptPlan` that succeeded rather than re-running provider selection through the outer `current_attempt` ContextVar (which could roll streaming back to primary after the tool loop had already switched to fallback) -- OpenAI structured-output calls continue to use `chat.completions.parse()` with strict schema enforcement, while tool-calling paths use `chat.completions.create()` without `strict:True` for broader proxy compatibility (OpenRouter, vLLM, Ollama) -- Gemini `cached_content` reuse keys now include `system_instruction` and `tool_config` so cache hits don't cross configurations that differ only in those fields -- `reverse` query parameter is now honored on the v3 workspace list (`POST /v3/workspaces/list`), peer list (`POST /v3/workspaces/{workspace_id}/peers/list`), workspace-scoped session list (`POST /v3/workspaces/{workspace_id}/sessions/list`), and peer-scoped session list (`POST /v3/workspaces/{workspace_id}/peers/{peer_id}/sessions`). Honcho SDKs at 2.1.0+ were already sending `reverse=true` for these routes but the server silently ignored it. Ties on `created_at` now fall back to the internal nanoid `id` so ordering remains stable across pages. +- Fallback `ModelConfig` temperature and `thinking_budget_tokens` reach the backend on the final retry — previously the primary's values were pre-populated into caller kwargs early and clobbered fallback values via `effective_config_for_call(update=...)` (#459) +- Stream-final retries pin to the `AttemptPlan` that succeeded rather than re-running provider selection through the outer `current_attempt` ContextVar (which could roll streaming back to primary after the tool loop had already switched to fallback) (#459) +- OpenAI structured-output calls continue to use `chat.completions.parse()` with strict schema enforcement, while tool-calling paths use `chat.completions.create()` without `strict:True` for broader proxy compatibility (OpenRouter, vLLM, Ollama) (#459) +- Gemini `cached_content` reuse keys now include `system_instruction` and `tool_config` so cache hits don't cross configurations that differ only in those fields (#459) +- Removed strict parameter validation for thinking params on Anthropic and OpenAI transports — was rejecting valid per-transport configs (#686) +- `reverse` query parameter is now honored on the v3 workspace list (`POST /v3/workspaces/list`), peer list (`POST /v3/workspaces/{workspace_id}/peers/list`), workspace-scoped session list (`POST /v3/workspaces/{workspace_id}/sessions/list`), and peer-scoped session list (`POST /v3/workspaces/{workspace_id}/peers/{peer_id}/sessions`). Honcho SDKs at 2.1.0+ were already sending `reverse=true` for these routes but the server silently ignored it. Ties on `created_at` now fall back to the internal nanoid `id` so ordering remains stable across pages (#685) +- LLM client factories now receive `base_url` from `LLMSettings` for default providers — previously the override path honored `base_url` but the default path didn't, so operators pointing at OpenAI-compatible proxies via `LLM__OPENAI_BASE_URL` were ignored (#643, fixes #641) +- Internal N+1 query in dialectic agent tool execution (DEV-1721) — collapsed per-iteration DB lookups into a single fetch (#652) +- Dreamer threshold and time-guard semantics: `check_and_schedule_dream` count filter now includes only `documents.level == 'explicit'` (dreamer-created levels are output, not input, and were inflating the threshold and creating a feedback loop); `last_dream_at` write relocated from `enqueue_dream` into `process_dream` so duplicate enqueues or failed runs no longer reset the 8-hour time guard (#573) +- Deriver: blank observations are filtered out before embedding (previously triggered noisy embedding calls and persisted empty rows); blank-observation filtering unified across tool paths (#615) +- Surprisal module: filter for level observations changed from `{"level": levels}` to `{"level": {"in": levels}}` — `apply_filter()` requires operator syntax, so the prior call silently returned 0 results and made the entire Surprisal phase of the Dream cycle a no-op (#581, fixes #559) +- Removed hardcoded `stop_sequences` override from Deriver `ModelConfig` (was clobbering operator-configured stop sequences) (#587) +- Removed stale `stop_sequences` from tests (#607) +- Embedding client: `embed()` now wraps single-string input in an array, restoring compatibility with OpenAI-compatible third-party providers that reject scalar input (#586) +- Docker Compose: deriver service startup gated on the API service healthcheck (prevents races where the deriver starts before the API has run migrations) (#689) +- Docker image: `HEALTHCHECK` directive removed from the shared base image — it probed an HTTP endpoint only the API serves, permanently marking deriver containers as unhealthy. Service-level health checks now belong in each service's own configuration (k8s readiness/liveness probes on the API Deployment only) (#530) +- `tests/unified`: `--test-dir`/`--test-file` arguments now use an argparse mutually-exclusive group instead of manual validation (#650) +- CrewAI example updated for the latest CrewAI protocol (#631) ### Removed -- `src/utils/clients.py` deleted; its responsibilities are split across `src/llm/registry.py`, `src/llm/credentials.py`, and the backend-specific modules +- `src/utils/clients.py` deleted; its responsibilities are split across `src/llm/registry.py`, `src/llm/credentials.py`, and the backend-specific modules (#459) +- `HEALTHCHECK` directive removed from the shared Docker image (#530) ## [3.0.6] - 2026-04-10 diff --git a/CLAUDE.md b/CLAUDE.md index 6fa9ce2f..fbd05871 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -140,6 +140,7 @@ The Deriver processes batches of incoming messages and extracts conclusions abou - **Output**: Explicit conclusions (direct facts) and deductive conclusions (inferences) saved to `(observer, observed)` collections. - **Entry point**: `src/deriver/__main__.py` → `queue_manager.main()`. - **Prompts**: `src/deriver/prompts.py` (`minimal_deriver_prompt`). +- **Custom instructions**: per-workspace/peer guidance can be threaded into the prompt via reasoning configuration; `DERIVER__MAX_CUSTOM_INSTRUCTIONS_TOKENS` caps the addition (default 2000) and `DERIVER__MAX_INPUT_TOKENS` defaults to 25000 to make room. #### 2. Dialectic (`src/dialectic/`) @@ -177,8 +178,9 @@ The Dreamer is an orchestrated multi-specialist system that runs during schedule #### Shared Agent Infrastructure - **Tool definitions** (`src/utils/agent_tools.py`): unified `TOOLS` dict; per-agent lists (`DIALECTIC_TOOLS`, `DIALECTIC_TOOLS_MINIMAL`, `DREAMER_TOOLS`, `DEDUCTION_SPECIALIST_TOOLS`, `INDUCTION_SPECIALIST_TOOLS`). -- **LLM subsystem** (`src/llm/`): provider-agnostic `honcho_llm_call()`. Backends in `src/llm/backends/` (`anthropic.py`, `gemini.py`, `openai.py`). Includes prompt caching (`caching.py`), structured output (`structured_output.py`), tool loop (`tool_loop.py`), history adapters for cross-provider message formats, and a model registry. +- **LLM subsystem** (`src/llm/`): provider-agnostic `honcho_llm_call()`. Backends in `src/llm/backends/` (`anthropic.py`, `gemini.py`, `openai.py`). Includes prompt caching (`caching.py`), structured output (`structured_output.py`), tool loop (`tool_loop.py`), history adapters for cross-provider message formats, and a model registry. Per-retry provider selection is pinned via an `AttemptPlan` so stream-final retries don't bounce back to primary after the tool loop has settled on fallback. - **Per-agent model config**: each agent has its own `MODEL_CONFIG` in `src/config.py` with fallback chains (see `ConfiguredModelSettings`, `FallbackModelSettings`). +- **Telemetry**: cloudevents in `src/telemetry/events/` cover API routes, dialectic, dream, deletion, reconciliation, representation, and per-call LLM accounting (`llm.py` — `LLMCallCompletedEvent` fires once per provider hit with full cost-attribution context). High-volume events are sampled deterministically per `run_id` via `TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE`. ### Project Structure @@ -192,7 +194,8 @@ src/ ├── dependencies.py # FastAPI DI (tracked_db, etc.) ├── exceptions.py # Custom exception types (HonchoException + subclasses) ├── security.py # JWT authentication -├── embedding_client.py # Embedding provider client +├── embedding_client.py # Embedding provider client (configurable dimensions +│ # via EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE) ├── schemas/ # Pydantic schemas │ ├── api.py # Public API request/response schemas │ ├── configuration.py # Per-resource configuration schemas diff --git a/README.md b/README.md index 140104b6..3aa3810a 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ --- -![Static Badge](https://img.shields.io/badge/Server-3.0.6-blue) +![Static Badge](https://img.shields.io/badge/Server-3.0.7-blue) [![PyPI version](https://img.shields.io/pypi/v/honcho-ai.svg)](https://pypi.org/project/honcho-ai/) [![NPM version](https://img.shields.io/npm/v/@honcho-ai/sdk.svg)](https://npmjs.org/package/@honcho-ai/sdk) [![Discord](https://img.shields.io/discord/1016845111637839922?style=flat&logo=discord&logoColor=23ffffff&label=Plastic%20Labs&labelColor=235865F2)](https://discord.gg/honcho) @@ -43,21 +43,21 @@ The Honcho project is split between several repositories, with this one hosting ## Start Here -| I want to... | Path | Get started | -|---|---|---| +| I want to... | Path | Get started | +| -------------------------------------- | ---------------------------------------------------------- | ----------------------------- | | Give my coding agent persistent memory | Claude Code, OpenCode, OpenClaw, Hermes, or any MCP client | [Integrations](#integrations) | -| Add memory to my product | Python or TypeScript SDK | [Quickstart](#quickstart) | -| Self-host Honcho | Docker / local development | [Self-hosting](#self-hosting) | +| Add memory to my product | Python or TypeScript SDK | [Quickstart](#quickstart) | +| Self-host Honcho | Docker / local development | [Self-hosting](#self-hosting) | ## Why Honcho -| Capability | What it means | -|---|---| -| Reasoning-first memory | Extracts conclusions from conversations and events, not just matching chunks. | -| Peer-centric model | Tracks users, agents, groups, projects, and ideas as entities that change over time. | -| Multi-peer perspective | Models what one peer knows about another when configured. | -| Managed or self-hosted | Use `api.honcho.dev` or run the FastAPI server yourself. | -| Agent-tool integrations | MCP, Claude Code, OpenCode, OpenClaw, Hermes, Cursor-compatible clients. | +| Capability | What it means | +| ----------------------- | ------------------------------------------------------------------------------------ | +| Reasoning-first memory | Extracts conclusions from conversations and events, not just matching chunks. | +| Peer-centric model | Tracks users, agents, groups, projects, and ideas as entities that change over time. | +| Multi-peer perspective | Models what one peer knows about another when configured. | +| Managed or self-hosted | Use `api.honcho.dev` or run the FastAPI server yourself. | +| Agent-tool integrations | MCP, Claude Code, OpenCode, OpenClaw, Hermes, Cursor-compatible clients. | ## The Honcho Loop @@ -139,7 +139,9 @@ await session.addMessages([ tutor.message("Absolutely. Send me your first problem!"), ]); -const answer = await alice.chat("What learning styles does the user respond to best?"); +const answer = await alice.chat( + "What learning styles does the user respond to best?", +); const context = await session.context({ summary: true, tokens: 10_000 }); const openai = new OpenAI(); @@ -153,15 +155,15 @@ const completion = await openai.chat.completions.create({ ## What Honcho Gives You -| Need | API | -|---|---| -| Save interaction history | `session.add_messages(...)` | -| Ask what Honcho knows about a peer | `peer.chat(...)` | -| Get prompt-ready context | `session.context(...).to_openai(...)` / `.to_anthropic(...)` | -| Hybrid search (BM25 + vector) | `peer.search(...)`, `session.search(...)`, `honcho.search(...)` | -| Low-latency static representations | `peer.representation(...)`, `session.representation(...)` | -| Import documents | `session.upload_file(...)` | -| Inspect background processing | `honcho.queue_status(...)` | +| Need | API | +| ---------------------------------- | --------------------------------------------------------------- | +| Save interaction history | `session.add_messages(...)` | +| Ask what Honcho knows about a peer | `peer.chat(...)` | +| Get prompt-ready context | `session.context(...).to_openai(...)` / `.to_anthropic(...)` | +| Hybrid search (BM25 + vector) | `peer.search(...)`, `session.search(...)`, `honcho.search(...)` | +| Low-latency static representations | `peer.representation(...)`, `session.representation(...)` | +| Import documents | `session.upload_file(...)` | +| Inspect background processing | `honcho.queue_status(...)` | See the full [SDK Reference](https://honcho.dev/docs/v3/documentation/reference/sdk) and [API Reference](https://honcho.dev/docs/v3/api-reference/introduction). diff --git a/docs/changelog/compatibility-guide.mdx b/docs/changelog/compatibility-guide.mdx index 17fd55c9..f12d3cee 100644 --- a/docs/changelog/compatibility-guide.mdx +++ b/docs/changelog/compatibility-guide.mdx @@ -10,14 +10,14 @@ This guide helps you match the right SDK version to your Honcho API version. New - **Latest:** v2.1.1 + **Latest:** v2.1.2 ```bash npm install @honcho-ai/sdk ``` - **Latest:** v2.1.1 + **Latest:** v2.1.2 ```bash pip install honcho-ai @@ -30,7 +30,8 @@ This guide helps you match the right SDK version to your Honcho API version. New | Honcho API Version | TypeScript SDK | Python SDK | |-------------------|---------------|------------| -| v3.0.6 (Current) | v2.1.1 | v2.1.1 | +| v3.0.7 (Current) | v2.1.2 | v2.1.2 | +| v3.0.6 | v2.1.1 | v2.1.1 | | v3.0.5 | v2.1.0 | v2.1.0 | | v3.0.4 | v2.1.0 | v2.1.0 | | v3.0.3 | v2.1.0 | v2.1.0 | diff --git a/docs/changelog/introduction.mdx b/docs/changelog/introduction.mdx index d2d31331..29c15fdf 100644 --- a/docs/changelog/introduction.mdx +++ b/docs/changelog/introduction.mdx @@ -27,7 +27,59 @@ Welcome to the Honcho changelog! This section documents all notable changes to t ### Honcho API and SDK Changelogs - + + ### Added + + - New `src/llm/` package as the single owner of provider runtime: clients, backends, history adapters, tool loop, request builder, credentials, and caching policy (#459) + - New cloudevent `LLMCallCompletedEvent` (`llm.call.completed`) fires once per provider hit with full cost-attribution context: transport/provider_label, model, token counts with cache breakdown, finish_reason, outcome, retry/fallback state, duration, tool-call shape, streaming flag, and agent correlation (`run_id` + iteration) (#637) + - `RepresentationCompletedEvent` now carries `total_input_tokens` for full-trace cost attribution; per-emitter `honcho_version` injection; deterministic per-`run_id` high-volume sampler via `TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE` (#637) + - Deriver custom instructions: per-workspace/peer guidance threaded into the deriver prompt with a `MAX_CUSTOM_INSTRUCTIONS_TOKENS` budget (default 2000); deriver `MAX_INPUT_TOKENS` raised 23000 → 25000 (#609) + - Configurable embedding dimensions: `EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE` (`auto`/`always`/`never`) controls whether OpenAI `dimensions=` is forwarded (#678) + - New `honcho-cli` package — Python CLI for inspecting and managing peers, sessions, and configuration against a Honcho deployment (#424) + - `HONCHO_API_URL` env var support in the MCP Worker for self-hosted deployments (#575) + - API ID `max_length` increased from 100 to 512 across `WorkspaceCreate`, `PeerCreate`, and `SessionCreate` to align with the DB schema (#684) + - `AttemptPlan` dataclass pins per-retry provider selection across stream-final retries so streaming doesn't bounce back to primary after the tool loop has settled on fallback (#459) + - Gemini JSON-schema sanitizer for `function_declarations` — strips keywords Gemini's validator rejects while preserving semantics for other backends (#459) + + ### Changed + + - All LLM orchestration moved out of `src/utils/clients.py` into `src/llm/` with modules split by responsibility (#459) + - Default `ModelConfig` factories (deriver, summary, dreamer specialists, dialectic levels) normalized with no extra parameters set by default; operators add transport/thinking overrides explicitly (#459) + - OpenAI reasoning-model routing widened to cover `gpt-5.x` and `o1/o3/o4` — these models receive `max_completion_tokens` instead of `max_tokens` (#459) + - Peer card prompts reframed as stable identity markers; induction specialist now opts out of peer card writes so only deduction touches the card (#686) + - Vector store queries no longer fetch embedding vectors — only document metadata is returned, reducing payload size and DB load (pgvector, lancedb, turbopuffer) (#682) + - Langfuse trace metadata now includes `namespace`, `model`, and `provider` so traces can be filtered by deployment slice (#565) + - Deriver: model-aware tokenizer (replaces the previously hardcoded encoding) and explicit guard on empty message content (#647) + - Dialectic level defaults now merge correctly with per-level overrides (#656) + - Default dialectic tool choice switched to `auto` (#630) + - Vector sync given a substantial retry budget to tolerate transient embedding provider outages (#604) + - `AgentToolConclusionsDeletedEvent` payload now carries `levels` (#612) + - Turbopuffer: `InternalServerError` caught and surfaced as a warning rather than a hard failure; vector store sync errors downgraded to warnings (#561) + + ### Fixed + + - `reverse` query parameter is now honored on the v3 workspace list, peer list, workspace-scoped session list, and peer-scoped session list. Honcho SDKs at 2.1.0+ were already sending `reverse=true` for these routes but the server silently ignored it. Ties on `created_at` now fall back to the internal nanoid `id` for stable ordering across pages (#685) + - LLM client factories now receive `base_url` from `LLMSettings` for default providers — operators pointing at OpenAI-compatible proxies via `LLM__OPENAI_BASE_URL` were previously ignored on the default path (#643, fixes #641) + - Internal N+1 query in dialectic agent tool execution — collapsed per-iteration DB lookups into a single fetch (#652) + - Dreamer threshold and time-guard semantics: count filter now includes only `documents.level == 'explicit'` (was inflating threshold via dreamer-created levels and creating a feedback loop); `last_dream_at` write relocated from enqueue to process so duplicate enqueues or failed runs no longer reset the 8-hour time guard (#573) + - Deriver: blank observations are filtered out before embedding (previously triggered noisy embedding calls and persisted empty rows) (#615) + - Surprisal module: filter format corrected from `{"level": levels}` to `{"level": {"in": levels}}` — the prior call silently returned 0 results and made the entire Surprisal phase of the Dream cycle a no-op (#581, fixes #559) + - Removed hardcoded `stop_sequences` override from Deriver `ModelConfig` (was clobbering operator-configured stop sequences) (#587) + - Embedding client: `embed()` now wraps single-string input in an array, restoring compatibility with OpenAI-compatible third-party providers that reject scalar input (#586) + - Docker Compose: deriver service startup gated on the API service healthcheck — prevents races where the deriver starts before the API has run migrations (#689) + - Docker image: `HEALTHCHECK` directive removed from the shared base image; service-level health checks now belong in each service's own configuration (#530) + - Removed strict parameter validation for thinking params on Anthropic and OpenAI transports — was rejecting valid per-transport configs (#686) + - Stream-final retries pin to the `AttemptPlan` that succeeded rather than re-running provider selection through the outer `current_attempt` ContextVar (#459) + - Gemini `cached_content` reuse keys now include `system_instruction` and `tool_config` so cache hits don't cross configurations (#459) + - CrewAI example updated for the latest CrewAI protocol (#631) + + ### Removed + + - `src/utils/clients.py` deleted; its responsibilities are split across `src/llm/registry.py`, `src/llm/credentials.py`, and the backend-specific modules (#459) + - `HEALTHCHECK` directive from the shared Docker image (#530) + + + ### Changed - Tightened transaction scopes across search, agent tools, queue manager, and webhook delivery to minimize DB connection hold time during external operations (#525) @@ -558,7 +610,17 @@ Welcome to the Honcho changelog! This section documents all notable changes to t [Python SDK](https://pypi.org/project/honcho-ai/) - + + ### Added + + - `page`, `size`, and `reverse` pagination parameters on `Honcho.workspaces()` and `HonchoAio.workspaces()`, closing the gap from 2.1.0 which added these to other list methods but not to `workspaces()`. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter. + - `peers` parameter on `Honcho.session()` and `HonchoAio.session()` — attach peers to a session at creation time instead of needing a follow-up `session.add_peers()` call. Accepts the same shapes as `Session.add_peers` (peer ID string, `Peer` object, list of either, or tuples with `SessionPeerConfig`). + + ### Changed + + - `WorkspaceCreateParams`, `PeerCreateParams`, and `SessionCreateParams` now accept IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7. + + ### Fixed - Broadened HTTP retry logic to cover `httpx.NetworkError` and `httpx.RemoteProtocolError` in addition to `httpx.TimeoutException` and `httpx.ConnectError`, improving resilience against transient network failures @@ -700,7 +762,20 @@ Welcome to the Honcho changelog! This section documents all notable changes to t [TypeScript SDK](https://www.npmjs.com/package/@honcho-ai/sdk) - + + ### Added + + - `peers` option on `Honcho.session()` — attach peers to a session at creation time instead of needing a follow-up `session.addPeers()` call. Accepts the same `PeerAddition` shape as `session.addPeers()` (peer ID strings, `Peer` objects, arrays of either, or a record with per-peer `observe_me`/`observe_others` config). + + ### Changed + + - ID validation in `validation.ts` now accepts workspace, peer, and session IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7. + + ### Fixed + + - `Honcho.workspaces()` now actually forwards the `reverse` option to the server. The 2.1.0 changelog listed `workspaces()` among the list methods that gained `reverse`, but `client.ts` was missing the field on the params type and request builder, so the option was silently dropped. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter. + + ### Fixed - Broadened fetch error retry logic to catch all `TypeError` network failures (connection resets, DNS errors, etc.) instead of only those with `'fetch'` in the message, improving resilience across runtimes (Node, Bun, browsers) diff --git a/docs/docs.json b/docs/docs.json index 6287bf44..5ccb4166 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -24,7 +24,7 @@ "navigation": { "versions": [ { - "version": "v3.0.6", + "version": "v3.0.7", "api": { "openapi": ["v3/openapi.json"] }, diff --git a/docs/v3/openapi.json b/docs/v3/openapi.json index 161cafc5..adcd9d29 100644 --- a/docs/v3/openapi.json +++ b/docs/v3/openapi.json @@ -9,11 +9,7 @@ "url": "https://honcho.dev/", "email": "hello@plasticlabs.ai" }, - "license": { - "name": "GNU Affero General Public License v3.0", - "url": "https://github.com/plastic-labs/honcho/blob/main/LICENSE" - }, - "version": "3.0.3" + "version": "3.0.7" }, "servers": [ { @@ -48,9 +44,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Workspace" - } + "schema": { "$ref": "#/components/schemas/Workspace" } } } }, @@ -58,19 +52,12 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } }, - "security": [ - { - "HTTPBearer": [] - }, - {} - ] + "security": [{ "HTTPBearer": [] }] } }, "/v3/workspaces/list": { @@ -79,13 +66,20 @@ "summary": "Get All Workspaces", "description": "Get all Workspaces, paginated with optional filters.", "operationId": "get_all_workspaces_v3_workspaces_list_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ + { + "name": "reverse", + "in": "query", + "required": false, + "schema": { + "type": "boolean", + "description": "Whether to reverse the order of results", + "default": false, + "title": "Reverse" + }, + "description": "Whether to reverse the order of results" + }, { "name": "page", "in": "query", @@ -99,25 +93,6 @@ }, "description": "Page number" }, - { - "name": "reverse", - "in": "query", - "required": false, - "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], - "description": "Whether to reverse the order of results", - "default": false, - "title": "Reverse" - }, - "description": "Whether to reverse the order of results" - }, { "name": "size", "in": "query", @@ -138,12 +113,8 @@ "application/json": { "schema": { "anyOf": [ - { - "$ref": "#/components/schemas/WorkspaceGet" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/WorkspaceGet" }, + { "type": "null" } ], "description": "Filtering and pagination options for the workspaces list", "title": "Options" @@ -156,9 +127,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_Workspace_" - } + "schema": { "$ref": "#/components/schemas/Page_Workspace_" } } } }, @@ -166,9 +135,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -181,21 +148,13 @@ "summary": "Update Workspace", "description": "Update Workspace metadata and/or configuration.", "operationId": "update_workspace_v3_workspaces__workspace_id__put", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "requestBody": { @@ -214,9 +173,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Workspace" - } + "schema": { "$ref": "#/components/schemas/Workspace" } } } }, @@ -224,9 +181,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -237,39 +192,25 @@ "summary": "Delete Workspace", "description": "Delete a Workspace. This accepts the deletion request and processes it in the background,\npermanently deleting all peers, messages, conclusions, and other resources associated\nwith the workspace.\n\nReturns 409 Conflict if the workspace contains active sessions.\nDelete all sessions first, then delete the workspace.\n\nThis action cannot be undone.", "operationId": "delete_workspace_v3_workspaces__workspace_id__delete", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "responses": { "202": { "description": "Successful Response", - "content": { - "application/json": { - "schema": {} - } - } + "content": { "application/json": { "schema": {} } } }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -282,21 +223,13 @@ "summary": "Search Workspace", "description": "Search messages in a Workspace using optional filters. Use `limit` to control the number of\nresults returned.", "operationId": "search_workspace_v3_workspaces__workspace_id__search_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "requestBody": { @@ -317,9 +250,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "$ref": "#/components/schemas/Message" - }, + "items": { "$ref": "#/components/schemas/Message" }, "title": "Response Search Workspace V3 Workspaces Workspace Id Search Post" } } @@ -329,9 +260,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -344,35 +273,20 @@ "summary": "Get Queue Status", "description": "Get the processing queue status for a Workspace, optionally scoped to an observer, sender,\nand/or session.\n\nOnly tracks user-facing task types (representation, summary, dream).\nInternal infrastructure tasks (reconciler, webhook, deletion) are excluded.\nNote: completed counts reflect items since the last periodic queue cleanup,\nnot lifetime totals.", "operationId": "get_queue_status_v3_workspaces__workspace_id__queue_status_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "observer_id", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Optional observer ID to filter by", "title": "Observer Id" }, @@ -383,14 +297,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Optional sender ID to filter by", "title": "Sender Id" }, @@ -401,14 +308,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Optional session ID to filter by", "title": "Session Id" }, @@ -420,9 +320,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/QueueStatus" - } + "schema": { "$ref": "#/components/schemas/QueueStatus" } } } }, @@ -430,9 +328,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -445,21 +341,13 @@ "summary": "Schedule Dream", "description": "Manually schedule a dream task for a specific collection.\n\nThis endpoint bypasses all automatic dream conditions (document threshold,\nminimum hours between dreams) and schedules the dream task for a future execution.\n\nCurrently this endpoint only supports scheduling immediate dreams. In the future,\nusers may pass a cron-style expression to schedule dreams at specific times.", "operationId": "schedule_dream_v3_workspaces__workspace_id__schedule_dream_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "requestBody": { @@ -474,16 +362,12 @@ } }, "responses": { - "204": { - "description": "Successful Response" - }, + "204": { "description": "Successful Response" }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -496,35 +380,20 @@ "summary": "Get Peers", "description": "Get all Peers for a Workspace, paginated with optional filters.", "operationId": "get_peers_v3_workspaces__workspace_id__peers_list_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "reverse", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Whether to reverse the order of results", "default": false, "title": "Reverse" @@ -564,12 +433,8 @@ "application/json": { "schema": { "anyOf": [ - { - "$ref": "#/components/schemas/PeerGet" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/PeerGet" }, + { "type": "null" } ], "description": "Filtering options for the peers list", "title": "Options" @@ -582,9 +447,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_Peer_" - } + "schema": { "$ref": "#/components/schemas/Page_Peer_" } } } }, @@ -592,9 +455,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -607,21 +468,13 @@ "summary": "Get Or Create Peer", "description": "Get a Peer by ID or create a new Peer with the given ID.\n\nIf peer_id is provided as a query parameter, it uses that (must match JWT workspace_id).\nOtherwise, it uses the peer_id from the JWT.", "operationId": "get_or_create_peer_v3_workspaces__workspace_id__peers_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "requestBody": { @@ -640,9 +493,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Peer" - } + "schema": { "$ref": "#/components/schemas/Peer" } } } }, @@ -650,9 +501,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -665,30 +514,19 @@ "summary": "Update Peer", "description": "Update a Peer's metadata and/or configuration.", "operationId": "update_peer_v3_workspaces__workspace_id__peers__peer_id__put", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Peer Id" - } + "schema": { "type": "string", "title": "Peer Id" } } ], "requestBody": { @@ -707,9 +545,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Peer" - } + "schema": { "$ref": "#/components/schemas/Peer" } } } }, @@ -717,9 +553,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -732,44 +566,26 @@ "summary": "Get Sessions For Peer", "description": "Get all Sessions for a Peer, paginated with optional filters.", "operationId": "get_sessions_for_peer_v3_workspaces__workspace_id__peers__peer_id__sessions_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Peer Id" - } + "schema": { "type": "string", "title": "Peer Id" } }, { "name": "reverse", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Whether to reverse the order of results", "default": false, "title": "Reverse" @@ -809,12 +625,8 @@ "application/json": { "schema": { "anyOf": [ - { - "$ref": "#/components/schemas/SessionGet" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/SessionGet" }, + { "type": "null" } ], "description": "Filtering options for the sessions list", "title": "Options" @@ -827,9 +639,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_Session_" - } + "schema": { "$ref": "#/components/schemas/Page_Session_" } } } }, @@ -837,9 +647,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -849,42 +657,29 @@ "/v3/workspaces/{workspace_id}/peers/{peer_id}/chat": { "post": { "tags": ["peers"], - "summary": "Query a Peer's representation using natural language", + "summary": "Chat", "description": "Query a Peer's representation using natural language. Performs agentic search and reasoning to comprehensively\nanswer the query based on all latent knowledge gathered about the peer from their messages and conclusions.", "operationId": "chat_v3_workspaces__workspace_id__peers__peer_id__chat_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Peer Id" - } + "schema": { "type": "string", "title": "Peer Id" } } ], "requestBody": { "required": true, "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/DialecticOptions" - } + "schema": { "$ref": "#/components/schemas/DialecticOptions" } } } }, @@ -896,14 +691,7 @@ "schema": { "properties": { "content": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Content" } }, @@ -919,9 +707,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -934,30 +720,19 @@ "summary": "Get Representation", "description": "Get a curated subset of a Peer's Representation. A Representation is always a subset of the total\nknowledge about the Peer. The subset can be scoped and filtered in various ways.\n\n\nIf a session_id is provided in the body, we get the Representation of the Peer scoped to that Session.\nIf a target is provided, we get the Representation of the target from the perspective of the Peer.\nIf no target is provided, we get the omniscient Honcho Representation of the Peer.", "operationId": "get_representation_v3_workspaces__workspace_id__peers__peer_id__representation_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Peer Id" - } + "schema": { "type": "string", "title": "Peer Id" } } ], "requestBody": { @@ -986,9 +761,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1001,21 +774,13 @@ "summary": "Get Peer Card", "description": "Get a peer card for a specific peer relationship.\n\nReturns the peer card that the observer peer has for the target peer if it exists.\nIf no target is specified, returns the observer's own peer card.", "operationId": "get_peer_card_v3_workspaces__workspace_id__peers__peer_id__card_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", @@ -1033,14 +798,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Optional target peer to retrieve a card for, from the observer's perspective. If not provided, returns the observer's own card", "title": "Target" }, @@ -1052,9 +810,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/PeerCardResponse" - } + "schema": { "$ref": "#/components/schemas/PeerCardResponse" } } } }, @@ -1062,9 +818,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1075,21 +829,13 @@ "summary": "Set Peer Card", "description": "Set a peer card for a specific peer relationship.\n\nSets the peer card that the observer peer has for the target peer.\nIf no target is specified, sets the observer's own peer card.", "operationId": "set_peer_card_v3_workspaces__workspace_id__peers__peer_id__card_put", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", @@ -1107,14 +853,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Optional target peer to set a card for, from the observer's perspective. If not provided, sets the observer's own card", "title": "Target" }, @@ -1137,9 +876,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/PeerCardResponse" - } + "schema": { "$ref": "#/components/schemas/PeerCardResponse" } } } }, @@ -1147,9 +884,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1162,21 +897,13 @@ "summary": "Get Peer Context", "description": "Get context for a peer, including their representation and peer card.\n\nThis endpoint returns a curated subset of the representation and peer card for a peer.\nIf a target is specified, returns the context for the target from the\nobserver peer's perspective. If no target is specified, returns the\npeer's own context (self-observation).\n\nThis is useful for getting all the context needed about a peer without\nmaking multiple API calls.", "operationId": "get_peer_context_v3_workspaces__workspace_id__peers__peer_id__context_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", @@ -1194,14 +921,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Optional target peer to get context for, from the observer's perspective. If not provided, returns the observer's own context (self-observation)", "title": "Target" }, @@ -1212,14 +932,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Optional query to curate the representation around semantic search results", "title": "Search Query" }, @@ -1231,14 +944,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "integer", - "maximum": 100, - "minimum": 1 - }, - { - "type": "null" - } + { "type": "integer", "maximum": 100, "minimum": 1 }, + { "type": "null" } ], "description": "Only used if `search_query` is provided. Number of semantic-search-retrieved conclusions to include", "title": "Search Top K" @@ -1251,14 +958,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "number", - "maximum": 1.0, - "minimum": 0.0 - }, - { - "type": "null" - } + { "type": "number", "maximum": 1.0, "minimum": 0.0 }, + { "type": "null" } ], "description": "Only used if `search_query` is provided. Maximum distance for semantically relevant conclusions", "title": "Search Max Distance" @@ -1283,14 +984,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "integer", - "maximum": 100, - "minimum": 1 - }, - { - "type": "null" - } + { "type": "integer", "maximum": 100, "minimum": 1 }, + { "type": "null" } ], "description": "Maximum number of conclusions to include in the representation", "title": "Max Conclusions" @@ -1303,9 +998,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/PeerContext" - } + "schema": { "$ref": "#/components/schemas/PeerContext" } } } }, @@ -1313,9 +1006,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1328,30 +1019,19 @@ "summary": "Search Peer", "description": "Search a Peer's messages, optionally filtered by various criteria.", "operationId": "search_peer_v3_workspaces__workspace_id__peers__peer_id__search_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "peer_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Peer Id" - } + "schema": { "type": "string", "title": "Peer Id" } } ], "requestBody": { @@ -1372,9 +1052,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "$ref": "#/components/schemas/Message" - }, + "items": { "$ref": "#/components/schemas/Message" }, "title": "Response Search Peer V3 Workspaces Workspace Id Peers Peer Id Search Post" } } @@ -1384,9 +1062,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1399,35 +1075,20 @@ "summary": "Get Sessions", "description": "Get all Sessions for a Workspace, paginated with optional filters.", "operationId": "get_sessions_v3_workspaces__workspace_id__sessions_list_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "reverse", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Whether to reverse the order of results", "default": false, "title": "Reverse" @@ -1467,12 +1128,8 @@ "application/json": { "schema": { "anyOf": [ - { - "$ref": "#/components/schemas/SessionGet" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/SessionGet" }, + { "type": "null" } ], "description": "Filtering and pagination options for the sessions list", "title": "Options" @@ -1485,9 +1142,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_Session_" - } + "schema": { "$ref": "#/components/schemas/Page_Session_" } } } }, @@ -1495,9 +1150,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1510,21 +1163,13 @@ "summary": "Get Or Create Session", "description": "Get a Session by ID or create a new Session with the given ID.\n\nIf Session ID is provided as a parameter, it verifies the Session is in the Workspace.\nOtherwise, it uses the session_id from the JWT for verification.", "operationId": "get_or_create_session_v3_workspaces__workspace_id__sessions_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "requestBody": { @@ -1543,9 +1188,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Session" - } + "schema": { "$ref": "#/components/schemas/Session" } } } }, @@ -1553,9 +1196,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1568,30 +1209,19 @@ "summary": "Update Session", "description": "Update a Session's metadata and/or configuration.", "operationId": "update_session_v3_workspaces__workspace_id__sessions__session_id__put", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "requestBody": { @@ -1610,9 +1240,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Session" - } + "schema": { "$ref": "#/components/schemas/Session" } } } }, @@ -1620,9 +1248,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1633,48 +1259,31 @@ "summary": "Delete Session", "description": "Delete a Session and all associated messages.\n\nThe Session is marked as inactive immediately and returns 202 Accepted. The actual\ndeletion of all related data happens asynchronously via the queue with retry support.\n\nThis action cannot be undone.", "operationId": "delete_session_v3_workspaces__workspace_id__sessions__session_id__delete", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "responses": { "202": { "description": "Successful Response", - "content": { - "application/json": { - "schema": {} - } - } + "content": { "application/json": { "schema": {} } } }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1687,44 +1296,26 @@ "summary": "Clone Session", "description": "Clone a Session, optionally up to a specific message ID.", "operationId": "clone_session_v3_workspaces__workspace_id__sessions__session_id__clone_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "message_id", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "Message ID to cut off the clone at", "title": "Message Id" }, @@ -1736,9 +1327,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Session" - } + "schema": { "$ref": "#/components/schemas/Session" } } } }, @@ -1746,9 +1335,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1761,30 +1348,19 @@ "summary": "Add Peers To Session", "description": "Add Peers to a Session. If a Peer does not yet exist, it will be created automatically.", "operationId": "add_peers_to_session_v3_workspaces__workspace_id__sessions__session_id__peers_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "requestBody": { @@ -1807,9 +1383,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Session" - } + "schema": { "$ref": "#/components/schemas/Session" } } } }, @@ -1817,9 +1391,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1830,30 +1402,19 @@ "summary": "Set Session Peers", "description": "Set the Peers in a Session. If a Peer does not yet exist, it will be created automatically.\n\nThis will fully replace the current set of Peers in the Session.", "operationId": "set_session_peers_v3_workspaces__workspace_id__sessions__session_id__peers_put", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "requestBody": { @@ -1876,9 +1437,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Session" - } + "schema": { "$ref": "#/components/schemas/Session" } } } }, @@ -1886,9 +1445,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1899,30 +1456,19 @@ "summary": "Remove Peers From Session", "description": "Remove Peers by ID from a Session.", "operationId": "remove_peers_from_session_v3_workspaces__workspace_id__sessions__session_id__peers_delete", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "requestBody": { @@ -1931,9 +1477,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "type": "string" - }, + "items": { "type": "string" }, "description": "List of peer IDs to remove from the session", "title": "Peers" } @@ -1945,9 +1489,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Session" - } + "schema": { "$ref": "#/components/schemas/Session" } } } }, @@ -1955,9 +1497,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -1968,30 +1508,19 @@ "summary": "Get Session Peers", "description": "Get all Peers in a Session. Results are paginated.", "operationId": "get_session_peers_v3_workspaces__workspace_id__sessions__session_id__peers_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "page", @@ -2026,9 +1555,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_Peer_" - } + "schema": { "$ref": "#/components/schemas/Page_Peer_" } } } }, @@ -2036,9 +1563,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2051,39 +1576,25 @@ "summary": "Get Peer Config", "description": "Get the configuration for a Peer in a Session.", "operationId": "get_peer_config_v3_workspaces__workspace_id__sessions__session_id__peers__peer_id__config_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "peer_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Peer Id" - } + "schema": { "type": "string", "title": "Peer Id" } } ], "responses": { @@ -2091,9 +1602,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/SessionPeerConfig" - } + "schema": { "$ref": "#/components/schemas/SessionPeerConfig" } } } }, @@ -2101,9 +1610,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2114,39 +1621,25 @@ "summary": "Set Peer Config", "description": "Set the configuration for a Peer in a Session.", "operationId": "set_peer_config_v3_workspaces__workspace_id__sessions__session_id__peers__peer_id__config_put", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "peer_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Peer Id" - } + "schema": { "type": "string", "title": "Peer Id" } } ], "requestBody": { @@ -2161,16 +1654,12 @@ } }, "responses": { - "204": { - "description": "Successful Response" - }, + "204": { "description": "Successful Response" }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2183,30 +1672,19 @@ "summary": "Get Session Context", "description": "Produce a context object from the Session. The caller provides an optional token limit which the entire context must fit into.\nIf not provided, the context will be exhaustive (within configured max tokens). To do this, we allocate 40% of the token limit\nto the summary, and 60% to recent messages -- as many as can fit. Note that the summary will usually take up less space than\nthis. If the caller does not want a summary, we allocate all the tokens to recent messages.", "operationId": "get_session_context_v3_workspaces__workspace_id__sessions__session_id__context_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "tokens", @@ -2214,13 +1692,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "integer", - "maximum": 100000 - }, - { - "type": "null" - } + { "type": "integer", "maximum": 100000 }, + { "type": "null" } ], "description": "Number of tokens to use for the context. Includes summary if set to true. Includes representation and peer card if they are included in the response. If not provided, the context will be exhaustive (within 100000 tokens)", "title": "Tokens" @@ -2232,14 +1705,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "A query string used to fetch semantically relevant conclusions", "title": "Search Query" }, @@ -2262,14 +1728,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "The target of the perspective. If given without `peer_perspective`, will get the Honcho-level representation and peer card for this peer. If given with `peer_perspective`, will get the representation and card for this peer *from the perspective of that peer*.", "title": "Peer Target" }, @@ -2280,14 +1739,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "A peer to get context for. If given, response will attempt to include representation and card from the perspective of that peer. Must be provided with `peer_target`.", "title": "Peer Perspective" }, @@ -2311,14 +1763,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "integer", - "maximum": 100, - "minimum": 1 - }, - { - "type": "null" - } + { "type": "integer", "maximum": 100, "minimum": 1 }, + { "type": "null" } ], "description": "Only used if `search_query` is provided. The number of semantic-search-retrieved conclusions to include in the representation", "title": "Search Top K" @@ -2331,14 +1777,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "number", - "maximum": 1.0, - "minimum": 0.0 - }, - { - "type": "null" - } + { "type": "number", "maximum": 1.0, "minimum": 0.0 }, + { "type": "null" } ], "description": "Only used if `search_query` is provided. The maximum distance to search for semantically relevant conclusions", "title": "Search Max Distance" @@ -2363,14 +1803,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "integer", - "maximum": 100, - "minimum": 1 - }, - { - "type": "null" - } + { "type": "integer", "maximum": 100, "minimum": 1 }, + { "type": "null" } ], "description": "Only used if `search_query` is provided. The maximum number of conclusions to include in the representation", "title": "Max Conclusions" @@ -2383,9 +1817,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/SessionContext" - } + "schema": { "$ref": "#/components/schemas/SessionContext" } } } }, @@ -2393,9 +1825,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2408,30 +1838,19 @@ "summary": "Get Session Summaries", "description": "Get available summaries for a Session.\n\nReturns both short and long summaries if available, including metadata like\nthe message ID they cover up to, creation timestamp, and token count.", "operationId": "get_session_summaries_v3_workspaces__workspace_id__sessions__session_id__summaries_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "responses": { @@ -2439,9 +1858,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/SessionSummaries" - } + "schema": { "$ref": "#/components/schemas/SessionSummaries" } } } }, @@ -2449,9 +1866,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2464,30 +1879,19 @@ "summary": "Search Session", "description": "Search a Session with optional filters. Use `limit` to control the number of results returned.", "operationId": "search_session_v3_workspaces__workspace_id__sessions__session_id__search_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "requestBody": { @@ -2508,9 +1912,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "$ref": "#/components/schemas/Message" - }, + "items": { "$ref": "#/components/schemas/Message" }, "title": "Response Search Session V3 Workspaces Workspace Id Sessions Session Id Search Post" } } @@ -2520,9 +1922,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2535,39 +1935,26 @@ "summary": "Create Messages For Session", "description": "Add new message(s) to a session.", "operationId": "create_messages_for_session_v3_workspaces__workspace_id__sessions__session_id__messages_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "requestBody": { "required": true, "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/MessageBatchCreate" - } + "schema": { "$ref": "#/components/schemas/MessageBatchCreate" } } } }, @@ -2578,9 +1965,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "$ref": "#/components/schemas/Message" - }, + "items": { "$ref": "#/components/schemas/Message" }, "title": "Response Create Messages For Session V3 Workspaces Workspace Id Sessions Session Id Messages Post" } } @@ -2590,9 +1975,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2605,30 +1988,19 @@ "summary": "Create Messages With File", "description": "Create messages from uploaded files. Files are converted to text and split into multiple messages.", "operationId": "create_messages_with_file_v3_workspaces__workspace_id__sessions__session_id__messages_upload_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } } ], "requestBody": { @@ -2648,9 +2020,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "$ref": "#/components/schemas/Message" - }, + "items": { "$ref": "#/components/schemas/Message" }, "title": "Response Create Messages With File V3 Workspaces Workspace Id Sessions Session Id Messages Upload Post" } } @@ -2660,9 +2030,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2675,44 +2043,26 @@ "summary": "Get Messages", "description": "Get all messages for a Session with optional filters. Results are paginated.", "operationId": "get_messages_v3_workspaces__workspace_id__sessions__session_id__messages_list_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "reverse", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "description": "Whether to reverse the order of results", "default": false, "title": "Reverse" @@ -2752,12 +2102,8 @@ "application/json": { "schema": { "anyOf": [ - { - "$ref": "#/components/schemas/MessageGet" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/MessageGet" }, + { "type": "null" } ], "description": "Filtering options for the message list", "title": "Options" @@ -2770,9 +2116,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_Message_" - } + "schema": { "$ref": "#/components/schemas/Page_Message_" } } } }, @@ -2780,9 +2124,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2795,39 +2137,25 @@ "summary": "Get Message", "description": "Get a single message by ID from a Session.", "operationId": "get_message_v3_workspaces__workspace_id__sessions__session_id__messages__message_id__get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "message_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Message Id" - } + "schema": { "type": "string", "title": "Message Id" } } ], "responses": { @@ -2835,9 +2163,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Message" - } + "schema": { "$ref": "#/components/schemas/Message" } } } }, @@ -2845,9 +2171,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2858,39 +2182,25 @@ "summary": "Update Message", "description": "Update the metadata of a message.\n\nThis will overwrite any existing metadata for the message.", "operationId": "update_message_v3_workspaces__workspace_id__sessions__session_id__messages__message_id__put", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "session_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Session Id" - } + "schema": { "type": "string", "title": "Session Id" } }, { "name": "message_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Message Id" - } + "schema": { "type": "string", "title": "Message Id" } } ], "requestBody": { @@ -2909,9 +2219,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Message" - } + "schema": { "$ref": "#/components/schemas/Message" } } } }, @@ -2919,9 +2227,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2934,21 +2240,13 @@ "summary": "Create Conclusions", "description": "Create one or more Conclusions.\n\nConclusions are logical certainties derived from interactions between Peers. They form the basis of a Peer's Representation.", "operationId": "create_conclusions_v3_workspaces__workspace_id__conclusions_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "requestBody": { @@ -2969,9 +2267,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "$ref": "#/components/schemas/Conclusion" - }, + "items": { "$ref": "#/components/schemas/Conclusion" }, "title": "Response Create Conclusions V3 Workspaces Workspace Id Conclusions Post" } } @@ -2981,9 +2277,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -2996,35 +2290,20 @@ "summary": "List Conclusions", "description": "List Conclusions using optional filters, ordered by recency unless `reverse` is true. Results are paginated.", "operationId": "list_conclusions_v3_workspaces__workspace_id__conclusions_list_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "reverse", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "description": "Whether to reverse the order of results", "default": false, "title": "Reverse" @@ -3064,12 +2343,8 @@ "application/json": { "schema": { "anyOf": [ - { - "$ref": "#/components/schemas/ConclusionGet" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/ConclusionGet" }, + { "type": "null" } ], "description": "Filtering options for the Conclusions list", "title": "Options" @@ -3082,9 +2357,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/Page_Conclusion_" - } + "schema": { "$ref": "#/components/schemas/Page_Conclusion_" } } } }, @@ -3092,9 +2365,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -3107,21 +2378,13 @@ "summary": "Query Conclusions", "description": "Query Conclusions using semantic search. Use `top_k` to control the number of results returned.", "operationId": "query_conclusions_v3_workspaces__workspace_id__conclusions_query_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } } ], "requestBody": { @@ -3142,9 +2405,7 @@ "application/json": { "schema": { "type": "array", - "items": { - "$ref": "#/components/schemas/Conclusion" - }, + "items": { "$ref": "#/components/schemas/Conclusion" }, "title": "Response Query Conclusions V3 Workspaces Workspace Id Conclusions Query Post" } } @@ -3154,9 +2415,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -3169,43 +2428,28 @@ "summary": "Delete Conclusion", "description": "Delete a single Conclusion by ID.\n\nThis action cannot be undone.", "operationId": "delete_conclusion_v3_workspaces__workspace_id__conclusions__conclusion_id__delete", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Workspace Id" - } + "schema": { "type": "string", "title": "Workspace Id" } }, { "name": "conclusion_id", "in": "path", "required": true, - "schema": { - "type": "string", - "title": "Conclusion Id" - } + "schema": { "type": "string", "title": "Conclusion Id" } } ], "responses": { - "204": { - "description": "Successful Response" - }, + "204": { "description": "Successful Response" }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -3218,26 +2462,14 @@ "summary": "Create Key", "description": "Create a new Key", "operationId": "create_key_v3_keys_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "ID of the workspace to scope the key to", "title": "Workspace Id" }, @@ -3248,14 +2480,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "ID of the peer to scope the key to", "title": "Peer Id" }, @@ -3266,14 +2491,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "description": "ID of the session to scope the key to", "title": "Session Id" }, @@ -3285,13 +2503,8 @@ "required": false, "schema": { "anyOf": [ - { - "type": "string", - "format": "date-time" - }, - { - "type": "null" - } + { "type": "string", "format": "date-time" }, + { "type": "null" } ], "title": "Expires At" } @@ -3300,19 +2513,13 @@ "responses": { "200": { "description": "Successful Response", - "content": { - "application/json": { - "schema": {} - } - } + "content": { "application/json": { "schema": {} } } }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -3325,12 +2532,7 @@ "summary": "Get Or Create Webhook Endpoint", "description": "Get or create a webhook endpoint URL.", "operationId": "get_or_create_webhook_endpoint_v3_workspaces__workspace_id__webhooks_post", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", @@ -3360,9 +2562,7 @@ "description": "Successful Response", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/WebhookEndpoint" - } + "schema": { "$ref": "#/components/schemas/WebhookEndpoint" } } } }, @@ -3370,9 +2570,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -3383,12 +2581,7 @@ "summary": "List Webhook Endpoints", "description": "List all webhook endpoints, optionally filtered by workspace.", "operationId": "list_webhook_endpoints_v3_workspaces__workspace_id__webhooks_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", @@ -3444,9 +2637,7 @@ "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -3459,12 +2650,7 @@ "summary": "Delete Webhook Endpoint", "description": "Delete a specific webhook endpoint.", "operationId": "delete_webhook_endpoint_v3_workspaces__workspace_id__webhooks__endpoint_id__delete", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", @@ -3490,16 +2676,12 @@ } ], "responses": { - "204": { - "description": "Successful Response" - }, + "204": { "description": "Successful Response" }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } @@ -3512,12 +2694,7 @@ "summary": "Test Emit", "description": "Test publishing a webhook event.", "operationId": "test_emit_v3_workspaces__workspace_id__webhooks_test_get", - "security": [ - { - "HTTPBearer": [] - }, - {} - ], + "security": [{ "HTTPBearer": [] }], "parameters": [ { "name": "workspace_id", @@ -3534,24 +2711,31 @@ "responses": { "200": { "description": "Successful Response", - "content": { - "application/json": { - "schema": {} - } - } + "content": { "application/json": { "schema": {} } } }, "422": { "description": "Validation Error", "content": { "application/json": { - "schema": { - "$ref": "#/components/schemas/HTTPValidationError" - } + "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } } } + }, + "/health": { + "get": { + "summary": "Health Check", + "description": "Health check endpoint for monitoring and container orchestration.", + "operationId": "health_check_health_get", + "responses": { + "200": { + "description": "Successful Response", + "content": { "application/json": { "schema": {} } } + } + } + } } }, "components": { @@ -3563,41 +2747,17 @@ "contentMediaType": "application/octet-stream", "title": "File" }, - "peer_id": { - "type": "string", - "title": "Peer Id" - }, + "peer_id": { "type": "string", "title": "Peer Id" }, "metadata": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Metadata" }, "configuration": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Configuration" }, "created_at": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Created At" } }, @@ -3607,14 +2767,8 @@ }, "Conclusion": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, - "content": { - "type": "string", - "title": "Content" - }, + "id": { "type": "string", "title": "Id" }, + "content": { "type": "string", "title": "Content" }, "observer_id": { "type": "string", "title": "Observer Id", @@ -3626,14 +2780,7 @@ "description": "The peer the conclusion is about" }, "session_id": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Session Id" }, "created_at": { @@ -3656,9 +2803,7 @@ "ConclusionBatchCreate": { "properties": { "conclusions": { - "items": { - "$ref": "#/components/schemas/ConclusionCreate" - }, + "items": { "$ref": "#/components/schemas/ConclusionCreate" }, "type": "array", "maxItems": 100, "minItems": 1, @@ -3689,14 +2834,7 @@ "description": "The peer the conclusion is about" }, "session_id": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Session Id", "description": "A session ID to store the conclusion in, if specified" } @@ -3710,13 +2848,8 @@ "properties": { "filters": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Filters" } @@ -3742,27 +2875,16 @@ }, "distance": { "anyOf": [ - { - "type": "number", - "maximum": 1.0, - "minimum": 0.0 - }, - { - "type": "null" - } + { "type": "number", "maximum": 1.0, "minimum": 0.0 }, + { "type": "null" } ], "title": "Distance", "description": "Maximum cosine distance threshold for results" }, "filters": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Filters", "description": "Additional filters to apply" @@ -3776,26 +2898,12 @@ "DialecticOptions": { "properties": { "session_id": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Session Id", "description": "ID of the session to scope the representation to" }, "target": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Target", "description": "Optional peer to get the representation for, from the perspective of this peer" }, @@ -3806,11 +2914,7 @@ "title": "Query", "description": "Dialectic API Prompt" }, - "stream": { - "type": "boolean", - "title": "Stream", - "default": false - }, + "stream": { "type": "boolean", "title": "Stream", "default": false }, "reasoning_level": { "type": "string", "enum": ["minimal", "low", "medium", "high", "max"], @@ -3826,14 +2930,7 @@ "DreamConfiguration": { "properties": { "enabled": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Enabled", "description": "Whether to enable dream functionality. If reasoning is disabled, dreams will also be disabled and this setting will be ignored." } @@ -3850,9 +2947,7 @@ "HTTPValidationError": { "properties": { "detail": { - "items": { - "$ref": "#/components/schemas/ValidationError" - }, + "items": { "$ref": "#/components/schemas/ValidationError" }, "type": "array", "title": "Detail" } @@ -3862,22 +2957,10 @@ }, "Message": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, - "content": { - "type": "string", - "title": "Content" - }, - "peer_id": { - "type": "string", - "title": "Peer Id" - }, - "session_id": { - "type": "string", - "title": "Session Id" - }, + "id": { "type": "string", "title": "Id" }, + "content": { "type": "string", "title": "Content" }, + "peer_id": { "type": "string", "title": "Peer Id" }, + "session_id": { "type": "string", "title": "Session Id" }, "metadata": { "additionalProperties": true, "type": "object", @@ -3888,14 +2971,8 @@ "format": "date-time", "title": "Created At" }, - "workspace_id": { - "type": "string", - "title": "Workspace Id" - }, - "token_count": { - "type": "integer", - "title": "Token Count" - } + "workspace_id": { "type": "string", "title": "Workspace Id" }, + "token_count": { "type": "integer", "title": "Token Count" } }, "type": "object", "required": [ @@ -3912,9 +2989,7 @@ "MessageBatchCreate": { "properties": { "messages": { - "items": { - "$ref": "#/components/schemas/MessageCreate" - }, + "items": { "$ref": "#/components/schemas/MessageCreate" }, "type": "array", "maxItems": 100, "minItems": 1, @@ -3930,12 +3005,8 @@ "properties": { "reasoning": { "anyOf": [ - { - "$ref": "#/components/schemas/ReasoningConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/ReasoningConfiguration" }, + { "type": "null" } ], "description": "Configuration for reasoning functionality." } @@ -3952,41 +3023,24 @@ "minLength": 0, "title": "Content" }, - "peer_id": { - "type": "string", - "title": "Peer Id" - }, + "peer_id": { "type": "string", "title": "Peer Id" }, "metadata": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Metadata" }, "configuration": { "anyOf": [ - { - "$ref": "#/components/schemas/MessageConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/MessageConfiguration" }, + { "type": "null" } ] }, "created_at": { "anyOf": [ - { - "type": "string", - "format": "date-time" - }, - { - "type": "null" - } + { "type": "string", "format": "date-time" }, + { "type": "null" } ], "title": "Created At" } @@ -3999,13 +3053,8 @@ "properties": { "filters": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Filters" } @@ -4022,13 +3071,8 @@ }, "filters": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Filters", "description": "Filters to scope the search" @@ -4050,13 +3094,8 @@ "properties": { "metadata": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Metadata" } @@ -4067,32 +3106,14 @@ "Page_Conclusion_": { "properties": { "items": { - "items": { - "$ref": "#/components/schemas/Conclusion" - }, + "items": { "$ref": "#/components/schemas/Conclusion" }, "type": "array", "title": "Items" }, - "total": { - "type": "integer", - "minimum": 0.0, - "title": "Total" - }, - "page": { - "type": "integer", - "minimum": 1.0, - "title": "Page" - }, - "size": { - "type": "integer", - "minimum": 1.0, - "title": "Size" - }, - "pages": { - "type": "integer", - "minimum": 0.0, - "title": "Pages" - } + "total": { "type": "integer", "minimum": 0.0, "title": "Total" }, + "page": { "type": "integer", "minimum": 1.0, "title": "Page" }, + "size": { "type": "integer", "minimum": 1.0, "title": "Size" }, + "pages": { "type": "integer", "minimum": 0.0, "title": "Pages" } }, "type": "object", "required": ["items", "total", "page", "size", "pages"], @@ -4101,32 +3122,14 @@ "Page_Message_": { "properties": { "items": { - "items": { - "$ref": "#/components/schemas/Message" - }, + "items": { "$ref": "#/components/schemas/Message" }, "type": "array", "title": "Items" }, - "total": { - "type": "integer", - "minimum": 0.0, - "title": "Total" - }, - "page": { - "type": "integer", - "minimum": 1.0, - "title": "Page" - }, - "size": { - "type": "integer", - "minimum": 1.0, - "title": "Size" - }, - "pages": { - "type": "integer", - "minimum": 0.0, - "title": "Pages" - } + "total": { "type": "integer", "minimum": 0.0, "title": "Total" }, + "page": { "type": "integer", "minimum": 1.0, "title": "Page" }, + "size": { "type": "integer", "minimum": 1.0, "title": "Size" }, + "pages": { "type": "integer", "minimum": 0.0, "title": "Pages" } }, "type": "object", "required": ["items", "total", "page", "size", "pages"], @@ -4135,32 +3138,14 @@ "Page_Peer_": { "properties": { "items": { - "items": { - "$ref": "#/components/schemas/Peer" - }, + "items": { "$ref": "#/components/schemas/Peer" }, "type": "array", "title": "Items" }, - "total": { - "type": "integer", - "minimum": 0.0, - "title": "Total" - }, - "page": { - "type": "integer", - "minimum": 1.0, - "title": "Page" - }, - "size": { - "type": "integer", - "minimum": 1.0, - "title": "Size" - }, - "pages": { - "type": "integer", - "minimum": 0.0, - "title": "Pages" - } + "total": { "type": "integer", "minimum": 0.0, "title": "Total" }, + "page": { "type": "integer", "minimum": 1.0, "title": "Page" }, + "size": { "type": "integer", "minimum": 1.0, "title": "Size" }, + "pages": { "type": "integer", "minimum": 0.0, "title": "Pages" } }, "type": "object", "required": ["items", "total", "page", "size", "pages"], @@ -4169,32 +3154,14 @@ "Page_Session_": { "properties": { "items": { - "items": { - "$ref": "#/components/schemas/Session" - }, + "items": { "$ref": "#/components/schemas/Session" }, "type": "array", "title": "Items" }, - "total": { - "type": "integer", - "minimum": 0.0, - "title": "Total" - }, - "page": { - "type": "integer", - "minimum": 1.0, - "title": "Page" - }, - "size": { - "type": "integer", - "minimum": 1.0, - "title": "Size" - }, - "pages": { - "type": "integer", - "minimum": 0.0, - "title": "Pages" - } + "total": { "type": "integer", "minimum": 0.0, "title": "Total" }, + "page": { "type": "integer", "minimum": 1.0, "title": "Page" }, + "size": { "type": "integer", "minimum": 1.0, "title": "Size" }, + "pages": { "type": "integer", "minimum": 0.0, "title": "Pages" } }, "type": "object", "required": ["items", "total", "page", "size", "pages"], @@ -4203,32 +3170,14 @@ "Page_WebhookEndpoint_": { "properties": { "items": { - "items": { - "$ref": "#/components/schemas/WebhookEndpoint" - }, + "items": { "$ref": "#/components/schemas/WebhookEndpoint" }, "type": "array", "title": "Items" }, - "total": { - "type": "integer", - "minimum": 0.0, - "title": "Total" - }, - "page": { - "type": "integer", - "minimum": 1.0, - "title": "Page" - }, - "size": { - "type": "integer", - "minimum": 1.0, - "title": "Size" - }, - "pages": { - "type": "integer", - "minimum": 0.0, - "title": "Pages" - } + "total": { "type": "integer", "minimum": 0.0, "title": "Total" }, + "page": { "type": "integer", "minimum": 1.0, "title": "Page" }, + "size": { "type": "integer", "minimum": 1.0, "title": "Size" }, + "pages": { "type": "integer", "minimum": 0.0, "title": "Pages" } }, "type": "object", "required": ["items", "total", "page", "size", "pages"], @@ -4237,32 +3186,14 @@ "Page_Workspace_": { "properties": { "items": { - "items": { - "$ref": "#/components/schemas/Workspace" - }, + "items": { "$ref": "#/components/schemas/Workspace" }, "type": "array", "title": "Items" }, - "total": { - "type": "integer", - "minimum": 0.0, - "title": "Total" - }, - "page": { - "type": "integer", - "minimum": 1.0, - "title": "Page" - }, - "size": { - "type": "integer", - "minimum": 1.0, - "title": "Size" - }, - "pages": { - "type": "integer", - "minimum": 0.0, - "title": "Pages" - } + "total": { "type": "integer", "minimum": 0.0, "title": "Total" }, + "page": { "type": "integer", "minimum": 1.0, "title": "Page" }, + "size": { "type": "integer", "minimum": 1.0, "title": "Size" }, + "pages": { "type": "integer", "minimum": 0.0, "title": "Pages" } }, "type": "object", "required": ["items", "total", "page", "size", "pages"], @@ -4270,14 +3201,8 @@ }, "Peer": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, - "workspace_id": { - "type": "string", - "title": "Workspace Id" - }, + "id": { "type": "string", "title": "Id" }, + "workspace_id": { "type": "string", "title": "Workspace Id" }, "created_at": { "type": "string", "format": "date-time", @@ -4301,26 +3226,12 @@ "PeerCardConfiguration": { "properties": { "use": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Use", "description": "Whether to use peer card related to this peer during reasoning process." }, "create": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Create", "description": "Whether to generate peer card based on content." } @@ -4332,15 +3243,8 @@ "properties": { "peer_card": { "anyOf": [ - { - "items": { - "type": "string" - }, - "type": "array" - }, - { - "type": "null" - } + { "items": { "type": "string" }, "type": "array" }, + { "type": "null" } ], "title": "Peer Card", "description": "The peer card content, or None if not found" @@ -4352,9 +3256,7 @@ "PeerCardSet": { "properties": { "peer_card": { - "items": { - "type": "string" - }, + "items": { "type": "string" }, "type": "array", "title": "Peer Card", "description": "The peer card content to set" @@ -4377,28 +3279,14 @@ "description": "The ID of the target peer being observed" }, "representation": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Representation", "description": "A curated subset of the representation of the target peer from the observer's perspective" }, "peer_card": { "anyOf": [ - { - "items": { - "type": "string" - }, - "type": "array" - }, - { - "type": "null" - } + { "items": { "type": "string" }, "type": "array" }, + { "type": "null" } ], "title": "Peer Card", "description": "The peer card for the target peer from the observer's perspective" @@ -4413,32 +3301,22 @@ "properties": { "id": { "type": "string", - "maxLength": 100, + "maxLength": 512, "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$", "title": "Id" }, "metadata": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Metadata" }, "configuration": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Configuration" } @@ -4451,13 +3329,8 @@ "properties": { "filters": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Filters" } @@ -4468,91 +3341,45 @@ "PeerRepresentationGet": { "properties": { "session_id": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Session Id", "description": "Optional session ID within which to scope the representation" }, "target": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Target", "description": "Optional peer ID to get the representation for, from the perspective of this peer" }, "search_query": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Search Query", "description": "Optional input to curate the representation around semantic search results" }, "search_top_k": { "anyOf": [ - { - "type": "integer", - "maximum": 100.0, - "minimum": 1.0 - }, - { - "type": "null" - } + { "type": "integer", "maximum": 100.0, "minimum": 1.0 }, + { "type": "null" } ], "title": "Search Top K", "description": "Only used if `search_query` is provided. Number of semantic-search-retrieved conclusions to include in the representation" }, "search_max_distance": { "anyOf": [ - { - "type": "number", - "maximum": 1.0, - "minimum": 0.0 - }, - { - "type": "null" - } + { "type": "number", "maximum": 1.0, "minimum": 0.0 }, + { "type": "null" } ], "title": "Search Max Distance", "description": "Only used if `search_query` is provided. Maximum distance to search for semantically relevant conclusions" }, "include_most_frequent": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Include Most Frequent", "description": "Only used if `search_query` is provided. Whether to include the most frequent conclusions in the representation" }, "max_conclusions": { "anyOf": [ - { - "type": "integer", - "maximum": 100.0, - "minimum": 1.0 - }, - { - "type": "null" - } + { "type": "integer", "maximum": 100.0, "minimum": 1.0 }, + { "type": "null" } ], "title": "Max Conclusions", "description": "Only used if `search_query` is provided. Maximum number of conclusions to include in the representation", @@ -4566,25 +3393,15 @@ "properties": { "metadata": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Metadata" }, "configuration": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Configuration" } @@ -4622,9 +3439,7 @@ }, "type": "object" }, - { - "type": "null" - } + { "type": "null" } ], "title": "Sessions", "description": "Per-session status when not filtered by session" @@ -4643,28 +3458,14 @@ "ReasoningConfiguration": { "properties": { "enabled": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Enabled", "description": "Whether to enable reasoning functionality." }, "custom_instructions": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Custom Instructions", - "description": "TODO: currently unused. Custom instructions to use for the reasoning system on this workspace/session/message." + "description": "Optional custom instructions for the reasoning system on this workspace/session/message. Rejected if they exceed the deriver custom-instruction token cap." } }, "type": "object", @@ -4672,10 +3473,7 @@ }, "RepresentationResponse": { "properties": { - "representation": { - "type": "string", - "title": "Representation" - } + "representation": { "type": "string", "title": "Representation" } }, "type": "object", "required": ["representation"], @@ -4689,14 +3487,7 @@ "description": "Observer peer name" }, "observed": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Observed", "description": "Observed peer name (defaults to observer if not specified)" }, @@ -4705,14 +3496,7 @@ "description": "Type of dream to schedule" }, "session_id": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Session Id", "description": "Session ID to scope the dream to if specified" } @@ -4723,18 +3507,9 @@ }, "Session": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, - "is_active": { - "type": "boolean", - "title": "Is Active" - }, - "workspace_id": { - "type": "string", - "title": "Workspace Id" - }, + "id": { "type": "string", "title": "Id" }, + "is_active": { "type": "boolean", "title": "Is Active" }, + "workspace_id": { "type": "string", "title": "Workspace Id" }, "metadata": { "additionalProperties": true, "type": "object", @@ -4759,45 +3534,29 @@ "properties": { "reasoning": { "anyOf": [ - { - "$ref": "#/components/schemas/ReasoningConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/ReasoningConfiguration" }, + { "type": "null" } ], "description": "Configuration for reasoning functionality." }, "peer_card": { "anyOf": [ - { - "$ref": "#/components/schemas/PeerCardConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/PeerCardConfiguration" }, + { "type": "null" } ], "description": "Configuration for peer card functionality. If reasoning is disabled, peer cards will also be disabled and these settings will be ignored." }, "summary": { "anyOf": [ - { - "$ref": "#/components/schemas/SummaryConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/SummaryConfiguration" }, + { "type": "null" } ], "description": "Configuration for summary functionality." }, "dream": { "anyOf": [ - { - "$ref": "#/components/schemas/DreamConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/DreamConfiguration" }, + { "type": "null" } ], "description": "Configuration for dream functionality. If reasoning is disabled, dreams will also be disabled and these settings will be ignored." } @@ -4809,51 +3568,28 @@ }, "SessionContext": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, + "id": { "type": "string", "title": "Id" }, "messages": { - "items": { - "$ref": "#/components/schemas/Message" - }, + "items": { "$ref": "#/components/schemas/Message" }, "type": "array", "title": "Messages" }, "summary": { "anyOf": [ - { - "$ref": "#/components/schemas/Summary" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/Summary" }, + { "type": "null" } ], "description": "The summary if available" }, "peer_representation": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Peer Representation", "description": "A curated subset of a peer representation, if context is requested from a specific perspective" }, "peer_card": { "anyOf": [ - { - "items": { - "type": "string" - }, - "type": "array" - }, - { - "type": "null" - } + { "items": { "type": "string" }, "type": "array" }, + { "type": "null" } ], "title": "Peer Card", "description": "The peer card, if context is requested from a specific perspective" @@ -4867,20 +3603,15 @@ "properties": { "id": { "type": "string", - "maxLength": 100, + "maxLength": 512, "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$", "title": "Id" }, "metadata": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Metadata" }, @@ -4892,20 +3623,14 @@ }, "type": "object" }, - { - "type": "null" - } + { "type": "null" } ], "title": "Peers" }, "configuration": { "anyOf": [ - { - "$ref": "#/components/schemas/SessionConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/SessionConfiguration" }, + { "type": "null" } ] } }, @@ -4917,13 +3642,8 @@ "properties": { "filters": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Filters" } @@ -4934,26 +3654,12 @@ "SessionPeerConfig": { "properties": { "observe_me": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Observe Me", "description": "Whether Honcho will use reasoning to form a representation of this peer" }, "observe_others": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Observe Others", "description": "Whether this peer should form a session-level theory-of-mind representation of other peers in the session" } @@ -4964,14 +3670,7 @@ "SessionQueueStatus": { "properties": { "session_id": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Session Id", "description": "Session ID if filtered by session" }, @@ -5008,29 +3707,18 @@ }, "SessionSummaries": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, + "id": { "type": "string", "title": "Id" }, "short_summary": { "anyOf": [ - { - "$ref": "#/components/schemas/Summary" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/Summary" }, + { "type": "null" } ], "description": "The short summary if available" }, "long_summary": { "anyOf": [ - { - "$ref": "#/components/schemas/Summary" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/Summary" }, + { "type": "null" } ], "description": "The long summary if available" } @@ -5043,24 +3731,15 @@ "properties": { "metadata": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Metadata" }, "configuration": { "anyOf": [ - { - "$ref": "#/components/schemas/SessionConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/SessionConfiguration" }, + { "type": "null" } ] } }, @@ -5108,39 +3787,22 @@ "SummaryConfiguration": { "properties": { "enabled": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "boolean" }, { "type": "null" }], "title": "Enabled", "description": "Whether to enable summary functionality." }, "messages_per_short_summary": { "anyOf": [ - { - "type": "integer", - "minimum": 10.0 - }, - { - "type": "null" - } + { "type": "integer", "minimum": 10.0 }, + { "type": "null" } ], "title": "Messages Per Short Summary", "description": "Number of messages per short summary. Must be positive, greater than or equal to 10, and less than messages_per_long_summary." }, "messages_per_long_summary": { "anyOf": [ - { - "type": "integer", - "minimum": 20.0 - }, - { - "type": "null" - } + { "type": "integer", "minimum": 20.0 }, + { "type": "null" } ], "title": "Messages Per Long Summary", "description": "Number of messages per long summary. Must be positive, greater than or equal to 20, and greater than messages_per_short_summary." @@ -5152,34 +3814,14 @@ "ValidationError": { "properties": { "loc": { - "items": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "integer" - } - ] - }, + "items": { "anyOf": [{ "type": "string" }, { "type": "integer" }] }, "type": "array", "title": "Location" }, - "msg": { - "type": "string", - "title": "Message" - }, - "type": { - "type": "string", - "title": "Error Type" - }, - "input": { - "title": "Input" - }, - "ctx": { - "type": "object", - "title": "Context" - } + "msg": { "type": "string", "title": "Message" }, + "type": { "type": "string", "title": "Error Type" }, + "input": { "title": "Input" }, + "ctx": { "type": "object", "title": "Context" } }, "type": "object", "required": ["loc", "msg", "type"], @@ -5187,25 +3829,12 @@ }, "WebhookEndpoint": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, + "id": { "type": "string", "title": "Id" }, "workspace_id": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], + "anyOf": [{ "type": "string" }, { "type": "null" }], "title": "Workspace Id" }, - "url": { - "type": "string", - "title": "Url" - }, + "url": { "type": "string", "title": "Url" }, "created_at": { "type": "string", "format": "date-time", @@ -5217,22 +3846,14 @@ "title": "WebhookEndpoint" }, "WebhookEndpointCreate": { - "properties": { - "url": { - "type": "string", - "title": "Url" - } - }, + "properties": { "url": { "type": "string", "title": "Url" } }, "type": "object", "required": ["url"], "title": "WebhookEndpointCreate" }, "Workspace": { "properties": { - "id": { - "type": "string", - "title": "Id" - }, + "id": { "type": "string", "title": "Id" }, "metadata": { "additionalProperties": true, "type": "object", @@ -5257,45 +3878,29 @@ "properties": { "reasoning": { "anyOf": [ - { - "$ref": "#/components/schemas/ReasoningConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/ReasoningConfiguration" }, + { "type": "null" } ], "description": "Configuration for reasoning functionality." }, "peer_card": { "anyOf": [ - { - "$ref": "#/components/schemas/PeerCardConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/PeerCardConfiguration" }, + { "type": "null" } ], "description": "Configuration for peer card functionality. If reasoning is disabled, peer cards will also be disabled and these settings will be ignored." }, "summary": { "anyOf": [ - { - "$ref": "#/components/schemas/SummaryConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/SummaryConfiguration" }, + { "type": "null" } ], "description": "Configuration for summary functionality." }, "dream": { "anyOf": [ - { - "$ref": "#/components/schemas/DreamConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/DreamConfiguration" }, + { "type": "null" } ], "description": "Configuration for dream functionality. If reasoning is disabled, dreams will also be disabled and these settings will be ignored." } @@ -5309,7 +3914,7 @@ "properties": { "id": { "type": "string", - "maxLength": 100, + "maxLength": 512, "minLength": 1, "pattern": "^[a-zA-Z0-9_-]+$", "title": "Id" @@ -5332,13 +3937,8 @@ "properties": { "filters": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Filters" } @@ -5350,24 +3950,15 @@ "properties": { "metadata": { "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "type": "null" - } + { "additionalProperties": true, "type": "object" }, + { "type": "null" } ], "title": "Metadata" }, "configuration": { "anyOf": [ - { - "$ref": "#/components/schemas/WorkspaceConfiguration" - }, - { - "type": "null" - } + { "$ref": "#/components/schemas/WorkspaceConfiguration" }, + { "type": "null" } ] } }, @@ -5375,11 +3966,6 @@ "title": "WorkspaceUpdate" } }, - "securitySchemes": { - "HTTPBearer": { - "type": "http", - "scheme": "bearer" - } - } + "securitySchemes": { "HTTPBearer": { "type": "http", "scheme": "bearer" } } } } diff --git a/pyproject.toml b/pyproject.toml index a2913ea5..31581efb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "honcho" -version = "3.0.6" +version = "3.0.7" description = "Honcho Server" authors = [ {name = "Plastic Labs", email = "hello@plasticlabs.ai"}, diff --git a/sdks/python/CHANGELOG.md b/sdks/python/CHANGELOG.md index a7056000..2c8d3f91 100644 --- a/sdks/python/CHANGELOG.md +++ b/sdks/python/CHANGELOG.md @@ -5,11 +5,16 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). -## [Unreleased] +## [2.1.2] - 2026-05-21 ### Added - `page`, `size`, and `reverse` pagination parameters on `Honcho.workspaces()` and `HonchoAio.workspaces()`, closing the gap from 2.1.0 which added these to `peers()`, `sessions()`, `messages()`, and `conclusions.list()` but not to `workspaces()`. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter. +- `peers` parameter on `Honcho.session()` and `HonchoAio.session()` — attach peers to a session at creation time instead of needing a follow-up `session.add_peers()` call. Accepts the same shapes as `Session.add_peers` (peer ID string, `Peer` object, list of either, or tuples with `SessionPeerConfig`). + +### Changed + +- `WorkspaceCreateParams`, `PeerCreateParams`, and `SessionCreateParams` now accept IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7. ## [2.1.1] - 2026-04-01 diff --git a/sdks/python/pyproject.toml b/sdks/python/pyproject.toml index 07b532f8..0e6ad9ba 100644 --- a/sdks/python/pyproject.toml +++ b/sdks/python/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "honcho-ai" -version = "2.1.1" +version = "2.1.2" description = "Official DX Optimized Python SDK for Honcho" dynamic = ["readme"] license = "Apache-2.0" diff --git a/sdks/typescript/CHANGELOG.md b/sdks/typescript/CHANGELOG.md index e5ab73ac..bd6dd844 100644 --- a/sdks/typescript/CHANGELOG.md +++ b/sdks/typescript/CHANGELOG.md @@ -5,7 +5,15 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). -## [Unreleased] +## [2.1.2] - 2026-05-21 + +### Added + +- `peers` option on `Honcho.session()` — attach peers to a session at creation time instead of needing a follow-up `session.addPeers()` call. Accepts the same `PeerAddition` shape as `session.addPeers()` (peer ID strings, `Peer` objects, arrays of either, or a record with per-peer `observe_me`/`observe_others` config). + +### Changed + +- ID validation in `validation.ts` now accepts workspace, peer, and session IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7. ### Fixed diff --git a/sdks/typescript/package.json b/sdks/typescript/package.json index 136b716a..4ac6cea4 100644 --- a/sdks/typescript/package.json +++ b/sdks/typescript/package.json @@ -1,6 +1,6 @@ { "name": "@honcho-ai/sdk", - "version": "2.1.1", + "version": "2.1.2", "description": "Official DX Optimized TypeScript SDK for Honcho", "author": "Plastic Labs ", "license": "Apache-2.0", diff --git a/uv.lock b/uv.lock index 75a59f80..d577957f 100644 --- a/uv.lock +++ b/uv.lock @@ -8,7 +8,7 @@ resolution-markers = [ ] [options] -exclude-newer = "2026-05-09T19:09:35.818254Z" +exclude-newer = "2026-05-16T17:58:57.678125Z" exclude-newer-span = "P5D" [manifest] @@ -1159,7 +1159,7 @@ wheels = [ [[package]] name = "honcho" -version = "3.0.6" +version = "3.0.7" source = { virtual = "." } dependencies = [ { name = "alembic" }, @@ -1270,7 +1270,7 @@ dev = [ [[package]] name = "honcho-ai" -version = "2.1.1" +version = "2.1.2" source = { editable = "sdks/python" } dependencies = [ { name = "httpx" }, From 85239a69b262c944de3c35900b91c88ba9b84f1a Mon Sep 17 00:00:00 2001 From: ajspig <46900795+ajspig@users.noreply.github.com> Date: Wed, 27 May 2026 15:25:55 -0400 Subject: [PATCH 7/7] Updating Design Patterns (#717) * docs: draft of design-patterns * fix: minor language changes * docs: adding unified memory guide * docs: simplifying design patterns * docs: minor edits * docs: language clarification * docs: simplification of intro --- docs/docs.json | 1 + .../core-concepts/design-patterns.mdx | 357 ++---------------- docs/v3/guides/overview.mdx | 9 + .../guides/recipes/unified-memory-setup.mdx | 186 +++++++++ 4 files changed, 237 insertions(+), 316 deletions(-) create mode 100644 docs/v3/guides/recipes/unified-memory-setup.mdx diff --git a/docs/docs.json b/docs/docs.json index 5ccb4166..64b56d86 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -110,6 +110,7 @@ { "group": "Tutorials", "pages": [ + "v3/guides/recipes/unified-memory-setup", "v3/guides/discord", "v3/guides/granola", "v3/guides/telegram", diff --git a/docs/v3/documentation/core-concepts/design-patterns.mdx b/docs/v3/documentation/core-concepts/design-patterns.mdx index 0e35114e..06582d80 100644 --- a/docs/v3/documentation/core-concepts/design-patterns.mdx +++ b/docs/v3/documentation/core-concepts/design-patterns.mdx @@ -5,364 +5,92 @@ icon: "cubes" --- -If you're using a coding agent (Claude Code, OpenCode, Cursor, etc.), the **`/honcho-integration` skill** walks you through these decisions interactively. It explores your codebase, interviews you about peers and sessions, and generates the integration code. The patterns below are the same ones the skill uses. +This page covers **how to structure** workspaces, peers, and sessions for real applications. For the conceptual model behind them, start with [Architecture](/v3/documentation/core-concepts/architecture). + +Ready to add Honcho to your codebase? The **`/honcho-integration` skill** applies these patterns for you — it explores your code, asks how your peers and sessions should map to your app, and wires in the Honcho SDK. Run it in any coding agent that supports skills (Claude Code, Cursor, and others). ## Quick Reference +**Workspaces isolate, peers persist, and sessions scope the active context.** + | Decision | Recommendation | |----------|---------------| -| How many workspaces? | One per application. Separate per-agent if you need hard data isolation. | -| Who should be a peer? | Any entity you want Honcho to reason about — users, agents, NPCs, students, customers. | -| How should I scope sessions? | Flexible -- per-conversation, per-channel, per-scene, etc. See [Session Design](#session-design) below. | -| Should I set `observe_me: false`? | Yes, for any peer you don't need Honcho to build a representation of — typically assistants or bots with deterministic behavior. | -| Do I need `observe_others`? | Only when different peers need distinct views of the same participant (e.g., games, multi-agent). Most apps can leave it at the default (false). | +| How many workspaces? | One workspace per application, tool, tenant, or collaboration boundary. Split workspaces only when you need hard isolation between products, customers, environments, or agents. | +| When should agents share a workspace? | When agents collaborate over the same product, project, team, user, customer, or game state. Separate them when they should not see or influence each other's memory. | +| Who should be a peer? | Any persistent participant whose messages should be attributed or reasoned about: users, agents, assistants, NPCs, students, or customers. Use one peer for the same entity across sessions and platforms. | +| How should I scope sessions? | Scope sessions to the active interaction: per-conversation, per-channel, per-task run, per-project, per-import, or other bounded context. Reuse a session when local context should keep accumulating. | +| How does cross-session reasoning work? | Session memory stays local to one session. Peer representations accumulate across every session where the peer is included, and `session.context()` becomes cross-session when you include a peer target. | +| Should I set `observe_me: false`? | Yes, for deterministic peers Honcho does not need to model, like bots or tool agents. Still save their messages so other peers have session context. Keep it enabled for users and evolving agents. | +| Do I need `observe_others`? | Only when a peer needs its own perspective on another participant, such as in games, multi-agent systems, or parent/subagent workflows. | ## Workspace Design -Workspaces are the top-level container. Everything inside a workspace (peers, sessions, messages, and all reasoning) is fully isolated from other workspaces. +A workspace is a hard isolation boundary. **Default to one workspace per application,** and split only at a real privacy, compliance, or product boundary (e.g. per-tenant SaaS, or a tool that needs intentionally isolated memory). Agents that collaborate over the same product, user, or game state belong in the *same* workspace so each can retrieve what the others produced. -**One workspace per application** is the most common pattern. Use separate workspaces when you need hard isolation: - -| Pattern | When to use | -|---------|-------------| -| Single workspace | Most applications. One product, one environment. | -| Per-tenant | Multi-tenant SaaS where each customer's data must be completely isolated. | +Honcho plugins default to one workspace *per host* (`hermes`, `claude_code`, `cursor`, `opencode`). To unify memory across them, point each at the same workspace — see [Unified Memory Setup](/v3/guides/recipes/unified-memory-setup). -If you are using the SDK, it will create a workspace called `default` if no name is specified for `workspace_id` +The SDK creates a workspace called `default` when no `workspace_id` is specified. --- ## Peer Design -A peer is any entity that participates in a session. Observation settings control which ones Honcho reasons about. +Give each real-world entity **one** stable peer ID and reuse it everywhere — splitting one entity across `user-web`, `user-discord`, and `user-slack` builds three separate representations. Prefix IDs by source for multi-channel apps (`discord_491827364`), and if a peer goes by multiple names, store the aliases in its peer card with `set_card()` / `setCard()`. -**What makes a good peer?** - -- It participates in sessions (a user, an agent, a character, an NPC) -- It persists across sessions -- It changes over time (preferences shift, knowledge grows), or it produces messages you want Honcho to see - -**Naming conventions** - -Give peers stable, unique identifiers scoped to your application: - - -```python Python -# Prefix with the source platform for multi-channel apps -peer = honcho.peer("discord_491827364") -peer = honcho.peer("slack_U04ABCDEF") - -# Use your own user IDs for backend integrations -peer = honcho.peer("user_abc123") - -# Use descriptive names for agents/assistants -peer = honcho.peer("assistant") -peer = honcho.peer("dungeon-master") -``` - -```typescript TypeScript -// Prefix with the source platform for multi-channel apps -const peer = await honcho.peer("discord_491827364"); -const peer = await honcho.peer("slack_U04ABCDEF"); - -// Use your own user IDs for backend integrations -const peer = await honcho.peer("user_abc123"); - -// Use descriptive names for agents/assistants -const peer = await honcho.peer("assistant"); -const peer = await honcho.peer("dungeon-master"); -``` - - -If your Peer represents an entity that may go by multiple different names, such as nicknames indicate that in the Peer Card: - - -```python Python -peer = honcho.peer("user_abc123") -peer.set_card([ - "Name: Alice. Also known as 'Ali' and 'A'.", - "College student, prefers casual tone.", -]) -``` - -```typescript TypeScript -const peer = await honcho.peer("user_abc123"); -await peer.setCard([ - "Name: Alice. Also known as 'Ali' and 'A'.", - "College student, prefers casual tone.", -]); -``` - - -**When to disable reasoning** - -Not every peer needs a representation. Set `observe_me: false` on peers that behave deterministically. - - -```python Python -from honcho.api_types import PeerConfig - -# The assistant doesn't need a representation -assistant = honcho.peer("assistant", configuration=PeerConfig(observe_me=False)) - -# The user does--this is who you want to understand -user = honcho.peer("user-123", configuration=PeerConfig(observe_me=True)) -``` - -```typescript TypeScript -const assistant = await honcho.peer("assistant", { configuration: { observeMe: false } }); -const user = await honcho.peer("user-123", { configuration: { observeMe: true } }); -``` - + +For unified context across Honcho plugins, set the same user peer ID (`peerName`) everywhere — that shared ID is what connects memory across Claude Code, Cursor, OpenCode, and your own app. See [Unified Memory Setup](/v3/guides/recipes/unified-memory-setup). + --- ## Session Design -Sessions define the temporal boundaries of an interaction. How you scope sessions directly affects how summaries are generated and how context is retrieved. +Sessions define the temporal boundaries of an interaction. How you scope them affects how summaries are generated, how context is retrieved, and when reasoning fires. **Common session patterns** | Pattern | Session scoped to | Example | |---------|-------------------|---------| -| Per-conversation | Each new chat thread | ChatGPT-style UI where each thread is a session | +| Per-conversation | Each new chat thread | ChatGPT or Claude Code style UI where each thread is a session | | Per-channel | A persistent channel or room | Discord channel, Slack thread | | Per-interaction | A bounded task or encounter | A support ticket, a game encounter | +| Per-project | A persistent work area | Coding agent memory for one repository | | Per-import | A batch of external data | Importing emails or documents for a single peer | -**When to create new sessions vs reuse** +Create a **new** session when context resets (new conversation, new day, new topic); **reuse** one when context should keep accumulating (ongoing channel, persistent thread). -- **New session** when the context resets (new conversation, new day, new topic) -- **Reuse session** when context should accumulate (ongoing channel, persistent thread) + +**Don't scope sessions too thin.** Honcho only reasons over a peer once it accumulates ~1,000 tokens *within a single session* ([token batching](/v3/documentation/core-concepts/reasoning#token-batching)). Many tiny sessions each stall below that threshold, so low-volume or trickle inputs should append to one ongoing session rather than fragment across many (nothing is lost — it just waits). + + +**How cross-session reasoning works** + +- **Session memory** is local to an interaction — summaries and recent-message context describe only what happened there. +- **Peer memory** (representations) accumulates reasoning across every session the peer is part of. + +So you can start a session fresh or pull in a peer's long-term memory. [`session.context()`](/v3/documentation/features/get-context) returns the current session's summary and recent messages; add a [peer target](/v3/documentation/features/get-context#peer-representation-in-context) to fold in that peer's cross-session history. --- -## Application Patterns - -### AI Companions - -An assistant that remembers the user across sessions and platforms. The [Honcho plugin for OpenClaw](/v3/guides/integrations/openclaw) is a production example--one assistant with memory across WhatsApp, Telegram, Discord, and Slack. - - -```python Python -from honcho import Honcho -from honcho.api_types import PeerConfig, SessionPeerConfig - -honcho = Honcho(workspace_id="my-companion-app") - -owner = honcho.peer("owner") -agent = honcho.peer("agent-main", configuration=PeerConfig(observe_me=False)) - -# Session key = thread + platform → separate histories, shared user memory -session = honcho.session("general-discord") -session.add_peers([ - (owner, SessionPeerConfig(observe_me=True, observe_others=False)), - (agent, SessionPeerConfig(observe_me=True, observe_others=True)), -]) - -session.add_messages([ - owner.message("I've been stressed about the move to Portland next month"), - agent.message("Moving is a big deal. What's weighing on you the most?"), - owner.message("Honestly just leaving my friend group behind"), -]) - -# Query from any session or platform -response = owner.chat("What's going on in this user's life right now?") -``` - -```typescript TypeScript -const honcho = new Honcho({ workspaceId: "my-companion-app" }); - -const owner = await honcho.peer("owner"); -const agent = await honcho.peer("agent-main", { configuration: { observeMe: false } }); - -const session = await honcho.session("general-discord"); -await session.addPeers([ - ["owner", { observeMe: true, observeOthers: false }], - ["agent-main", { observeMe: true, observeOthers: true }], -]); - -await session.addMessages([ - owner.message("I've been stressed about the move to Portland next month"), - agent.message("Moving is a big deal. What's weighing on you the most?"), - owner.message("Honestly just leaving my friend group behind"), -]); - -const response = await owner.chat("What's going on in this user's life right now?"); -``` - - -**Key decisions (from the [OpenClaw plugin](/v3/guides/integrations/openclaw)):** -- **Session key = thread + platform** — `general-discord` and `general-telegram` are separate sessions but share a single owner representation, so Honcho learns from every channel -- **Dynamic agent peers** — each agent gets its own peer (`agent-{id}`), resolved via a workspace-level map. Renaming an agent recovers the peer by metadata lookup -- **Subagent hierarchy** — when a primary agent spawns a subagent, the parent joins the child's session as a silent observer (`observe_me: false, observe_others: true`), giving Honcho visibility into the full agent tree -- **Asymmetric observation** — both owner and agent are observed, but with different scopes: owner has `observe_others: false` (default view), while the agent has `observe_others: true` so it can build its own representation of the owner. Subagents get lighter context (peer card only, no session summary) - -See the [OpenClaw integration guide](/v3/guides/integrations/openclaw) for the full plugin setup. - ---- - -### Coding Agents - -Coding agents survive terminal restarts, editor switches, and project hops. The [Honcho plugin for Claude Code](/v3/guides/integrations/claude-code) is a production example of this pattern. - - -```python Python -from honcho import Honcho -from honcho.api_types import PeerConfig - -honcho = Honcho(workspace_id="claude_code") - -# Developer is observed; agent is not -developer = honcho.peer("user") -agent = honcho.peer("claude", configuration=PeerConfig(observe_me=False)) - -# Session per project directory -- stable across restarts -session = honcho.session("user-honcho-repo") -session.add_peers([developer, agent]) - -session.add_messages([ - developer.message("refactor the auth module to use dependency injection"), - agent.message("I'll extract the auth dependencies into a provider pattern..."), - developer.message("actually let's keep it simpler, just pass the config directly"), -]) - -# In a future session, query what Honcho learned -context = developer.chat("What are this developer's preferences for code architecture?") -# Honcho knows: prefers simplicity, reverses decisions when simpler approach exists -``` - -```typescript TypeScript -const honcho = new Honcho({ workspaceId: "claude_code" }); - -const developer = await honcho.peer("user"); -const agent = await honcho.peer("claude", { configuration: { observeMe: false } }); - -const session = await honcho.session("user-honcho-repo"); -await session.addPeers([developer, agent]); - -await session.addMessages([ - developer.message("refactor the auth module to use dependency injection"), - agent.message("I'll extract the auth dependencies into a provider pattern..."), - developer.message("actually let's keep it simpler, just pass the config directly"), -]); - -const context = await developer.chat("What are this developer's preferences for code architecture?"); -``` - - -**Key decisions (from the Claude Code plugin):** -- **One workspace per tool** -- Claude Code and Cursor each get their own workspace, with optional cross-linking for read access -- **Asymmetric peers** -- developer is observed (memory formation), agent is not observed but still stores messages so Honcho sees both sides -- **Session-per-directory** by default -- each project accumulates its own memory. Prefix with peer name (`user-honcho-repo`) so multiple developers on the same workspace don't collide. Alternative strategies: `git-branch` (session switches on branch change) or `chat-instance` (clean slate each time) -- **Filter what you store** -- user messages go in real-time; agent messages are filtered to skip trivial tool output and keep substantive explanations -- **Import external data** with single-peer sessions to ingest READMEs, architecture docs, or commit history into a developer's representation - -See the [Claude Code integration guide](/v3/guides/integrations/claude-code) for the full plugin setup. - ---- - -### Games - -Games introduce multi-peer scenarios where **information asymmetry matters**. An NPC should only know what it has witnessed, not the full game state. - - -```python Python -from honcho import Honcho -from honcho.api_types import SessionPeerConfig - -honcho = Honcho(workspace_id="my-rpg") - -# Every character is a peer -player = honcho.peer("player-one") -merchant = honcho.peer("merchant-grim") -thief = honcho.peer("thief-shadow") - -# Scene 1: Player talks to the merchant -tavern = honcho.session("tavern-scene") -tavern.add_peers([player, merchant]) - -# Enable the merchant to build its own representation of the player -tavern.set_peer_configuration(merchant, SessionPeerConfig(observe_others=True)) - -tavern.add_messages([ - player.message("I'm looking for a rare gemstone. Money is no object."), - merchant.message("I may know of one... but it won't come cheap."), -]) - -# Scene 2: Player talks to the thief (merchant isn't here) -alley = honcho.session("dark-alley") -alley.add_peers([player, thief]) -alley.set_peer_configuration(thief, SessionPeerConfig(observe_others=True)) - -alley.add_messages([ - player.message("I need that gemstone stolen from the merchant. Quietly."), - thief.message("Consider it done. Half up front."), -]) - -# The merchant's view of the player: wealthy buyer seeking a gemstone -merchant_view = merchant.chat("What do I know about this player?", target="player-one") - -# The thief's view: someone willing to steal from the merchant -thief_view = thief.chat("What do I know about this player?", target="player-one") - -# Honcho's global view: knows both sides of the story -full_view = player.chat("What is this player up to?") -``` - -```typescript TypeScript -const honcho = new Honcho({ workspaceId: "my-rpg" }); - -const player = await honcho.peer("player-one"); -const merchant = await honcho.peer("merchant-grim"); -const thief = await honcho.peer("thief-shadow"); - -const tavern = await honcho.session("tavern-scene"); -await tavern.addPeers([player, merchant]); -await tavern.setPeerConfiguration(merchant, { observeOthers: true }); - -await tavern.addMessages([ - player.message("I'm looking for a rare gemstone. Money is no object."), - merchant.message("I may know of one... but it won't come cheap."), -]); - -const alley = await honcho.session("dark-alley"); -await alley.addPeers([player, thief]); -await alley.setPeerConfiguration(thief, { observeOthers: true }); - -await alley.addMessages([ - player.message("I need that gemstone stolen from the merchant. Quietly."), - thief.message("Consider it done. Half up front."), -]); - -const merchantView = await merchant.chat("What do I know about this player?", { target: "player-one" }); -const thiefView = await thief.chat("What do I know about this player?", { target: "player-one" }); -const fullView = await player.chat("What is this player up to?"); -``` - - -**Key decisions:** -- Every character (player, NPC) is a peer -- `observe_others: true` lets NPCs build their own representations of the player based only on what they've witnessed -- Session-per-scene or session-per-encounter so context scopes to specific interactions -- Use `target` when querying to get a specific NPC's perspective rather than Honcho's omniscient view -- See [Representation Scopes](/v3/documentation/features/advanced/representation-scopes) for the full details - ---- ## Common Mistakes +- **Splitting one identity across peer IDs** -- If the same user is `alice`, `alice-discord`, and `alice-cursor`, Honcho builds separate representations. Use one stable peer ID when you want unified memory. +- **Too many tiny sessions** -- Summaries and recent messages are session-scoped, and reasoning only fires past ~1,000 tokens per session. Splitting a continuous conversation across many sessions fragments local context and can stall reasoning. Reuse a session when context should flow continuously. +- **Separating agents that should collaborate** -- If agents need shared product, customer, or team context, put them in the same workspace. Separate workspaces are hard isolation boundaries. - **Leaving `observe_me` on for assistants** -- Wastes reasoning compute on a peer you control. Deterministic behavior doesn't need to be modeled. -- **Not storing messages** -- Honcho reasons about messages asynchronously. If you don't call `add_messages()`, there's nothing to reason about — no messages means no memory. See [Storing Data](/v3/documentation/features/storing-data) for details. -- **Creating a new workspace per user** -- Use peers within a single workspace instead. Workspaces are for isolation between applications, not between users. -- **Too many tiny sessions** -- Summaries and `session.context()` are scoped to a single session. If you split a continuous conversation across many sessions, context is fragmented and each session is too short to summarize. Reuse a session when context should flow continuously. +- **Turning on `observe_others` everywhere** -- Directional representations are powerful, but they add complexity. Use them when peers need distinct perspectives, not just because a session has multiple peers. +- **Forgetting `peer_target` on session context** -- `session.context()` defaults to the active session's summary and recent messages, which are session-scoped. It becomes cross-session only through adding a peer_target which includes the peer representation. - **Blocking on processing** -- Messages are processed asynchronously in the background. Don't poll or wait for reasoning to complete before continuing your application flow. ## Next Steps + + Wire these patterns into one shared workspace across four integrations + Retrieve formatted context from sessions for your LLM @@ -372,7 +100,4 @@ const fullView = await player.chat("What is this player up to?"); Fine-tune what gets reasoned about and how - - Directional representations for multi-peer scenarios - diff --git a/docs/v3/guides/overview.mdx b/docs/v3/guides/overview.mdx index 4411629d..4be16f39 100644 --- a/docs/v3/guides/overview.mdx +++ b/docs/v3/guides/overview.mdx @@ -7,6 +7,15 @@ icon: 'puzzle-piece' Honcho plugs into whatever you're already building. Add memory to an AI assistant, connect an external data source, wire Honcho into your agent framework, or migrate from another provider. +## Recipes +Compose the core primitives across multiple integrations: + + + + One shared workspace across a chat companion, coding agent, autonomous agent, and ingestion job + + + ## AI Assistants Add persistent memory to AI assistants and agents: diff --git a/docs/v3/guides/recipes/unified-memory-setup.mdx b/docs/v3/guides/recipes/unified-memory-setup.mdx new file mode 100644 index 00000000..1d547db2 --- /dev/null +++ b/docs/v3/guides/recipes/unified-memory-setup.mdx @@ -0,0 +1,186 @@ +--- +title: "Unified Memory Setup" +sidebarTitle: "Unified Memory" +icon: "diagram-project" +description: "Wire one shared Honcho workspace across a chat companion, a coding agent, an autonomous agent, and a scheduled ingestion job" +--- + +This is a how-to, not an intro. It assumes you know what workspaces, peers, and +sessions are. If you don't, start with [Core Concepts](/v3/documentation/core-concepts/). + + +This guide wires four integration points into a single Honcho setup: a +chat companion (Discord/Slack), a coding agent (Claude Code), an autonomous agent +(Hermes), and a cron job that ingests external data. They share **one workspace** and +**one peer** for the user, so everything Honcho learns about your user in one place is +available everywhere else. Each section below notes the per-host setup. + +--- + +## 1. Chat companion (Discord / Slack) + +**One session per conversation surface, one peer per participant.** The channel, thread, +or DM is the session; everyone who speaks in it gets their own peer: + +- Channel → `discord-channel-{channel_id}` +- Thread → `discord-thread-{thread_id}` +- DM → `discord-dm-{user_id}` + +Derive each peer ID from the immutable platform ID (`discord-{user_id}`), not the +display name (which can change). Keep the display name in peer metadata instead. A shared +channel then naturally holds several human peers in one session, with the bot joining +as its own peer (everyone observed on defaults): + +```python +session = honcho.session(f"discord-channel-{channel_id}") +session.add_peers([user, assistant]) # plus any other humans in the channel +``` + +Slack mirrors this with `slack_{user_id}` peers and `slack-{channel}` sessions. For a +full bot walkthrough — message ingestion, watchlists, and storing turns — see the +[Discord guide](/v3/guides/discord). + +--- + +## 2. Coding agent (Claude Code) + +**Scope sessions per project directory, prefixed with the user** — `{USER_PEER_ID}-{repo_name}` +— so multiple developers sharing the workspace don't collide on a session ID. Switch +to a `git-branch` scope only when each branch is genuinely a separate line of work. + +The Claude Code plugin reads its workspace and peers from `.honcho/config.json`. Point +each host at the same `workspace` and use the same top-level `peerName`, so every host +attributes you to one peer: + +```json .honcho/config.json +{ + "peerName": "your-user-id", + "hosts": { + "claude_code": { "workspace": "my-product", "aiPeer": "claude" }, + "opencode": { "workspace": "my-product", "aiPeer": "opencode" } + } +} +``` + + +This is a minimal, illustrative snippet — the real config file carries more fields +(session maps, recall mode, observation strategy, etc.). See the [integration](/v3/guides/overview/) +guides for the full schema and per-host options. + + +Add the user peer and the `claude` agent peer (no special observation config needed), +then store turns — stripping `tool_use` blocks from the assistant message so only +substantive explanation lands in the session. + +Because this uses the **same user peer** as the companion, a preference the user +states while coding ("keep it simple, pass config directly") is queryable from the +Discord bot via `user.chat(...)`, and vice versa — both write to the same peer +representation. + + +**A shared workspace is not the default.** The Honcho plugins — Claude Code, OpenCode, +Hermes, Cursor — each default to a *per-host* workspace (`Claude_Code`, `hermes`, …), +keeping memory isolated per tool. The unification above only happens when you set the +same workspace **and** the same user peer across all of them; otherwise each builds its +own separate representation. + + +--- + +## 3. Autonomous agent (Hermes) + +The Hermes Honcho plugin is configured through `honcho.json`, set its `workspace` +and `aiPeer` there. See the [Hermes guide](/v3/guides/integrations/hermes) for the full config schema. + +- **Sessions** follow a `session_strategy` (default `per-directory`, like the coding + agent above; `per-repo` or `per-session` for a fresh Honcho session each run). The + user peer defaults to `user-{channel}-{chat_id}` unless you pin a `peerName`. +- **Observation** defaults to `directional` — both the user and the `hermes` agent + peer are observed, consistent with the defaults above, so Hermes builds a + representation of itself as well as the user. +- Hermes exposes Honcho as agent **tools** (`honcho_reasoning` for synthesized + answers, plus lighter `honcho_search` and `honcho_context` lookups) and decides when + to call them mid-task. Unlike other sources, you write no retrieval code — + the agent pulls cross-session context on its own. + +--- + +## 4. Scheduled data ingestion (cron) + +A scheduled job feeds external data (emails, meeting notes, CRM records) into Honcho. +Attribute the messages to the peer the data is *about* — not to an agent — and group +them into a session. **How you scope that session is the main decision here**, because +it controls when Honcho reasons over the data (more on that below). + +```python +from datetime import datetime, timezone + +session = honcho.session(f"email-import-{datetime.now(timezone.utc):%Y-%m-%d}") +session.add_peers([user]) + +messages = [ + user.message( + f"Subject: {e['subject']}\nFrom: {e['from']}\n\n{e['body']}", + metadata={"source": "gmail", "thread_id": e["thread_id"]}, + created_at=e["timestamp"], # the event's time, NOT import time + ) + for e in emails +] +# add_messages accepts at most 100 messages per call — split into requests of 100 +for i in range(0, len(messages), 100): + session.add_messages(messages[i:i + 100]) +``` + +Honcho only reasons over a peer once it accumulates ~1,000 tokens *within a single session* +([token batching](/v3/documentation/core-concepts/reasoning#token-batching)). Scope +the session to the volume you ingest: + +- **High-volume runs** (a day of emails, a CRM export) clear the threshold easily — a + per-run session like `email-import-{date}` is fine. +- **Low-volume or trickle imports** (a few short records at a time) should append to + one **ongoing per-source session** (e.g. `email-import-gmail`), so content + accumulates across runs instead of fragmenting into thin sessions that each stall + below the threshold (nothing is lost — it just waits). + +The [Gmail](/v3/guides/gmail) and [Granola](/v3/guides/granola) guides are related +import examples. + + +If a cron run also posts as a deterministic agent (a bot or tool agent whose behavior +you fully control), set `observe_me=False` on that peer so Honcho doesn't spend +reasoning modeling it. Its messages still land in the session for context. + +```python +agent = honcho.peer("cron_agent", configuration=PeerConfig(observe_me=False)) +``` + + +--- + +## What you end up with + +From any integration, the same call — `user.chat("What is this user working on, and +what do they care about?")` — draws on all four sources at once: Discord chats, coding +decisions, Hermes task runs, and imported emails. They blend because: + +- **One workspace and one user peer**, so the representation accumulates in one place + instead of fragmenting into `user-discord`, `user-cursor`, etc. +- **Sessions scoped to the live interaction** (channel, repo, task run, import batch), + so local context stays coherent while the user peer carries the long view. + +## Next Steps + + + + The reasoning behind every decision in this guide. + + + Pull session + cross-session context into your LLM calls. + + + An interactive import using the per-import session and created_at patterns. + + + Production multi-platform companion with parent/subagent tracking. + +