RomanNum3ral
/
honcho
mirror of https://github.com/plastic-labs/honcho.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into main

This commit is contained in:
Anush 2026-05-29 19:21:43 +05:30 committed by GitHub
parent d369e77391 85239a69b2
commit 06eb3d8ef5
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
101 changed files with 9257 additions and 3024 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.pre-commit-config.yaml
View File

@ -60,7 +60,7 @@ repos:
language: system
files: ^(src/|tests/|sdks/python/|scripts/).*\.py$
require_serial: true
pass_filenames: false
pass_filenames: true
# Run main application tests
- id: pytest-main

77
CHANGELOG.md
View File

@ -5,39 +5,74 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/).
## [Unreleased]
## [3.0.7] - 2026-05-21
### Added
- Support for using [Qdrant](https://qdrant.tech/) as the vector search provider.
- New `src/llm/` package as the single owner of provider runtime: clients, backends, history adapters, tool loop, request builder, credentials, and caching policy
- `AttemptPlan` dataclass captures per-retry provider selection (client, model, reasoning_effort, thinking_budget_tokens, selected_config) and pins it across stream-final retries so streaming doesn't bounce back to primary after the tool loop has settled on fallback
- Gemini JSON-schema sanitizer for `function_declarations` — strips keywords Gemini's validator rejects (`additionalProperties`, `allOf`, etc.) while preserving semantics for all other backends
- Dreamer specialists derive `effective_max_tokens` from `model_config.max_output_tokens` with a per-specialist default fallback
- Regression tests covering fallback-config thinking-param reach, provider_params → extra_params boundary, OpenAI reasoning-model parameter routing, Gemini blocked finish_reason handling, and fail-fast `max_tool_iterations` validation
- New `src/llm/` module as the single owner of provider runtime: clients, backends, history adapters, tool loop, request builder, credentials, and caching policy (#459)
- `AttemptPlan` dataclass captures per-retry provider selection (client, model, reasoning_effort, thinking_budget_tokens, selected_config) and pins it across stream-final retries so streaming doesn't bounce back to primary after the tool loop has settled on fallback (#459)
- Gemini JSON-schema sanitizer for `function_declarations` — strips keywords Gemini's validator rejects (`additionalProperties`, `allOf`, etc.) while preserving semantics for all other backends (#459)
- Dreamer specialists derive `effective_max_tokens` from `model_config.max_output_tokens` with a per-specialist default fallback (#459)
- New cloudevent `LLMCallCompletedEvent` (`llm.call.completed`) fires once per provider hit with full cost-attribution context: transport/provider_label, model, token counts with cache breakdown, finish_reason, outcome, `is_final_attempt`, retry/fallback state, duration, tool-call shape, streaming flag, and agent correlation (`run_id` + iteration). Includes a `CallPurpose` closed enum (`deriver.representation`, `dialectic.answer`, `dream.deduction|induction`, `summary.short|long`) (#637)
- `RepresentationCompletedEvent` now carries `total_input_tokens` for full-trace cost attribution (#637)
- Per-emitter `honcho_version` injection on all CloudEvents plus emitter health metrics (#637)
- `TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE` (default 1.0) — deterministic per-`run_id` sampler so an entire agent trace is kept or dropped together; aggregate envelopes bypass the sampler (#637)
- Deriver custom instructions: per-workspace/peer guidance threaded into the deriver prompt with a `MAX_CUSTOM_INSTRUCTIONS_TOKENS` budget (default 2000); deriver `MAX_INPUT_TOKENS` raised 23000 → 25000 to make room (#609)
- Configurable embedding dimensions: `EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE` (`auto`/`always`/`never`) controls whether the OpenAI `dimensions=` parameter is forwarded; `auto` (default) sends it when the operator explicitly set `EMBEDDING_VECTOR_DIMENSIONS` and the model is not on the known-rejecting allowlist (#678)
- New `honcho-cli` package — Python CLI for inspecting and managing peers, sessions, and configuration against a Honcho deployment (#424)
- `HONCHO_API_URL` env var support in the MCP Worker, enabling self-hosted Honcho deployments to point the Worker at their own instance instead of `https://api.honcho.dev` (#575)
- API ID `max_length` increased from 100 to 512 across `WorkspaceCreate`, `PeerCreate`, and `SessionCreate` to align the API contract with the underlying DB schema (#684)
- Regression tests covering fallback-config thinking-param reach, provider_params → extra_params boundary, OpenAI reasoning-model parameter routing, Gemini blocked finish_reason handling, and fail-fast `max_tool_iterations` validation (#459)
### Changed
- All LLM orchestration moved out of `src/utils/clients.py` into `src/llm/` with modules split by responsibility (api, executor, tool_loop, runtime, registry, conversation, request_builder, credentials, caching, backends, history_adapters)
- Default `ModelConfig` factories (deriver, summary, dreamer specialists, dialectic levels) normalized to `openai/gpt-5.4-mini` with no extra parameters set by default; operators add transport/thinking overrides explicitly
- OpenAI reasoning-model routing widened via `_uses_max_completion_tokens` heuristic covering `gpt-5.x` and `o1/o3/o4` — these models receive `max_completion_tokens` instead of `max_tokens`
- Override client factories switched from unbounded `@cache` to `@lru_cache(maxsize=128)` for predictable memory growth on long-running processes
- `get_backend` now delegates to `client_for_model_config`, so the live-test path and production path share one missing-API-key validation
- Blocked Gemini responses (`SAFETY`, `RECITATION`, `PROHIBITED_CONTENT`, `BLOCKLIST`) raise `LLMError` in the streaming path too (previously only the non-streaming path), ensuring retry/fallback logic fires uniformly
- Transport-change env overrides now strip transport-specific thinking params (thinking_budget_tokens vs. reasoning_effort) during config merge, including at the dialectic-level merge, so switching from Anthropic → OpenAI doesn't leave orphaned Anthropic-only params that the OpenAI backend would reject
- `max_tool_iterations` out-of-range inputs now raise `ValidationException` instead of being silently clamped
- Troubleshooting docs updated to reflect nested-env-var form for per-component thinking-budget overrides
- All LLM orchestration moved out of `src/utils/clients.py` into `src/llm/` with modules split by responsibility (api, executor, tool_loop, runtime, registry, conversation, request_builder, credentials, caching, backends, history_adapters) (#459)
- Default `ModelConfig` factories (deriver, summary, dreamer specialists, dialectic levels) normalized to `openai/gpt-5.4-mini` with no extra parameters set by default; operators add transport/thinking overrides explicitly (#459)
- OpenAI reasoning-model routing widened via `_uses_max_completion_tokens` heuristic covering `gpt-5.x` and `o1/o3/o4` — these models receive `max_completion_tokens` instead of `max_tokens` (#459)
- Override client factories switched from unbounded `@cache` to `@lru_cache(maxsize=128)` for predictable memory growth on long-running processes (#459)
- `get_backend` now delegates to `client_for_model_config`, so the live-test path and production path share one missing-API-key validation (#459)
- Blocked Gemini responses (`SAFETY`, `RECITATION`, `PROHIBITED_CONTENT`, `BLOCKLIST`) raise `LLMError` in the streaming path too (previously only the non-streaming path), ensuring retry/fallback logic fires uniformly (#459)
- Transport-change env overrides now strip transport-specific thinking params (thinking_budget_tokens vs. reasoning_effort) during config merge, including at the dialectic-level merge, so switching from Anthropic → OpenAI doesn't leave orphaned Anthropic-only params that the OpenAI backend would reject (#459)
- `max_tool_iterations` out-of-range inputs now raise `ValidationException` instead of being silently clamped (#459)
- Public API schemas (`WorkspaceCreate`, `PeerCreate`, `SessionCreate`) and SDK validation (`api_types.py`, `validation.ts`) accept IDs up to 512 chars (was 100) (#684)
- Peer card prompts reframed as stable identity markers (replaces the prior "biographical/profile facts" language). Induction specialist is now opted out of peer card writes (`can_update_peer_card = False`) so only deduction touches the card (#686)
- Vector store queries no longer fetch embedding vectors — only document metadata is returned, reducing payload size and DB load (pgvector, lancedb, turbopuffer) (#682)
- Langfuse trace metadata now includes `namespace`, `model`, and `provider` so traces can be filtered by deployment slice (#565)
- Deriver: model-aware tokenizer (replaces the previously hardcoded encoding) and explicit guard on empty message content (#647)
- Dialectic level defaults now merge correctly with per-level overrides in `src/config` (DEV-1733) (#656)
- Default dialectic tool choice switched from forced/required to `auto` (#630)
- Vector sync given a substantial retry budget to tolerate transient embedding provider outages (#604)
- `AgentToolConclusionsDeletedEvent` payload now carries `levels` for parity with the rest of the conclusion event surface (#612)
- Turbopuffer vector store: `InternalServerError` caught and surfaced as a warning rather than a hard failure; unused `upsert_with_retry` and `VectorUpsertResult` removed; explicit silent and explicit-error paths for vector DB server errors (#561)
- Troubleshooting docs updated to reflect nested-env-var form for per-component thinking-budget overrides (#459)
- README refresh (#681)
- CLAUDE.md refreshed against the current `src/` layout (#680)
### Fixed
- Fallback `ModelConfig` temperature and `thinking_budget_tokens` reach the backend on the final retry — previously the primary's values were pre-populated into caller kwargs early and clobbered fallback values via `effective_config_for_call(update=...)`
- Stream-final retries pin to the `AttemptPlan` that succeeded rather than re-running provider selection through the outer `current_attempt` ContextVar (which could roll streaming back to primary after the tool loop had already switched to fallback)
- OpenAI structured-output calls continue to use `chat.completions.parse()` with strict schema enforcement, while tool-calling paths use `chat.completions.create()` without `strict:True` for broader proxy compatibility (OpenRouter, vLLM, Ollama)
- Gemini `cached_content` reuse keys now include `system_instruction` and `tool_config` so cache hits don't cross configurations that differ only in those fields
- Fallback `ModelConfig` temperature and `thinking_budget_tokens` reach the backend on the final retry — previously the primary's values were pre-populated into caller kwargs early and clobbered fallback values via `effective_config_for_call(update=...)` (#459)
- Stream-final retries pin to the `AttemptPlan` that succeeded rather than re-running provider selection through the outer `current_attempt` ContextVar (which could roll streaming back to primary after the tool loop had already switched to fallback) (#459)
- OpenAI structured-output calls continue to use `chat.completions.parse()` with strict schema enforcement, while tool-calling paths use `chat.completions.create()` without `strict:True` for broader proxy compatibility (OpenRouter, vLLM, Ollama) (#459)
- Gemini `cached_content` reuse keys now include `system_instruction` and `tool_config` so cache hits don't cross configurations that differ only in those fields (#459)
- Removed strict parameter validation for thinking params on Anthropic and OpenAI transports — was rejecting valid per-transport configs (#686)
- `reverse` query parameter is now honored on the v3 workspace list (`POST /v3/workspaces/list`), peer list (`POST /v3/workspaces/{workspace_id}/peers/list`), workspace-scoped session list (`POST /v3/workspaces/{workspace_id}/sessions/list`), and peer-scoped session list (`POST /v3/workspaces/{workspace_id}/peers/{peer_id}/sessions`). Honcho SDKs at 2.1.0+ were already sending `reverse=true` for these routes but the server silently ignored it. Ties on `created_at` now fall back to the internal nanoid `id` so ordering remains stable across pages (#685)
- LLM client factories now receive `base_url` from `LLMSettings` for default providers — previously the override path honored `base_url` but the default path didn't, so operators pointing at OpenAI-compatible proxies via `LLM__OPENAI_BASE_URL` were ignored (#643, fixes #641)
- Internal N+1 query in dialectic agent tool execution (DEV-1721) — collapsed per-iteration DB lookups into a single fetch (#652)
- Dreamer threshold and time-guard semantics: `check_and_schedule_dream` count filter now includes only `documents.level == 'explicit'` (dreamer-created levels are output, not input, and were inflating the threshold and creating a feedback loop); `last_dream_at` write relocated from `enqueue_dream` into `process_dream` so duplicate enqueues or failed runs no longer reset the 8-hour time guard (#573)
- Deriver: blank observations are filtered out before embedding (previously triggered noisy embedding calls and persisted empty rows); blank-observation filtering unified across tool paths (#615)
- Surprisal module: filter for level observations changed from `{"level": levels}` to `{"level": {"in": levels}}` — `apply_filter()` requires operator syntax, so the prior call silently returned 0 results and made the entire Surprisal phase of the Dream cycle a no-op (#581, fixes #559)
- Removed hardcoded `stop_sequences` override from Deriver `ModelConfig` (was clobbering operator-configured stop sequences) (#587)
- Removed stale `stop_sequences` from tests (#607)
- Embedding client: `embed()` now wraps single-string input in an array, restoring compatibility with OpenAI-compatible third-party providers that reject scalar input (#586)
- Docker Compose: deriver service startup gated on the API service healthcheck (prevents races where the deriver starts before the API has run migrations) (#689)
- Docker image: `HEALTHCHECK` directive removed from the shared base image — it probed an HTTP endpoint only the API serves, permanently marking deriver containers as unhealthy. Service-level health checks now belong in each service's own configuration (k8s readiness/liveness probes on the API Deployment only) (#530)
- `tests/unified`: `--test-dir`/`--test-file` arguments now use an argparse mutually-exclusive group instead of manual validation (#650)
- CrewAI example updated for the latest CrewAI protocol (#631)
### Removed
- `src/utils/clients.py` deleted; its responsibilities are split across `src/llm/registry.py`, `src/llm/credentials.py`, and the backend-specific modules
- `src/utils/clients.py` deleted; its responsibilities are split across `src/llm/registry.py`, `src/llm/credentials.py`, and the backend-specific modules (#459)
- `HEALTHCHECK` directive removed from the shared Docker image (#530)
## [3.0.6] - 2026-04-10

7
CLAUDE.md
View File

@ -140,6 +140,7 @@ The Deriver processes batches of incoming messages and extracts conclusions abou
- **Output**: Explicit conclusions (direct facts) and deductive conclusions (inferences) saved to `(observer, observed)` collections.
- **Entry point**: `src/deriver/__main__.py` → `queue_manager.main()`.
- **Prompts**: `src/deriver/prompts.py` (`minimal_deriver_prompt`).
- **Custom instructions**: per-workspace/peer guidance can be threaded into the prompt via reasoning configuration; `DERIVER__MAX_CUSTOM_INSTRUCTIONS_TOKENS` caps the addition (default 2000) and `DERIVER__MAX_INPUT_TOKENS` defaults to 25000 to make room.
#### 2. Dialectic (`src/dialectic/`)
@ -177,8 +178,9 @@ The Dreamer is an orchestrated multi-specialist system that runs during schedule
#### Shared Agent Infrastructure
- **Tool definitions** (`src/utils/agent_tools.py`): unified `TOOLS` dict; per-agent lists (`DIALECTIC_TOOLS`, `DIALECTIC_TOOLS_MINIMAL`, `DREAMER_TOOLS`, `DEDUCTION_SPECIALIST_TOOLS`, `INDUCTION_SPECIALIST_TOOLS`).
- **LLM subsystem** (`src/llm/`): provider-agnostic `honcho_llm_call()`. Backends in `src/llm/backends/` (`anthropic.py`, `gemini.py`, `openai.py`). Includes prompt caching (`caching.py`), structured output (`structured_output.py`), tool loop (`tool_loop.py`), history adapters for cross-provider message formats, and a model registry.
- **LLM subsystem** (`src/llm/`): provider-agnostic `honcho_llm_call()`. Backends in `src/llm/backends/` (`anthropic.py`, `gemini.py`, `openai.py`). Includes prompt caching (`caching.py`), structured output (`structured_output.py`), tool loop (`tool_loop.py`), history adapters for cross-provider message formats, and a model registry. Per-retry provider selection is pinned via an `AttemptPlan` so stream-final retries don't bounce back to primary after the tool loop has settled on fallback.
- **Per-agent model config**: each agent has its own `MODEL_CONFIG` in `src/config.py` with fallback chains (see `ConfiguredModelSettings`, `FallbackModelSettings`).
- **Telemetry**: cloudevents in `src/telemetry/events/` cover API routes, dialectic, dream, deletion, reconciliation, representation, and per-call LLM accounting (`llm.py` — `LLMCallCompletedEvent` fires once per provider hit with full cost-attribution context). High-volume events are sampled deterministically per `run_id` via `TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE`.
### Project Structure
@ -192,7 +194,8 @@ src/
├── dependencies.py # FastAPI DI (tracked_db, etc.)
├── exceptions.py # Custom exception types (HonchoException + subclasses)
├── security.py # JWT authentication
├── embedding_client.py # Embedding provider client
├── embedding_client.py # Embedding provider client (configurable dimensions
│ # via EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE)
├── schemas/ # Pydantic schemas
│ ├── api.py # Public API request/response schemas
│ ├── configuration.py # Per-resource configuration schemas

46
README.md
View File

@ -8,7 +8,7 @@
---
![Static Badge](https://img.shields.io/badge/Server-3.0.6-blue)
![Static Badge](https://img.shields.io/badge/Server-3.0.7-blue)
[![PyPI version](https://img.shields.io/pypi/v/honcho-ai.svg)](https://pypi.org/project/honcho-ai/)
[![NPM version](https://img.shields.io/npm/v/@honcho-ai/sdk.svg)](https://npmjs.org/package/@honcho-ai/sdk)
[![Discord](https://img.shields.io/discord/1016845111637839922?style=flat&logo=discord&logoColor=23ffffff&label=Plastic%20Labs&labelColor=235865F2)](https://discord.gg/honcho)
@ -43,21 +43,21 @@ The Honcho project is split between several repositories, with this one hosting
## Start Here
| I want to... | Path | Get started |
|---|---|---|
| I want to... | Path | Get started |
| -------------------------------------- | ---------------------------------------------------------- | ----------------------------- |
| Give my coding agent persistent memory | Claude Code, OpenCode, OpenClaw, Hermes, or any MCP client | [Integrations](#integrations) |
| Add memory to my product | Python or TypeScript SDK | [Quickstart](#quickstart) |
| Self-host Honcho | Docker / local development | [Self-hosting](#self-hosting) |
| Add memory to my product | Python or TypeScript SDK | [Quickstart](#quickstart) |
| Self-host Honcho | Docker / local development | [Self-hosting](#self-hosting) |
## Why Honcho
| Capability | What it means |
|---|---|
| Reasoning-first memory | Extracts conclusions from conversations and events, not just matching chunks. |
| Peer-centric model | Tracks users, agents, groups, projects, and ideas as entities that change over time. |
| Multi-peer perspective | Models what one peer knows about another when configured. |
| Managed or self-hosted | Use `api.honcho.dev` or run the FastAPI server yourself. |
| Agent-tool integrations | MCP, Claude Code, OpenCode, OpenClaw, Hermes, Cursor-compatible clients. |
| Capability | What it means |
| ----------------------- | ------------------------------------------------------------------------------------ |
| Reasoning-first memory | Extracts conclusions from conversations and events, not just matching chunks. |
| Peer-centric model | Tracks users, agents, groups, projects, and ideas as entities that change over time. |
| Multi-peer perspective | Models what one peer knows about another when configured. |
| Managed or self-hosted | Use `api.honcho.dev` or run the FastAPI server yourself. |
| Agent-tool integrations | MCP, Claude Code, OpenCode, OpenClaw, Hermes, Cursor-compatible clients. |
## The Honcho Loop
@ -139,7 +139,9 @@ await session.addMessages([
tutor.message("Absolutely. Send me your first problem!"),
]);
const answer = await alice.chat("What learning styles does the user respond to best?");
const answer = await alice.chat(
"What learning styles does the user respond to best?",
);
const context = await session.context({ summary: true, tokens: 10_000 });
const openai = new OpenAI();
@ -153,15 +155,15 @@ const completion = await openai.chat.completions.create({
## What Honcho Gives You
| Need | API |
|---|---|
| Save interaction history | `session.add_messages(...)` |
| Ask what Honcho knows about a peer | `peer.chat(...)` |
| Get prompt-ready context | `session.context(...).to_openai(...)` / `.to_anthropic(...)` |
| Hybrid search (BM25 + vector) | `peer.search(...)`, `session.search(...)`, `honcho.search(...)` |
| Low-latency static representations | `peer.representation(...)`, `session.representation(...)` |
| Import documents | `session.upload_file(...)` |
| Inspect background processing | `honcho.queue_status(...)` |
| Need | API |
| ---------------------------------- | --------------------------------------------------------------- |
| Save interaction history | `session.add_messages(...)` |
| Ask what Honcho knows about a peer | `peer.chat(...)` |
| Get prompt-ready context | `session.context(...).to_openai(...)` / `.to_anthropic(...)` |
| Hybrid search (BM25 + vector) | `peer.search(...)`, `session.search(...)`, `honcho.search(...)` |
| Low-latency static representations | `peer.representation(...)`, `session.representation(...)` |
| Import documents | `session.upload_file(...)` |
| Inspect background processing | `honcho.queue_status(...)` |
See the full [SDK Reference](https://honcho.dev/docs/v3/documentation/reference/sdk) and [API Reference](https://honcho.dev/docs/v3/api-reference/introduction).

14
docker-compose.yml.example
View File

@ -21,6 +21,18 @@ services:
condition: service_healthy
ports:
- "127.0.0.1:8000:8000"
healthcheck:
test:
[
"CMD",
"/app/.venv/bin/python",
"-c",
"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).read()",
]
interval: 5s
timeout: 5s
retries: 5
start_period: 10s
# -- Development: mount source for live reload --
# volumes:
# - .:/app
@ -40,6 +52,8 @@ services:
dockerfile: Dockerfile
entrypoint: ["/app/.venv/bin/python", "-m", "src.deriver"]
depends_on:
api:
condition: service_healthy
database:
condition: service_healthy
redis:

7
docs/changelog/compatibility-guide.mdx
View File

@ -10,14 +10,14 @@ This guide helps you match the right SDK version to your Honcho API version. New
<CardGroup cols={2}>
<Card title="TypeScript SDK" icon="js">
**Latest:** v2.1.1
**Latest:** v2.1.2
```bash
npm install @honcho-ai/sdk
```
</Card>
<Card title="Python SDK" icon="python">
**Latest:** v2.1.1
**Latest:** v2.1.2
```bash
pip install honcho-ai
@ -30,7 +30,8 @@ This guide helps you match the right SDK version to your Honcho API version. New
| Honcho API Version | TypeScript SDK | Python SDK |
|-------------------|---------------|------------|
| v3.0.6 (Current) | v2.1.1 | v2.1.1 |
| v3.0.7 (Current) | v2.1.2 | v2.1.2 |
| v3.0.6 | v2.1.1 | v2.1.1 |
| v3.0.5 | v2.1.0 | v2.1.0 |
| v3.0.4 | v2.1.0 | v2.1.0 |
| v3.0.3 | v2.1.0 | v2.1.0 |

81
docs/changelog/introduction.mdx
View File

@ -27,7 +27,59 @@ Welcome to the Honcho changelog! This section documents all notable changes to t
### Honcho API and SDK Changelogs
<Tabs>
<Tab title="Honcho API">
<Update label="v3.0.6 (Current)">
<Update label="v3.0.7 (Current)">
### Added
- New `src/llm/` package as the single owner of provider runtime: clients, backends, history adapters, tool loop, request builder, credentials, and caching policy (#459)
- New cloudevent `LLMCallCompletedEvent` (`llm.call.completed`) fires once per provider hit with full cost-attribution context: transport/provider_label, model, token counts with cache breakdown, finish_reason, outcome, retry/fallback state, duration, tool-call shape, streaming flag, and agent correlation (`run_id` + iteration) (#637)
- `RepresentationCompletedEvent` now carries `total_input_tokens` for full-trace cost attribution; per-emitter `honcho_version` injection; deterministic per-`run_id` high-volume sampler via `TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE` (#637)
- Deriver custom instructions: per-workspace/peer guidance threaded into the deriver prompt with a `MAX_CUSTOM_INSTRUCTIONS_TOKENS` budget (default 2000); deriver `MAX_INPUT_TOKENS` raised 23000 → 25000 (#609)
- Configurable embedding dimensions: `EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE` (`auto`/`always`/`never`) controls whether OpenAI `dimensions=` is forwarded (#678)
- New `honcho-cli` package — Python CLI for inspecting and managing peers, sessions, and configuration against a Honcho deployment (#424)
- `HONCHO_API_URL` env var support in the MCP Worker for self-hosted deployments (#575)
- API ID `max_length` increased from 100 to 512 across `WorkspaceCreate`, `PeerCreate`, and `SessionCreate` to align with the DB schema (#684)
- `AttemptPlan` dataclass pins per-retry provider selection across stream-final retries so streaming doesn't bounce back to primary after the tool loop has settled on fallback (#459)
- Gemini JSON-schema sanitizer for `function_declarations` — strips keywords Gemini's validator rejects while preserving semantics for other backends (#459)
### Changed
- All LLM orchestration moved out of `src/utils/clients.py` into `src/llm/` with modules split by responsibility (#459)
- Default `ModelConfig` factories (deriver, summary, dreamer specialists, dialectic levels) normalized with no extra parameters set by default; operators add transport/thinking overrides explicitly (#459)
- OpenAI reasoning-model routing widened to cover `gpt-5.x` and `o1/o3/o4` — these models receive `max_completion_tokens` instead of `max_tokens` (#459)
- Peer card prompts reframed as stable identity markers; induction specialist now opts out of peer card writes so only deduction touches the card (#686)
- Vector store queries no longer fetch embedding vectors — only document metadata is returned, reducing payload size and DB load (pgvector, lancedb, turbopuffer) (#682)
- Langfuse trace metadata now includes `namespace`, `model`, and `provider` so traces can be filtered by deployment slice (#565)
- Deriver: model-aware tokenizer (replaces the previously hardcoded encoding) and explicit guard on empty message content (#647)
- Dialectic level defaults now merge correctly with per-level overrides (#656)
- Default dialectic tool choice switched to `auto` (#630)
- Vector sync given a substantial retry budget to tolerate transient embedding provider outages (#604)
- `AgentToolConclusionsDeletedEvent` payload now carries `levels` (#612)
- Turbopuffer: `InternalServerError` caught and surfaced as a warning rather than a hard failure; vector store sync errors downgraded to warnings (#561)
### Fixed
- `reverse` query parameter is now honored on the v3 workspace list, peer list, workspace-scoped session list, and peer-scoped session list. Honcho SDKs at 2.1.0+ were already sending `reverse=true` for these routes but the server silently ignored it. Ties on `created_at` now fall back to the internal nanoid `id` for stable ordering across pages (#685)
- LLM client factories now receive `base_url` from `LLMSettings` for default providers — operators pointing at OpenAI-compatible proxies via `LLM__OPENAI_BASE_URL` were previously ignored on the default path (#643, fixes #641)
- Internal N+1 query in dialectic agent tool execution — collapsed per-iteration DB lookups into a single fetch (#652)
- Dreamer threshold and time-guard semantics: count filter now includes only `documents.level == 'explicit'` (was inflating threshold via dreamer-created levels and creating a feedback loop); `last_dream_at` write relocated from enqueue to process so duplicate enqueues or failed runs no longer reset the 8-hour time guard (#573)
- Deriver: blank observations are filtered out before embedding (previously triggered noisy embedding calls and persisted empty rows) (#615)
- Surprisal module: filter format corrected from `{"level": levels}` to `{"level": {"in": levels}}` — the prior call silently returned 0 results and made the entire Surprisal phase of the Dream cycle a no-op (#581, fixes #559)
- Removed hardcoded `stop_sequences` override from Deriver `ModelConfig` (was clobbering operator-configured stop sequences) (#587)
- Embedding client: `embed()` now wraps single-string input in an array, restoring compatibility with OpenAI-compatible third-party providers that reject scalar input (#586)
- Docker Compose: deriver service startup gated on the API service healthcheck — prevents races where the deriver starts before the API has run migrations (#689)
- Docker image: `HEALTHCHECK` directive removed from the shared base image; service-level health checks now belong in each service's own configuration (#530)
- Removed strict parameter validation for thinking params on Anthropic and OpenAI transports — was rejecting valid per-transport configs (#686)
- Stream-final retries pin to the `AttemptPlan` that succeeded rather than re-running provider selection through the outer `current_attempt` ContextVar (#459)
- Gemini `cached_content` reuse keys now include `system_instruction` and `tool_config` so cache hits don't cross configurations (#459)
- CrewAI example updated for the latest CrewAI protocol (#631)
### Removed
- `src/utils/clients.py` deleted; its responsibilities are split across `src/llm/registry.py`, `src/llm/credentials.py`, and the backend-specific modules (#459)
- `HEALTHCHECK` directive from the shared Docker image (#530)
</Update>
<Update label="v3.0.6">
### Changed
- Tightened transaction scopes across search, agent tools, queue manager, and webhook delivery to minimize DB connection hold time during external operations (#525)
@ -558,7 +610,17 @@ Welcome to the Honcho changelog! This section documents all notable changes to t
<Tab title="Python SDK">
[Python SDK](https://pypi.org/project/honcho-ai/)
<Update label="v2.1.1 (Current)">
<Update label="v2.1.2 (Current)">
### Added
- `page`, `size`, and `reverse` pagination parameters on `Honcho.workspaces()` and `HonchoAio.workspaces()`, closing the gap from 2.1.0 which added these to other list methods but not to `workspaces()`. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter.
- `peers` parameter on `Honcho.session()` and `HonchoAio.session()` — attach peers to a session at creation time instead of needing a follow-up `session.add_peers()` call. Accepts the same shapes as `Session.add_peers` (peer ID string, `Peer` object, list of either, or tuples with `SessionPeerConfig`).
### Changed
- `WorkspaceCreateParams`, `PeerCreateParams`, and `SessionCreateParams` now accept IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7.
</Update>
<Update label="v2.1.1">
### Fixed
- Broadened HTTP retry logic to cover `httpx.NetworkError` and `httpx.RemoteProtocolError` in addition to `httpx.TimeoutException` and `httpx.ConnectError`, improving resilience against transient network failures
@ -700,7 +762,20 @@ Welcome to the Honcho changelog! This section documents all notable changes to t
<Tab title="TypeScript SDK">
[TypeScript SDK](https://www.npmjs.com/package/@honcho-ai/sdk)
<Update label="v2.1.1 (Current)">
<Update label="v2.1.2 (Current)">
### Added
- `peers` option on `Honcho.session()` — attach peers to a session at creation time instead of needing a follow-up `session.addPeers()` call. Accepts the same `PeerAddition` shape as `session.addPeers()` (peer ID strings, `Peer` objects, arrays of either, or a record with per-peer `observe_me`/`observe_others` config).
### Changed
- ID validation in `validation.ts` now accepts workspace, peer, and session IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7.
### Fixed
- `Honcho.workspaces()` now actually forwards the `reverse` option to the server. The 2.1.0 changelog listed `workspaces()` among the list methods that gained `reverse`, but `client.ts` was missing the field on the params type and request builder, so the option was silently dropped. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter.
</Update>
<Update label="v2.1.1">
### Fixed
- Broadened fetch error retry logic to catch all `TypeError` network failures (connection resets, DNS errors, etc.) instead of only those with `'fetch'` in the message, improving resilience across runtimes (Node, Bun, browsers)

3
docs/docs.json
View File

@ -24,7 +24,7 @@
"navigation": {
"versions": [
{
"version": "v3.0.6",
"version": "v3.0.7",
"api": {
"openapi": ["v3/openapi.json"]
},
@ -110,6 +110,7 @@
{
"group": "Tutorials",
"pages": [
"v3/guides/recipes/unified-memory-setup",
"v3/guides/discord",
"v3/guides/granola",
"v3/guides/telegram",

357
docs/v3/documentation/core-concepts/design-patterns.mdx
View File

@ -5,364 +5,92 @@ icon: "cubes"
---
<Info>
If you're using a coding agent (Claude Code, OpenCode, Cursor, etc.), the **`/honcho-integration` skill** walks you through these decisions interactively. It explores your codebase, interviews you about peers and sessions, and generates the integration code. The patterns below are the same ones the skill uses.
This page covers **how to structure** workspaces, peers, and sessions for real applications. For the conceptual model behind them, start with [Architecture](/v3/documentation/core-concepts/architecture).
Ready to add Honcho to your codebase? The **`/honcho-integration` skill** applies these patterns for you — it explores your code, asks how your peers and sessions should map to your app, and wires in the Honcho SDK. Run it in any coding agent that supports skills (Claude Code, Cursor, and others).
</Info>
## Quick Reference
**Workspaces isolate, peers persist, and sessions scope the active context.**
| Decision | Recommendation |
|----------|---------------|
| How many workspaces? | One per application. Separate per-agent if you need hard data isolation. |
| Who should be a peer? | Any entity you want Honcho to reason about — users, agents, NPCs, students, customers. |
| How should I scope sessions? | Flexible -- per-conversation, per-channel, per-scene, etc. See [Session Design](#session-design) below. |
| Should I set `observe_me: false`? | Yes, for any peer you don't need Honcho to build a representation of — typically assistants or bots with deterministic behavior. |
| Do I need `observe_others`? | Only when different peers need distinct views of the same participant (e.g., games, multi-agent). Most apps can leave it at the default (false). |
| How many workspaces? | One workspace per application, tool, tenant, or collaboration boundary. Split workspaces only when you need hard isolation between products, customers, environments, or agents. |
| When should agents share a workspace? | When agents collaborate over the same product, project, team, user, customer, or game state. Separate them when they should not see or influence each other's memory. |
| Who should be a peer? | Any persistent participant whose messages should be attributed or reasoned about: users, agents, assistants, NPCs, students, or customers. Use one peer for the same entity across sessions and platforms. |
| How should I scope sessions? | Scope sessions to the active interaction: per-conversation, per-channel, per-task run, per-project, per-import, or other bounded context. Reuse a session when local context should keep accumulating. |
| How does cross-session reasoning work? | Session memory stays local to one session. Peer representations accumulate across every session where the peer is included, and `session.context()` becomes cross-session when you include a peer target. |
| Should I set `observe_me: false`? | Yes, for deterministic peers Honcho does not need to model, like bots or tool agents. Still save their messages so other peers have session context. Keep it enabled for users and evolving agents. |
| Do I need `observe_others`? | Only when a peer needs its own perspective on another participant, such as in games, multi-agent systems, or parent/subagent workflows. |
## Workspace Design
Workspaces are the top-level container. Everything inside a workspace (peers, sessions, messages, and all reasoning) is fully isolated from other workspaces.
A workspace is a hard isolation boundary. **Default to one workspace per application,** and split only at a real privacy, compliance, or product boundary (e.g. per-tenant SaaS, or a tool that needs intentionally isolated memory). Agents that collaborate over the same product, user, or game state belong in the *same* workspace so each can retrieve what the others produced.
**One workspace per application** is the most common pattern. Use separate workspaces when you need hard isolation:
| Pattern | When to use |
|---------|-------------|
| Single workspace | Most applications. One product, one environment. |
| Per-tenant | Multi-tenant SaaS where each customer's data must be completely isolated. |
Honcho plugins default to one workspace *per host* (`hermes`, `claude_code`, `cursor`, `opencode`). To unify memory across them, point each at the same workspace — see [Unified Memory Setup](/v3/guides/recipes/unified-memory-setup).
<Info>
If you are using the SDK, it will create a workspace called `default` if no name is specified for `workspace_id`
The SDK creates a workspace called `default` when no `workspace_id` is specified.
</Info>
---
## Peer Design
A peer is any entity that participates in a session. Observation settings control which ones Honcho reasons about.
Give each real-world entity **one** stable peer ID and reuse it everywhere — splitting one entity across `user-web`, `user-discord`, and `user-slack` builds three separate representations. Prefix IDs by source for multi-channel apps (`discord_491827364`), and if a peer goes by multiple names, store the aliases in its peer card with `set_card()` / `setCard()`.
**What makes a good peer?**
- It participates in sessions (a user, an agent, a character, an NPC)
- It persists across sessions
- It changes over time (preferences shift, knowledge grows), or it produces messages you want Honcho to see
**Naming conventions**
Give peers stable, unique identifiers scoped to your application:
<CodeGroup>
```python Python
# Prefix with the source platform for multi-channel apps
peer = honcho.peer("discord_491827364")
peer = honcho.peer("slack_U04ABCDEF")
# Use your own user IDs for backend integrations
peer = honcho.peer("user_abc123")
# Use descriptive names for agents/assistants
peer = honcho.peer("assistant")
peer = honcho.peer("dungeon-master")
```
```typescript TypeScript
// Prefix with the source platform for multi-channel apps
const peer = await honcho.peer("discord_491827364");
const peer = await honcho.peer("slack_U04ABCDEF");
// Use your own user IDs for backend integrations
const peer = await honcho.peer("user_abc123");
// Use descriptive names for agents/assistants
const peer = await honcho.peer("assistant");
const peer = await honcho.peer("dungeon-master");
```
</CodeGroup>
If your Peer represents an entity that may go by multiple different names, such as nicknames indicate that in the Peer Card:
<CodeGroup>
```python Python
peer = honcho.peer("user_abc123")
peer.set_card([
"Name: Alice. Also known as 'Ali' and 'A'.",
"College student, prefers casual tone.",
])
```
```typescript TypeScript
const peer = await honcho.peer("user_abc123");
await peer.setCard([
"Name: Alice. Also known as 'Ali' and 'A'.",
"College student, prefers casual tone.",
]);
```
</CodeGroup>
**When to disable reasoning**
Not every peer needs a representation. Set `observe_me: false` on peers that behave deterministically.
<CodeGroup>
```python Python
from honcho.api_types import PeerConfig
# The assistant doesn't need a representation
assistant = honcho.peer("assistant", configuration=PeerConfig(observe_me=False))
# The user does--this is who you want to understand
user = honcho.peer("user-123", configuration=PeerConfig(observe_me=True))
```
```typescript TypeScript
const assistant = await honcho.peer("assistant", { configuration: { observeMe: false } });
const user = await honcho.peer("user-123", { configuration: { observeMe: true } });
```
</CodeGroup>
<Tip>
For unified context across Honcho plugins, set the same user peer ID (`peerName`) everywhere — that shared ID is what connects memory across Claude Code, Cursor, OpenCode, and your own app. See [Unified Memory Setup](/v3/guides/recipes/unified-memory-setup).
</Tip>
---
## Session Design
Sessions define the temporal boundaries of an interaction. How you scope sessions directly affects how summaries are generated and how context is retrieved.
Sessions define the temporal boundaries of an interaction. How you scope them affects how summaries are generated, how context is retrieved, and when reasoning fires.
**Common session patterns**
| Pattern | Session scoped to | Example |
|---------|-------------------|---------|
| Per-conversation | Each new chat thread | ChatGPT-style UI where each thread is a session |
| Per-conversation | Each new chat thread | ChatGPT or Claude Code style UI where each thread is a session |
| Per-channel | A persistent channel or room | Discord channel, Slack thread |
| Per-interaction | A bounded task or encounter | A support ticket, a game encounter |
| Per-project | A persistent work area | Coding agent memory for one repository |
| Per-import | A batch of external data | Importing emails or documents for a single peer |
**When to create new sessions vs reuse**
Create a **new** session when context resets (new conversation, new day, new topic); **reuse** one when context should keep accumulating (ongoing channel, persistent thread).
- **New session** when the context resets (new conversation, new day, new topic)
- **Reuse session** when context should accumulate (ongoing channel, persistent thread)
<Warning>
**Don't scope sessions too thin.** Honcho only reasons over a peer once it accumulates ~1,000 tokens *within a single session* ([token batching](/v3/documentation/core-concepts/reasoning#token-batching)). Many tiny sessions each stall below that threshold, so low-volume or trickle inputs should append to one ongoing session rather than fragment across many (nothing is lost — it just waits).
</Warning>
**How cross-session reasoning works**
- **Session memory** is local to an interaction — summaries and recent-message context describe only what happened there.
- **Peer memory** (representations) accumulates reasoning across every session the peer is part of.
So you can start a session fresh or pull in a peer's long-term memory. [`session.context()`](/v3/documentation/features/get-context) returns the current session's summary and recent messages; add a [peer target](/v3/documentation/features/get-context#peer-representation-in-context) to fold in that peer's cross-session history.
---
## Application Patterns
### AI Companions
An assistant that remembers the user across sessions and platforms. The [Honcho plugin for OpenClaw](/v3/guides/integrations/openclaw) is a production example--one assistant with memory across WhatsApp, Telegram, Discord, and Slack.
<CodeGroup>
```python Python
from honcho import Honcho
from honcho.api_types import PeerConfig, SessionPeerConfig
honcho = Honcho(workspace_id="my-companion-app")
owner = honcho.peer("owner")
agent = honcho.peer("agent-main", configuration=PeerConfig(observe_me=False))
# Session key = thread + platform → separate histories, shared user memory
session = honcho.session("general-discord")
session.add_peers([
(owner, SessionPeerConfig(observe_me=True, observe_others=False)),
(agent, SessionPeerConfig(observe_me=True, observe_others=True)),
])
session.add_messages([
owner.message("I've been stressed about the move to Portland next month"),
agent.message("Moving is a big deal. What's weighing on you the most?"),
owner.message("Honestly just leaving my friend group behind"),
])
# Query from any session or platform
response = owner.chat("What's going on in this user's life right now?")
```
```typescript TypeScript
const honcho = new Honcho({ workspaceId: "my-companion-app" });
const owner = await honcho.peer("owner");
const agent = await honcho.peer("agent-main", { configuration: { observeMe: false } });
const session = await honcho.session("general-discord");
await session.addPeers([
["owner", { observeMe: true, observeOthers: false }],
["agent-main", { observeMe: true, observeOthers: true }],
]);
await session.addMessages([
owner.message("I've been stressed about the move to Portland next month"),
agent.message("Moving is a big deal. What's weighing on you the most?"),
owner.message("Honestly just leaving my friend group behind"),
]);
const response = await owner.chat("What's going on in this user's life right now?");
```
</CodeGroup>
**Key decisions (from the [OpenClaw plugin](/v3/guides/integrations/openclaw)):**
- **Session key = thread + platform** — `general-discord` and `general-telegram` are separate sessions but share a single owner representation, so Honcho learns from every channel
- **Dynamic agent peers** — each agent gets its own peer (`agent-{id}`), resolved via a workspace-level map. Renaming an agent recovers the peer by metadata lookup
- **Subagent hierarchy** — when a primary agent spawns a subagent, the parent joins the child's session as a silent observer (`observe_me: false, observe_others: true`), giving Honcho visibility into the full agent tree
- **Asymmetric observation** — both owner and agent are observed, but with different scopes: owner has `observe_others: false` (default view), while the agent has `observe_others: true` so it can build its own representation of the owner. Subagents get lighter context (peer card only, no session summary)
See the [OpenClaw integration guide](/v3/guides/integrations/openclaw) for the full plugin setup.
---
### Coding Agents
Coding agents survive terminal restarts, editor switches, and project hops. The [Honcho plugin for Claude Code](/v3/guides/integrations/claude-code) is a production example of this pattern.
<CodeGroup>
```python Python
from honcho import Honcho
from honcho.api_types import PeerConfig
honcho = Honcho(workspace_id="claude_code")
# Developer is observed; agent is not
developer = honcho.peer("user")
agent = honcho.peer("claude", configuration=PeerConfig(observe_me=False))
# Session per project directory -- stable across restarts
session = honcho.session("user-honcho-repo")
session.add_peers([developer, agent])
session.add_messages([
developer.message("refactor the auth module to use dependency injection"),
agent.message("I'll extract the auth dependencies into a provider pattern..."),
developer.message("actually let's keep it simpler, just pass the config directly"),
])
# In a future session, query what Honcho learned
context = developer.chat("What are this developer's preferences for code architecture?")
# Honcho knows: prefers simplicity, reverses decisions when simpler approach exists
```
```typescript TypeScript
const honcho = new Honcho({ workspaceId: "claude_code" });
const developer = await honcho.peer("user");
const agent = await honcho.peer("claude", { configuration: { observeMe: false } });
const session = await honcho.session("user-honcho-repo");
await session.addPeers([developer, agent]);
await session.addMessages([
developer.message("refactor the auth module to use dependency injection"),
agent.message("I'll extract the auth dependencies into a provider pattern..."),
developer.message("actually let's keep it simpler, just pass the config directly"),
]);
const context = await developer.chat("What are this developer's preferences for code architecture?");
```
</CodeGroup>
**Key decisions (from the Claude Code plugin):**
- **One workspace per tool** -- Claude Code and Cursor each get their own workspace, with optional cross-linking for read access
- **Asymmetric peers** -- developer is observed (memory formation), agent is not observed but still stores messages so Honcho sees both sides
- **Session-per-directory** by default -- each project accumulates its own memory. Prefix with peer name (`user-honcho-repo`) so multiple developers on the same workspace don't collide. Alternative strategies: `git-branch` (session switches on branch change) or `chat-instance` (clean slate each time)
- **Filter what you store** -- user messages go in real-time; agent messages are filtered to skip trivial tool output and keep substantive explanations
- **Import external data** with single-peer sessions to ingest READMEs, architecture docs, or commit history into a developer's representation
See the [Claude Code integration guide](/v3/guides/integrations/claude-code) for the full plugin setup.
---
### Games
Games introduce multi-peer scenarios where **information asymmetry matters**. An NPC should only know what it has witnessed, not the full game state.
<CodeGroup>
```python Python
from honcho import Honcho
from honcho.api_types import SessionPeerConfig
honcho = Honcho(workspace_id="my-rpg")
# Every character is a peer
player = honcho.peer("player-one")
merchant = honcho.peer("merchant-grim")
thief = honcho.peer("thief-shadow")
# Scene 1: Player talks to the merchant
tavern = honcho.session("tavern-scene")
tavern.add_peers([player, merchant])
# Enable the merchant to build its own representation of the player
tavern.set_peer_configuration(merchant, SessionPeerConfig(observe_others=True))
tavern.add_messages([
player.message("I'm looking for a rare gemstone. Money is no object."),
merchant.message("I may know of one... but it won't come cheap."),
])
# Scene 2: Player talks to the thief (merchant isn't here)
alley = honcho.session("dark-alley")
alley.add_peers([player, thief])
alley.set_peer_configuration(thief, SessionPeerConfig(observe_others=True))
alley.add_messages([
player.message("I need that gemstone stolen from the merchant. Quietly."),
thief.message("Consider it done. Half up front."),
])
# The merchant's view of the player: wealthy buyer seeking a gemstone
merchant_view = merchant.chat("What do I know about this player?", target="player-one")
# The thief's view: someone willing to steal from the merchant
thief_view = thief.chat("What do I know about this player?", target="player-one")
# Honcho's global view: knows both sides of the story
full_view = player.chat("What is this player up to?")
```
```typescript TypeScript
const honcho = new Honcho({ workspaceId: "my-rpg" });
const player = await honcho.peer("player-one");
const merchant = await honcho.peer("merchant-grim");
const thief = await honcho.peer("thief-shadow");
const tavern = await honcho.session("tavern-scene");
await tavern.addPeers([player, merchant]);
await tavern.setPeerConfiguration(merchant, { observeOthers: true });
await tavern.addMessages([
player.message("I'm looking for a rare gemstone. Money is no object."),
merchant.message("I may know of one... but it won't come cheap."),
]);
const alley = await honcho.session("dark-alley");
await alley.addPeers([player, thief]);
await alley.setPeerConfiguration(thief, { observeOthers: true });
await alley.addMessages([
player.message("I need that gemstone stolen from the merchant. Quietly."),
thief.message("Consider it done. Half up front."),
]);
const merchantView = await merchant.chat("What do I know about this player?", { target: "player-one" });
const thiefView = await thief.chat("What do I know about this player?", { target: "player-one" });
const fullView = await player.chat("What is this player up to?");
```
</CodeGroup>
**Key decisions:**
- Every character (player, NPC) is a peer
- `observe_others: true` lets NPCs build their own representations of the player based only on what they've witnessed
- Session-per-scene or session-per-encounter so context scopes to specific interactions
- Use `target` when querying to get a specific NPC's perspective rather than Honcho's omniscient view
- See [Representation Scopes](/v3/documentation/features/advanced/representation-scopes) for the full details
---
## Common Mistakes
- **Splitting one identity across peer IDs** -- If the same user is `alice`, `alice-discord`, and `alice-cursor`, Honcho builds separate representations. Use one stable peer ID when you want unified memory.
- **Too many tiny sessions** -- Summaries and recent messages are session-scoped, and reasoning only fires past ~1,000 tokens per session. Splitting a continuous conversation across many sessions fragments local context and can stall reasoning. Reuse a session when context should flow continuously.
- **Separating agents that should collaborate** -- If agents need shared product, customer, or team context, put them in the same workspace. Separate workspaces are hard isolation boundaries.
- **Leaving `observe_me` on for assistants** -- Wastes reasoning compute on a peer you control. Deterministic behavior doesn't need to be modeled.
- **Not storing messages** -- Honcho reasons about messages asynchronously. If you don't call `add_messages()`, there's nothing to reason about — no messages means no memory. See [Storing Data](/v3/documentation/features/storing-data) for details.
- **Creating a new workspace per user** -- Use peers within a single workspace instead. Workspaces are for isolation between applications, not between users.
- **Too many tiny sessions** -- Summaries and `session.context()` are scoped to a single session. If you split a continuous conversation across many sessions, context is fragmented and each session is too short to summarize. Reuse a session when context should flow continuously.
- **Turning on `observe_others` everywhere** -- Directional representations are powerful, but they add complexity. Use them when peers need distinct perspectives, not just because a session has multiple peers.
- **Forgetting `peer_target` on session context** -- `session.context()` defaults to the active session's summary and recent messages, which are session-scoped. It becomes cross-session only through adding a peer_target which includes the peer representation.
- **Blocking on processing** -- Messages are processed asynchronously in the background. Don't poll or wait for reasoning to complete before continuing your application flow.
## Next Steps
<CardGroup cols={2}>
<Card title="Unified Memory Setup" icon="diagram-project" href="/v3/guides/recipes/unified-memory-setup">
Wire these patterns into one shared workspace across four integrations
</Card>
<Card title="Get Context" icon="messages" href="/v3/documentation/features/get-context">
Retrieve formatted context from sessions for your LLM
</Card>
@ -372,7 +100,4 @@ const fullView = await player.chat("What is this player up to?");
<Card title="Reasoning Configuration" icon="wrench" href="/v3/documentation/features/advanced/reasoning-configuration">
Fine-tune what gets reasoned about and how
</Card>
<Card title="Representation Scopes" icon="circle" href="/v3/documentation/features/advanced/representation-scopes">
Directional representations for multi-peer scenarios
</Card>
</CardGroup>

9
docs/v3/guides/overview.mdx
View File

@ -7,6 +7,15 @@ icon: 'puzzle-piece'
Honcho plugs into whatever you're already building. Add memory to an AI assistant, connect an external data source, wire Honcho into your agent framework, or migrate from another provider.
## Recipes
Compose the core primitives across multiple integrations:
<CardGroup cols={2}>
<Card title="Unified Memory Setup" icon="diagram-project" href="/v3/guides/recipes/unified-memory-setup">
One shared workspace across a chat companion, coding agent, autonomous agent, and ingestion job
</Card>
</CardGroup>
## AI Assistants
Add persistent memory to AI assistants and agents:

186
docs/v3/guides/recipes/unified-memory-setup.mdx Normal file
View File

@ -0,0 +1,186 @@
---
title: "Unified Memory Setup"
sidebarTitle: "Unified Memory"
icon: "diagram-project"
description: "Wire one shared Honcho workspace across a chat companion, a coding agent, an autonomous agent, and a scheduled ingestion job"
---
<Info>
This is a how-to, not an intro. It assumes you know what workspaces, peers, and
sessions are. If you don't, start with [Core Concepts](/v3/documentation/core-concepts/).
</Info>
This guide wires four integration points into a single Honcho setup: a
chat companion (Discord/Slack), a coding agent (Claude Code), an autonomous agent
(Hermes), and a cron job that ingests external data. They share **one workspace** and
**one peer** for the user, so everything Honcho learns about your user in one place is
available everywhere else. Each section below notes the per-host setup.
---
## 1. Chat companion (Discord / Slack)
**One session per conversation surface, one peer per participant.** The channel, thread,
or DM is the session; everyone who speaks in it gets their own peer:
- Channel → `discord-channel-{channel_id}`
- Thread → `discord-thread-{thread_id}`
- DM → `discord-dm-{user_id}`
Derive each peer ID from the immutable platform ID (`discord-{user_id}`), not the
display name (which can change). Keep the display name in peer metadata instead. A shared
channel then naturally holds several human peers in one session, with the bot joining
as its own peer (everyone observed on defaults):
```python
session = honcho.session(f"discord-channel-{channel_id}")
session.add_peers([user, assistant]) # plus any other humans in the channel
```
Slack mirrors this with `slack_{user_id}` peers and `slack-{channel}` sessions. For a
full bot walkthrough — message ingestion, watchlists, and storing turns — see the
[Discord guide](/v3/guides/discord).
---
## 2. Coding agent (Claude Code)
**Scope sessions per project directory, prefixed with the user** — `{USER_PEER_ID}-{repo_name}`
— so multiple developers sharing the workspace don't collide on a session ID. Switch
to a `git-branch` scope only when each branch is genuinely a separate line of work.
The Claude Code plugin reads its workspace and peers from `.honcho/config.json`. Point
each host at the same `workspace` and use the same top-level `peerName`, so every host
attributes you to one peer:
```json .honcho/config.json
{
"peerName": "your-user-id",
"hosts": {
"claude_code": { "workspace": "my-product", "aiPeer": "claude" },
"opencode": { "workspace": "my-product", "aiPeer": "opencode" }
}
}
```
<Note>
This is a minimal, illustrative snippet — the real config file carries more fields
(session maps, recall mode, observation strategy, etc.). See the [integration](/v3/guides/overview/)
guides for the full schema and per-host options.
</Note>
Add the user peer and the `claude` agent peer (no special observation config needed),
then store turns — stripping `tool_use` blocks from the assistant message so only
substantive explanation lands in the session.
Because this uses the **same user peer** as the companion, a preference the user
states while coding ("keep it simple, pass config directly") is queryable from the
Discord bot via `user.chat(...)`, and vice versa — both write to the same peer
representation.
<Warning>
**A shared workspace is not the default.** The Honcho plugins — Claude Code, OpenCode,
Hermes, Cursor — each default to a *per-host* workspace (`Claude_Code`, `hermes`, …),
keeping memory isolated per tool. The unification above only happens when you set the
same workspace **and** the same user peer across all of them; otherwise each builds its
own separate representation.
</Warning>
---
## 3. Autonomous agent (Hermes)
The Hermes Honcho plugin is configured through `honcho.json`, set its `workspace`
and `aiPeer` there. See the [Hermes guide](/v3/guides/integrations/hermes) for the full config schema.
- **Sessions** follow a `session_strategy` (default `per-directory`, like the coding
agent above; `per-repo` or `per-session` for a fresh Honcho session each run). The
user peer defaults to `user-{channel}-{chat_id}` unless you pin a `peerName`.
- **Observation** defaults to `directional` — both the user and the `hermes` agent
peer are observed, consistent with the defaults above, so Hermes builds a
representation of itself as well as the user.
- Hermes exposes Honcho as agent **tools** (`honcho_reasoning` for synthesized
answers, plus lighter `honcho_search` and `honcho_context` lookups) and decides when
to call them mid-task. Unlike other sources, you write no retrieval code —
the agent pulls cross-session context on its own.
---
## 4. Scheduled data ingestion (cron)
A scheduled job feeds external data (emails, meeting notes, CRM records) into Honcho.
Attribute the messages to the peer the data is *about* — not to an agent — and group
them into a session. **How you scope that session is the main decision here**, because
it controls when Honcho reasons over the data (more on that below).
```python
from datetime import datetime, timezone
session = honcho.session(f"email-import-{datetime.now(timezone.utc):%Y-%m-%d}")
session.add_peers([user])
messages = [
user.message(
f"Subject: {e['subject']}\nFrom: {e['from']}\n\n{e['body']}",
metadata={"source": "gmail", "thread_id": e["thread_id"]},
created_at=e["timestamp"], # the event's time, NOT import time
)
for e in emails
]
# add_messages accepts at most 100 messages per call — split into requests of 100
for i in range(0, len(messages), 100):
session.add_messages(messages[i:i + 100])
```
Honcho only reasons over a peer once it accumulates ~1,000 tokens *within a single session*
([token batching](/v3/documentation/core-concepts/reasoning#token-batching)). Scope
the session to the volume you ingest:
- **High-volume runs** (a day of emails, a CRM export) clear the threshold easily — a
per-run session like `email-import-{date}` is fine.
- **Low-volume or trickle imports** (a few short records at a time) should append to
one **ongoing per-source session** (e.g. `email-import-gmail`), so content
accumulates across runs instead of fragmenting into thin sessions that each stall
below the threshold (nothing is lost — it just waits).
The [Gmail](/v3/guides/gmail) and [Granola](/v3/guides/granola) guides are related
import examples.
<Tip>
If a cron run also posts as a deterministic agent (a bot or tool agent whose behavior
you fully control), set `observe_me=False` on that peer so Honcho doesn't spend
reasoning modeling it. Its messages still land in the session for context.
```python
agent = honcho.peer("cron_agent", configuration=PeerConfig(observe_me=False))
```
</Tip>
---
## What you end up with
From any integration, the same call — `user.chat("What is this user working on, and
what do they care about?")` — draws on all four sources at once: Discord chats, coding
decisions, Hermes task runs, and imported emails. They blend because:
- **One workspace and one user peer**, so the representation accumulates in one place
instead of fragmenting into `user-discord`, `user-cursor`, etc.
- **Sessions scoped to the live interaction** (channel, repo, task run, import batch),
so local context stays coherent while the user peer carries the long view.
## Next Steps
<CardGroup cols={2}>
<Card title="Design Patterns" icon="cubes" href="/v3/documentation/core-concepts/design-patterns">
The reasoning behind every decision in this guide.
</Card>
<Card title="Get Context" icon="messages" href="/v3/documentation/features/get-context">
Pull session + cross-session context into your LLM calls.
</Card>
<Card title="Granola" icon="microphone" href="/v3/guides/granola">
An interactive import using the per-import session and created_at patterns.
</Card>
<Card title="OpenClaw" icon="lobster" href="/v3/guides/integrations/openclaw">
Production multi-platform companion with parent/subagent tracking.
</Card>
</CardGroup>

2306
docs/v3/openapi.json
View File

File diff suppressed because it is too large Load Diff

2
pyproject.toml
View File

@ -1,6 +1,6 @@
[project]
name = "honcho"
version = "3.0.6"
version = "3.0.7"
description = "Honcho Server"
authors = [
{name = "Plastic Labs", email = "hello@plasticlabs.ai"},

11
sdks/python/CHANGELOG.md
View File

@ -5,6 +5,17 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/).
## [2.1.2] - 2026-05-21
### Added
- `page`, `size`, and `reverse` pagination parameters on `Honcho.workspaces()` and `HonchoAio.workspaces()`, closing the gap from 2.1.0 which added these to `peers()`, `sessions()`, `messages()`, and `conclusions.list()` but not to `workspaces()`. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter.
- `peers` parameter on `Honcho.session()` and `HonchoAio.session()` — attach peers to a session at creation time instead of needing a follow-up `session.add_peers()` call. Accepts the same shapes as `Session.add_peers` (peer ID string, `Peer` object, list of either, or tuples with `SessionPeerConfig`).
### Changed
- `WorkspaceCreateParams`, `PeerCreateParams`, and `SessionCreateParams` now accept IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7.
## [2.1.1] - 2026-04-01
### Fixed

2
sdks/python/pyproject.toml
View File

@ -1,6 +1,6 @@
[project]
name = "honcho-ai"
version = "2.1.1"
version = "2.1.2"
description = "Official DX Optimized Python SDK for Honcho"
dynamic = ["readme"]
license = "Apache-2.0"

48
sdks/python/src/honcho/aio.py
View File

@ -92,24 +92,30 @@ class HonchoAio(AsyncMetadataConfigMixin):
_honcho: "Honcho"
def __init__(self, honcho: "Honcho") -> None:
"""Create an async view backed by a sync Honcho client."""
self._honcho = honcho
# AsyncMetadataConfigMixin implementation
def _get_async_http_client(self):
"""Return the async HTTP client used by metadata helpers."""
return self._honcho._async_http_client
def _get_fetch_route(self) -> str:
"""Return the workspace fetch route for metadata helpers."""
return routes.workspaces()
def _get_update_route(self) -> str:
"""Return the workspace update route for metadata helpers."""
return routes.workspace(self._honcho.workspace_id)
def _get_fetch_body(self) -> dict[str, Any]:
"""Return the request body used to fetch this workspace."""
return {"id": self._honcho.workspace_id}
def _parse_response(
self, data: dict[str, Any]
) -> tuple[dict[str, object], dict[str, object]]:
"""Parse workspace metadata and configuration from an API response."""
workspace = WorkspaceResponse.model_validate(data)
# Return configuration as dict for mixin compatibility
return workspace.metadata or {}, workspace.configuration.model_dump(
@ -117,18 +123,22 @@ class HonchoAio(AsyncMetadataConfigMixin):
)
def _set_metadata(self, metadata: dict[str, object]) -> None:
"""Update cached workspace metadata on the parent client."""
self._honcho._metadata = metadata
def _set_configuration(self, configuration: dict[str, object]) -> None:
"""Update cached workspace configuration on the parent client."""
# Convert dict to typed configuration
self._honcho._configuration = WorkspaceConfiguration.model_validate(
configuration
)
def _get_metadata(self) -> dict[str, object]:
"""Return cached workspace metadata from the parent client."""
return self._honcho._metadata or {}
def _get_configuration(self) -> dict[str, object]:
"""Return cached workspace configuration from the parent client."""
if self._honcho._configuration is None:
return {}
return self._honcho._configuration.model_dump(exclude_none=True)
@ -216,6 +226,7 @@ class HonchoAio(AsyncMetadataConfigMixin):
)
def transform(peer: PeerResponse) -> Peer:
"""Convert a peer API response into a Peer SDK object."""
return Peer(
peer.id,
self._honcho,
@ -225,6 +236,7 @@ class HonchoAio(AsyncMetadataConfigMixin):
)
async def fetch_next(next_page: int) -> AsyncPage[PeerResponse, Peer]:
"""Fetch the next page while preserving filters and ordering."""
next_query: dict[str, Any] = {"page": next_page, "size": size}
if reverse:
next_query["reverse"] = "true"
@ -243,6 +255,14 @@ class HonchoAio(AsyncMetadataConfigMixin):
*,
metadata: dict[str, object] | None = None,
configuration: SessionConfiguration | None = None,
peers: str
| PeerBase
| tuple[str, SessionPeerConfig]
| tuple[PeerBase, SessionPeerConfig]
| list[PeerBase | str]
| list[tuple[PeerBase | str, SessionPeerConfig]]
| list[PeerBase | str | tuple[PeerBase | str, SessionPeerConfig]]
| None = None,
) -> Session:
"""
Get or create a session with the given ID asynchronously.
@ -251,6 +271,9 @@ class HonchoAio(AsyncMetadataConfigMixin):
id: Unique identifier for the session within the workspace.
metadata: Optional metadata dictionary to associate with this session.
configuration: Optional configuration to set for this session.
peers: Optional peers to attach to the session at creation. Accepts the
same shape as Session.add_peers (peer ID string, Peer object, list
of either, or tuples with SessionPeerConfig).
Returns:
A Session object with cached values from the API response.
@ -261,6 +284,8 @@ class HonchoAio(AsyncMetadataConfigMixin):
body["metadata"] = metadata
if configuration is not None:
body["configuration"] = configuration.model_dump(exclude_none=True)
if peers is not None:
body["peers"] = normalize_peers_to_dict(peers)
data = await self._honcho._async_http_client.post(
routes.sessions(self._honcho.workspace_id), body=body
@ -305,6 +330,7 @@ class HonchoAio(AsyncMetadataConfigMixin):
)
def transform(session: SessionResponse) -> Session:
"""Convert a session API response into a Session SDK object."""
return Session(
session.id,
self._honcho,
@ -315,6 +341,7 @@ class HonchoAio(AsyncMetadataConfigMixin):
)
async def fetch_next(next_page: int) -> AsyncPage[SessionResponse, Session]:
"""Fetch the next page while preserving filters and ordering."""
next_query: dict[str, Any] = {"page": next_page, "size": size}
if reverse:
next_query["reverse"] = "true"
@ -328,22 +355,37 @@ class HonchoAio(AsyncMetadataConfigMixin):
return AsyncPage(data, SessionResponse, transform, fetch_next)
async def workspaces(
self, filters: dict[str, object] | None = None
self,
filters: dict[str, object] | None = None,
*,
page: int = 1,
size: int = 50,
reverse: bool = False,
) -> AsyncPage[WorkspaceResponse, str]:
"""Get all workspace IDs asynchronously."""
query: dict[str, Any] = {"page": page, "size": size}
if reverse:
query["reverse"] = "true"
data = await self._honcho._async_http_client.post(
routes.workspaces_list(),
body={"filters": filters} if filters else None,
query=query,
)
def transform(workspace: WorkspaceResponse) -> str:
"""Convert a workspace API response into its workspace ID."""
return workspace.id
async def fetch_next(page: int) -> AsyncPage[WorkspaceResponse, str]:
async def fetch_next(next_page: int) -> AsyncPage[WorkspaceResponse, str]:
"""Fetch the next page while preserving filters and ordering."""
next_query: dict[str, Any] = {"page": next_page, "size": size}
if reverse:
next_query["reverse"] = "true"
next_data = await self._honcho._async_http_client.post(
routes.workspaces_list(),
body={"filters": filters} if filters else None,
query={"page": page},
query=next_query,
)
return AsyncPage(next_data, WorkspaceResponse, transform, fetch_next)

55
sdks/python/src/honcho/client.py
View File

@ -17,6 +17,7 @@ from .api_types import (
PeerResponse,
QueueStatusResponse,
SessionConfiguration,
SessionPeerConfig,
SessionResponse,
WorkspaceConfiguration,
WorkspaceResponse,
@ -28,7 +29,7 @@ from .mixins import MetadataConfigMixin
from .pagination import SyncPage
from .peer import Peer
from .session import Session
from .utils import resolve_id
from .utils import normalize_peers_to_dict, resolve_id
logger = logging.getLogger(__name__)
@ -82,20 +83,25 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
# MetadataConfigMixin implementation
def _get_http_client(self):
"""Return the sync HTTP client used by metadata helpers."""
return self._http
def _get_fetch_route(self) -> str:
"""Return the workspace fetch route for metadata helpers."""
return routes.workspaces()
def _get_update_route(self) -> str:
"""Return the workspace update route for metadata helpers."""
return routes.workspace(self.workspace_id)
def _get_fetch_body(self) -> dict[str, Any]:
"""Return the request body used to fetch this workspace."""
return {"id": self.workspace_id}
def _parse_response(
self, data: dict[str, Any]
) -> tuple[dict[str, object], dict[str, object]]:
"""Parse workspace metadata and configuration from an API response."""
workspace = WorkspaceResponse.model_validate(data)
# Return configuration as dict for mixin compatibility
return workspace.metadata or {}, workspace.configuration.model_dump(
@ -364,6 +370,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
)
def transform(peer: PeerResponse) -> Peer:
"""Convert a peer API response into a Peer SDK object."""
return Peer(
peer.id,
self,
@ -373,6 +380,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
)
def fetch_next(next_page: int) -> SyncPage[PeerResponse, Peer]:
"""Fetch the next page while preserving filters and ordering."""
next_query: dict[str, Any] = {"page": next_page, "size": size}
if reverse:
next_query["reverse"] = "true"
@ -400,6 +408,17 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
None,
description="Optional configuration to set for this session. If set, will get/create session immediately with flags.",
),
peers: str
| PeerBase
| tuple[str, SessionPeerConfig]
| tuple[PeerBase, SessionPeerConfig]
| list[PeerBase | str]
| list[tuple[PeerBase | str, SessionPeerConfig]]
| list[PeerBase | str | tuple[PeerBase | str, SessionPeerConfig]]
| None = Field(
None,
description="Optional peers to attach to the session at creation. Accepts the same shape as Session.add_peers.",
),
) -> Session:
"""
Get or create a session with the given ID.
@ -411,6 +430,9 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
id: Unique identifier for the session within the workspace.
metadata: Optional metadata dictionary to associate with this session.
configuration: Optional configuration to set for this session.
peers: Optional peers to attach to the session at creation. Accepts the
same shape as Session.add_peers (peer ID string, Peer object, list
of either, or tuples with SessionPeerConfig).
Returns:
A Session object with cached metadata, configuration, created_at, and is_active.
@ -421,6 +443,8 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
body["metadata"] = metadata
if configuration is not None:
body["configuration"] = configuration.model_dump(exclude_none=True)
if peers is not None:
body["peers"] = normalize_peers_to_dict(peers)
data = self._http.post(routes.sessions(self.workspace_id), body=body)
session_data = SessionResponse.model_validate(data)
@ -466,6 +490,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
)
def transform(session: SessionResponse) -> Session:
"""Convert a session API response into a Session SDK object."""
return Session(
session.id,
self,
@ -476,6 +501,7 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
)
def fetch_next(next_page: int) -> SyncPage[SessionResponse, Session]:
"""Fetch the next page while preserving filters and ordering."""
next_query: dict[str, Any] = {"page": next_page, "size": size}
if reverse:
next_query["reverse"] = "true"
@ -489,7 +515,12 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
return SyncPage(data, SessionResponse, transform, fetch_next)
def workspaces(
self, filters: dict[str, object] | None = None
self,
filters: dict[str, object] | None = None,
*,
page: int = 1,
size: int = 50,
reverse: bool = False,
) -> SyncPage[WorkspaceResponse, str]:
"""
Get all workspace IDs from the Honcho instance.
@ -497,22 +528,38 @@ class Honcho(BaseModel, MetadataConfigMixin): # pyright: ignore[reportUnsafeMul
Makes an API call to retrieve all workspace IDs that the authenticated
user has access to.
Args:
filters: Optional filter criteria.
page: Page number (1-indexed). Default: 1.
size: Number of items per page. Default: 50.
reverse: If True, reverses the default ordering. Default: False.
Returns:
A paginated SyncPage of workspace ID strings
"""
query: dict[str, Any] = {"page": page, "size": size}
if reverse:
query["reverse"] = "true"
data = self._http.post(
routes.workspaces_list(),
body={"filters": filters} if filters else None,
query=query,
)
def transform(workspace: WorkspaceResponse) -> str:
"""Convert a workspace API response into its workspace ID."""
return workspace.id
def fetch_next(page: int) -> SyncPage[WorkspaceResponse, str]:
def fetch_next(next_page: int) -> SyncPage[WorkspaceResponse, str]:
"""Fetch the next page while preserving filters and ordering."""
next_query: dict[str, Any] = {"page": next_page, "size": size}
if reverse:
next_query["reverse"] = "true"
next_data = self._http.post(
routes.workspaces_list(),
body={"filters": filters} if filters else None,
query={"page": page},
query=next_query,
)
return SyncPage(next_data, WorkspaceResponse, transform, fetch_next)

14
sdks/typescript/CHANGELOG.md
View File

@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/).
## [2.1.2] - 2026-05-21
### Added
- `peers` option on `Honcho.session()` — attach peers to a session at creation time instead of needing a follow-up `session.addPeers()` call. Accepts the same `PeerAddition` shape as `session.addPeers()` (peer ID strings, `Peer` objects, arrays of either, or a record with per-peer `observe_me`/`observe_others` config).
### Changed
- ID validation in `validation.ts` now accepts workspace, peer, and session IDs up to 512 characters (was 100), matching the server-side schema change in Honcho v3.0.7.
### Fixed
- `Honcho.workspaces()` now actually forwards the `reverse` option to the server. The 2.1.0 changelog listed `workspaces()` among the list methods that gained `reverse`, but `client.ts` was missing the field on the params type and request builder, so the option was silently dropped. Honoring `reverse` on the workspace/peer/session list routes also requires a Honcho server with the matching API fix; older servers silently ignore the parameter.
## [2.1.1] - 2026-04-01
### Fixed

36
sdks/typescript/__tests__/session.test.ts
View File

@ -95,6 +95,42 @@ describe('Session', () => {
expect(session1.id).toBe(session2.id)
expect(session2.metadata).toEqual({ version: 2 })
})
test('creates session with peers from string array', async () => {
const session = await client.session('session-with-peers-strings', {
peers: ['create-peer-a', 'create-peer-b'],
})
const peers = await session.peers()
const ids = peers.map((p) => p.id)
expect(ids).toContain('create-peer-a')
expect(ids).toContain('create-peer-b')
})
test('creates session with peers from Peer objects', async () => {
const peerA = await client.peer('create-obj-peer-a')
const peerB = await client.peer('create-obj-peer-b')
const session = await client.session('session-with-peer-objects', {
peers: [peerA, peerB],
})
const peers = await session.peers()
const ids = peers.map((p) => p.id)
expect(ids).toContain('create-obj-peer-a')
expect(ids).toContain('create-obj-peer-b')
})
test('creates session with peers and per-peer config', async () => {
const session = await client.session('session-with-peer-config', {
peers: [
['create-config-peer', { observeMe: true, observeOthers: false }],
],
})
const config = await session.getPeerConfiguration('create-config-peer')
expect(config.observeMe).toBe(true)
expect(config.observeOthers).toBe(false)
})
})
// ===========================================================================

2
sdks/typescript/package.json
View File

@ -1,6 +1,6 @@
{
"name": "@honcho-ai/sdk",
"version": "2.1.1",
"version": "2.1.2",
"description": "Official DX Optimized TypeScript SDK for Honcho",
"author": "Plastic Labs <hello@plasticlabs.ai>",
"license": "Apache-2.0",

29
sdks/typescript/src/client.ts
View File

@ -22,6 +22,8 @@ import {
HonchoConfigSchema,
LimitSchema,
normalizeListOptions,
type PeerAddition,
PeerAdditionToApiSchema,
type PeerConfig,
PeerConfigSchema,
PeerIdSchema,
@ -230,6 +232,7 @@ export class Honcho {
filters?: Record<string, unknown>
page?: number
size?: number
reverse?: boolean
}): Promise<PageResponse<WorkspaceResponse>> {
return this._http.post<PageResponse<WorkspaceResponse>>(
`/${API_VERSION}/workspaces/list`,
@ -240,6 +243,7 @@ export class Honcho {
query: {
page: params?.page,
size: params?.size,
reverse: params?.reverse ? 'true' : undefined,
},
}
)
@ -338,6 +342,10 @@ export class Honcho {
id: string
metadata?: Record<string, unknown>
configuration?: SessionConfig
peers?: Record<
string,
{ observe_me?: boolean | null; observe_others?: boolean | null }
>
}
): Promise<SessionResponse> {
return this._http.post<SessionResponse>(
@ -347,6 +355,7 @@ export class Honcho {
id: params.id,
metadata: params.metadata,
configuration: sessionConfigToApi(params.configuration),
peers: params.peers,
},
}
)
@ -488,10 +497,13 @@ export class Honcho {
* @param id - Unique identifier for the session within the workspace. Should be a
* stable identifier that can be used consistently to reference the
* same conversation
* @param metadata - Optional metadata dictionary to associate with this session.
* @param options.metadata - Optional metadata dictionary to associate with this session.
* If set, will get/create session immediately with metadata.
* @param configuration - Optional configuration to set for this session.
* @param options.configuration - Optional configuration to set for this session.
* If set, will get/create session immediately with flags.
* @param options.peers - Optional peers to attach to the session at creation.
* Accepts the same shape as `session.addPeers()` (peer ID strings,
* Peer objects, arrays of either, or a record with per-peer config).
* @returns Promise resolving to a Session object that can be used to add peers,
* send messages, and manage conversation context
* @throws Error if the session ID is empty or invalid
@ -501,6 +513,7 @@ export class Honcho {
options?: {
metadata?: SessionMetadata
configuration?: SessionConfig
peers?: PeerAddition
}
): Promise<Session> {
await this._ensureWorkspace()
@ -511,11 +524,16 @@ export class Honcho {
const validatedConfiguration = options?.configuration
? SessionConfigSchema.parse(options.configuration)
: undefined
const validatedPeers =
options?.peers !== undefined
? PeerAdditionToApiSchema.parse(options.peers)
: undefined
const sessionData = await this._getOrCreateSession(this.workspaceId, {
id: validatedId,
configuration: validatedConfiguration,
metadata: validatedMetadata,
peers: validatedPeers,
})
return new Session(
validatedId,
@ -691,7 +709,7 @@ export class Honcho {
* user has access to.
*
* @param options - Either a legacy raw filter object or an options object with
* `filters`, `page`, and `size`. See
* `filters`, `page`, `size`, and `reverse`. See
* [search filters documentation](https://honcho.dev/docs/v3/documentation/core-concepts/features/using-filters).
* @returns Promise resolving to a Page of workspace ID strings. Returns an empty
* page if no workspaces are accessible or none exist
@ -703,20 +721,24 @@ export class Honcho {
filters?: Filters
page?: number
size?: number
reverse?: boolean
}
): Promise<Page<string, WorkspaceResponse>> {
const normalizedOptions = normalizeListOptions(options, [
'filters',
'page',
'size',
'reverse',
])
const validatedFilter = normalizedOptions.filters
? FilterSchema.parse(normalizedOptions.filters)
: undefined
const reverse = normalizedOptions.reverse
const workspacesPage = await this._listWorkspaces({
filters: validatedFilter,
page: normalizedOptions.page,
size: normalizedOptions.size,
reverse,
})
const fetchNextPage = async (
@ -727,6 +749,7 @@ export class Honcho {
filters: validatedFilter,
page,
size,
reverse,
})
}

5
sdks/typescript/src/types/api.ts
View File

@ -35,6 +35,7 @@ export interface WorkspaceListParams {
filters?: Record<string, unknown>
page?: number
size?: number
reverse?: boolean
}
// =============================================================================
@ -64,6 +65,7 @@ export interface PeerListParams {
filters?: Record<string, unknown>
page?: number
size?: number
reverse?: boolean
}
export interface PeerChatParams {
@ -141,6 +143,7 @@ export interface SessionListParams {
filters?: Record<string, unknown>
page?: number
size?: number
reverse?: boolean
}
export interface SessionCloneParams {
@ -226,6 +229,7 @@ export interface MessageListParams {
filters?: Record<string, unknown>
page?: number
size?: number
reverse?: boolean
}
export interface MessageSearchParams {
@ -262,6 +266,7 @@ export interface ConclusionListParams {
filters?: Record<string, unknown>
page?: number
size?: number
reverse?: boolean
}
export interface ConclusionQueryParams {

33
src/_version.py Normal file
View File

@ -0,0 +1,33 @@
"""Single source of truth for the Honcho service version.
Reads pyproject.toml directly so the value never drifts from the authoritative
source. Falls back to installed package metadata for wheel-only deploys where
pyproject.toml may not be shipped.
Used by:
- src/main.py for the FastAPI app `version=...`
- src/telemetry/emitter.py and src/telemetry/events/base.py for event tagging
"""
from __future__ import annotations
from importlib.metadata import PackageNotFoundError
from importlib.metadata import version as _pkg_version
from pathlib import Path
import tomllib
def _read_version() -> str:
pyproject = Path(__file__).resolve().parent.parent / "pyproject.toml"
try:
with pyproject.open("rb") as f:
return tomllib.load(f)["project"]["version"]
except (OSError, KeyError, tomllib.TOMLDecodeError):
try:
return _pkg_version("honcho")
except PackageNotFoundError:
return "unknown"
HONCHO_VERSION: str = _read_version()

15
src/config.py
View File

@ -1071,6 +1071,21 @@ class TelemetrySettings(HonchoSettings):
# Namespace for instance identification (propagated from top-level NAMESPACE if not set)
NAMESPACE: str | None = None
# Sample rate for high-volume events: llm.call.completed, embedding.call.completed,
# agent.iteration, agent.tool.call.completed. Deterministic on run_id so traces
# remain coherent end-to-end. Aggregate envelopes (RepresentationCompleted,
# DialecticCompleted, DreamRun, etc.) are NEVER sampled — they're calibration
# ground truth.
#
# Design trade-off: at rate < 1.0, aggregate events still emit but their
# high-volume children get dropped. Downstream `JOIN ... ON run_id` queries
# will see parents without complete children — this is intentional (the
# aggregates carry totals; detail events are best-effort), but consumers
# MUST NOT rebuild per-call analytics from the sampled children alone or
# they'll undercount. If you tune this below 1.0, audit dashboards/queries
# that join high-volume events to aggregate envelopes first.
HIGH_VOLUME_SAMPLE_RATE: Annotated[float, Field(default=1.0, ge=0.0, le=1.0)] = 1.0
class CacheSettings(HonchoSettings):
model_config = SettingsConfigDict(env_prefix="CACHE_", extra="ignore") # pyright: ignore

46
src/crud/message.py
View File

@ -12,8 +12,10 @@ from src.config import settings
from src.dependencies import tracked_db
from src.embedding_client import embedding_client
from src.exceptions import VectorStoreError
from src.telemetry.events import EmbeddingCallPurpose
from src.utils.filter import apply_filter
from src.utils.formatting import ILIKE_ESCAPE_CHAR, escape_ilike_pattern
from src.utils.types import embedding_call_purpose
from src.vector_store import VectorRecord, get_external_vector_store
from .session import get_or_create_session
@ -283,11 +285,17 @@ async def create_messages(
for message in message_objects
if message.content and message.content.strip()
}
embedding_dict = (
await embedding_client.batch_embed(id_resource_dict)
if id_resource_dict
else {}
)
if id_resource_dict:
with embedding_call_purpose(
EmbeddingCallPurpose.MESSAGE_CREATE.value,
workspace_name=workspace_name,
parent_category="api",
):
embedding_dict = await embedding_client.batch_embed(
id_resource_dict
)
else:
embedding_dict = {}
external_vector_store = get_external_vector_store()
@ -891,9 +899,17 @@ async def search_messages(
Each snippet may contain multiple matches if they were close together.
Context messages are ordered chronologically and include the matched messages.
"""
query_embedding = (
embedding if embedding is not None else await embedding_client.embed(query)
)
if embedding is not None:
query_embedding = embedding
else:
# Caller didn't precompute; tag this fallback path as search_messages.
# Callers that have a more specific intent should set their own
# context manager before calling and pass the precomputed embedding.
with embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MESSAGES.value,
workspace_name=workspace_name,
):
query_embedding = await embedding_client.embed(query)
return await _semantic_search_messages(
workspace_name,
session_name,
@ -1083,9 +1099,17 @@ async def search_messages_temporal(
List of tuples: (matched_messages, context_messages)
Each snippet may contain multiple matches if they were close together.
"""
query_embedding = (
embedding if embedding is not None else await embedding_client.embed(query)
)
if embedding is not None:
query_embedding = embedding
else:
# Caller didn't precompute; tag this fallback path as search_messages.
# Callers that have a more specific intent should set their own
# context manager before calling and pass the precomputed embedding.
with embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MESSAGES.value,
workspace_name=workspace_name,
):
query_embedding = await embedding_client.embed(query)
return await _semantic_search_messages(
workspace_name,
session_name,

17
src/crud/peer.py
View File

@ -1,3 +1,5 @@
"""CRUD helpers for peer records and peer-scoped session queries."""
from logging import getLogger
from typing import Any
@ -210,13 +212,17 @@ async def get_peer(
async def get_peers(
workspace_name: str,
filters: dict[str, str] | None = None,
filters: dict[str, Any] | None = None,
reverse: bool = False,
) -> Select[tuple[models.Peer]]:
"""Build a filtered peer list query ordered by creation time."""
stmt = select(models.Peer).where(models.Peer.workspace_name == workspace_name)
stmt = apply_filter(stmt, models.Peer, filters)
return stmt.order_by(models.Peer.created_at)
if reverse:
return stmt.order_by(models.Peer.created_at.desc(), models.Peer.id.desc())
return stmt.order_by(models.Peer.created_at.asc(), models.Peer.id.asc())
async def update_peer(
@ -285,6 +291,7 @@ async def get_sessions_for_peer(
workspace_name: str,
peer_name: str,
filters: dict[str, Any] | None = None,
reverse: bool = False,
) -> Select[tuple[models.Session]]:
"""
Get all sessions for a peer through the session_peers relationship.
@ -293,6 +300,7 @@ async def get_sessions_for_peer(
workspace_name: Name of the workspace
peer_name: Name of the peer
filters: Filter sessions by metadata
reverse: Whether to reverse the default creation order
Returns:
SQLAlchemy Select statement
@ -310,6 +318,9 @@ async def get_sessions_for_peer(
stmt = apply_filter(stmt, models.Session, filters)
stmt: Select[tuple[models.Session]] = stmt.order_by(models.Session.created_at)
if reverse:
stmt = stmt.order_by(models.Session.created_at.desc(), models.Session.id.desc())
else:
stmt = stmt.order_by(models.Session.created_at.asc(), models.Session.id.asc())
return stmt

45
src/crud/representation.py
View File

@ -15,6 +15,7 @@ from src.dependencies import tracked_db
from src.dreamer.dream_scheduler import check_and_schedule_dream
from src.embedding_client import embedding_client
from src.schemas import ResolvedConfiguration
from src.telemetry.events import EmbeddingCallPurpose
from src.telemetry.logging import accumulate_metric
from src.utils.formatting import format_datetime_utc
from src.utils.representation import (
@ -22,6 +23,7 @@ from src.utils.representation import (
ExplicitObservation,
Representation,
)
from src.utils.types import embedding_call_purpose
logger = logging.getLogger(__name__)
@ -96,7 +98,14 @@ class RepresentationManager:
observation_texts = [_observation_text(obs) for obs in all_observations]
try:
embeddings = await embedding_client.simple_batch_embed(observation_texts)
with embedding_call_purpose(
EmbeddingCallPurpose.CREATE_OBSERVATIONS.value,
workspace_name=self.workspace_name,
parent_category="representation",
):
embeddings = await embedding_client.simple_batch_embed(
observation_texts
)
except ValueError as e:
raise exceptions.ValidationException(
"Observation content exceeds maximum token limit of "
@ -210,6 +219,8 @@ class RepresentationManager:
semantic_search_max_distance: float | None = None,
include_most_derived: bool = False,
max_observations: int = settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS,
parent_category: str | None = None,
embedding_purpose: EmbeddingCallPurpose = EmbeddingCallPurpose.SEARCH_MEMORY,
) -> Representation:
"""
Get working representation with flexible query options.
@ -224,13 +235,32 @@ class RepresentationManager:
semantic_search_max_distance: Maximum distance for semantic search
include_most_derived: Include most derived observations
max_observations: Maximum total observations to return
parent_category: Optional workflow attribution forwarded to the
fallback embedding call when the caller didn't pre-compute
an embedding (or pre-compute failed).
embedding_purpose: Embedding call_purpose tag to use on the
fallback embed when no pre-computed embedding was supplied.
Defaults to SEARCH_MEMORY; callers whose route-level
precompute uses a more specific purpose (e.g.
SESSION_CONTEXT_SEARCH) should pass that here so the
fallback path lands in the same analytics bucket.
Returns:
Representation combining various query strategies
"""
if include_semantic_query and embedding is None:
with suppress(Exception):
# Best-effort precompute
# Best-effort precompute when caller didn't supply one (or their
# precompute was suppressed). The purpose is parameterized so
# this fallback shows up in the same telemetry bucket as the
# successful path — see embedding_purpose docstring above.
with (
suppress(Exception),
embedding_call_purpose(
embedding_purpose.value,
workspace_name=self.workspace_name,
parent_category=parent_category,
),
):
embedding = await embedding_client.embed(include_semantic_query)
if db is not None:
@ -496,6 +526,8 @@ async def get_working_representation(
semantic_search_max_distance: float | None = None,
include_most_derived: bool = False,
max_observations: int = settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS,
parent_category: str | None = None,
embedding_purpose: EmbeddingCallPurpose = EmbeddingCallPurpose.SEARCH_MEMORY,
) -> Representation:
"""
Get raw working representation data from the relevant document collection.
@ -507,6 +539,11 @@ async def get_working_representation(
db: Optional database session. If provided, uses it directly;
otherwise creates a new session via tracked_db.
embedding: Pre-computed embedding for the semantic query.
parent_category: Workflow attribution forwarded to the fallback
embedding call when no pre-computed embedding was supplied.
embedding_purpose: Embedding call_purpose for the fallback embed;
callers should match it to whatever purpose their route-level
precompute used so failure/retry paths stay in the same bucket.
"""
manager = RepresentationManager(
workspace_name=workspace_name,
@ -522,4 +559,6 @@ async def get_working_representation(
semantic_search_max_distance=semantic_search_max_distance,
include_most_derived=include_most_derived,
max_observations=max_observations,
parent_category=parent_category,
embedding_purpose=embedding_purpose,
)

15
src/crud/session.py
View File

@ -1,3 +1,5 @@
"""CRUD helpers for sessions and session-related relationship data."""
from dataclasses import dataclass
from logging import getLogger
from typing import Any
@ -114,9 +116,18 @@ def count_observers_in_config(
async def get_sessions(
workspace_name: str,
filters: dict[str, Any] | None = None,
reverse: bool = False,
) -> Select[tuple[models.Session]]:
"""
Get all active sessions in a workspace.
Args:
workspace_name: Name of the workspace
filters: Optional filters to apply to the query
reverse: If True, order by created_at descending; if False, ascending
Returns:
Select statement for Session objects
"""
stmt = (
select(models.Session)
@ -126,7 +137,9 @@ async def get_sessions(
stmt = apply_filter(stmt, models.Session, filters)
return stmt.order_by(models.Session.created_at)
if reverse:
return stmt.order_by(models.Session.created_at.desc(), models.Session.id.desc())
return stmt.order_by(models.Session.created_at.asc(), models.Session.id.asc())
async def get_or_create_session(

14
src/crud/workspace.py
View File

@ -1,3 +1,5 @@
"""CRUD helpers for workspace records and workspace deletion checks."""
from dataclasses import dataclass
from logging import getLogger
from typing import Any
@ -154,17 +156,25 @@ async def get_or_create_workspace(
async def get_all_workspaces(
filters: dict[str, Any] | None = None,
reverse: bool = False,
) -> Select[tuple[models.Workspace]]:
"""
Get all workspaces.
Args:
db: Database session
filters: Filter the workspaces by a dictionary of metadata
reverse: Whether to reverse the default creation order
"""
stmt = select(models.Workspace)
stmt = apply_filter(stmt, models.Workspace, filters)
stmt: Select[tuple[models.Workspace]] = stmt.order_by(models.Workspace.created_at)
if reverse:
stmt = stmt.order_by(
models.Workspace.created_at.desc(), models.Workspace.id.desc()
)
else:
stmt = stmt.order_by(
models.Workspace.created_at.asc(), models.Workspace.id.asc()
)
return stmt

191
src/deriver/consumer.py
View File

@ -9,7 +9,7 @@ from src import crud, models
from src.dependencies import tracked_db
from src.deriver.deriver import process_representation_tasks_batch
from src.dreamer import process_dream
from src.exceptions import ResourceNotFoundException
from src.exceptions import ResourceNotFoundException, ValidationException
from src.models import Message
from src.reconciler.queue_cleanup import cleanup_queue_items
from src.reconciler.sync_vectors import run_vector_reconciliation_cycle
@ -158,6 +158,9 @@ async def process_representation_batch(
observers: list[str] | None,
observed: str | None,
queue_item_message_ids: list[int],
hit_batch_token_cap: bool = False,
was_flush_enabled: bool = False,
batch_max_tokens: int = 0,
) -> None:
"""
Prepares and processes a batch of messages for representation tasks.
@ -168,6 +171,9 @@ async def process_representation_batch(
observers: List of observers for the messages
observed: The observed of the messages
queue_item_message_ids: Message IDs from queue items
hit_batch_token_cap: whether the queue batcher clamped this batch to fit
was_flush_enabled: snapshot of DERIVER.FLUSH_ENABLED at fetch time
batch_max_tokens: DERIVER.REPRESENTATION_BATCH_MAX_TOKENS snapshot
"""
if not messages or not messages[0]:
logger.debug("process_representation_batch received no messages")
@ -182,6 +188,9 @@ async def process_representation_batch(
observers=observers,
observed=observed,
queue_item_message_ids=queue_item_message_ids,
hit_batch_token_cap=hit_batch_token_cap,
was_flush_enabled=was_flush_enabled,
batch_max_tokens=batch_max_tokens,
)
@ -218,92 +227,107 @@ async def process_deletion(
workspace_name,
)
async with tracked_db("process_deletion") as db:
if deletion_type == "session":
try:
result = await crud.delete_session(
db, workspace_name=workspace_name, session_name=resource_id
)
messages_deleted = result.messages_deleted
conclusions_deleted = result.conclusions_deleted
logger.info(
"Successfully deleted session %s in workspace %s "
+ "(messages=%d, conclusions=%d)",
resource_id,
workspace_name,
messages_deleted,
conclusions_deleted,
)
except ResourceNotFoundException as e:
# Session not found - may have already been deleted, treat as success
logger.warning(
"Session %s not found during deletion (may already be deleted): %s",
resource_id,
str(e),
)
# try/except/finally so the event ALWAYS fires — both success and
# failure (unsupported type, unexpected CRUD error). Previously the
# unsupported-type branch raised before the emit ran, and unexpected
# CRUD errors bubbled up without telemetry.
try:
async with tracked_db("process_deletion") as db:
if deletion_type == "session":
try:
result = await crud.delete_session(
db, workspace_name=workspace_name, session_name=resource_id
)
messages_deleted = result.messages_deleted
conclusions_deleted = result.conclusions_deleted
logger.info(
"Successfully deleted session %s in workspace %s "
+ "(messages=%d, conclusions=%d)",
resource_id,
workspace_name,
messages_deleted,
conclusions_deleted,
)
except ResourceNotFoundException as e:
# Session not found - may have already been deleted, treat as success
logger.warning(
"Session %s not found during deletion (may already be deleted): %s",
resource_id,
str(e),
)
elif deletion_type == "observation":
try:
await crud.delete_document_by_id(
db, workspace_name=workspace_name, document_id=resource_id
)
conclusions_deleted = 1 # Single observation deleted
logger.info(
"Successfully deleted observation %s in workspace %s",
resource_id,
workspace_name,
)
except ResourceNotFoundException as e:
# Document not found - may have already been deleted, treat as success
logger.warning(
"Observation %s not found during deletion (may already be deleted): %s",
resource_id,
str(e),
)
elif deletion_type == "observation":
try:
await crud.delete_document_by_id(
db, workspace_name=workspace_name, document_id=resource_id
)
conclusions_deleted = 1 # Single observation deleted
logger.info(
"Successfully deleted observation %s in workspace %s",
resource_id,
workspace_name,
)
except ResourceNotFoundException as e:
# Document not found - may have already been deleted, treat as success
logger.warning(
"Observation %s not found during deletion (may already be deleted): %s",
resource_id,
str(e),
)
elif deletion_type == "workspace":
try:
result = await crud.delete_workspace(db, workspace_name=workspace_name)
peers_deleted = result.peers_deleted
sessions_deleted = result.sessions_deleted
messages_deleted = result.messages_deleted
conclusions_deleted = result.conclusions_deleted
logger.info(
"Successfully deleted workspace %s "
+ "(peers=%d, sessions=%d, messages=%d, conclusions=%d)",
workspace_name,
peers_deleted,
sessions_deleted,
messages_deleted,
conclusions_deleted,
)
except ResourceNotFoundException as e:
# Workspace not found - may have already been deleted, treat as success
logger.warning(
"Workspace %s not found during deletion (may already be deleted): %s",
workspace_name,
str(e),
)
elif deletion_type == "workspace":
try:
result = await crud.delete_workspace(
db, workspace_name=workspace_name
)
peers_deleted = result.peers_deleted
sessions_deleted = result.sessions_deleted
messages_deleted = result.messages_deleted
conclusions_deleted = result.conclusions_deleted
logger.info(
"Successfully deleted workspace %s "
+ "(peers=%d, sessions=%d, messages=%d, conclusions=%d)",
workspace_name,
peers_deleted,
sessions_deleted,
messages_deleted,
conclusions_deleted,
)
except ResourceNotFoundException as e:
# Workspace not found - may have already been deleted, treat as success
logger.warning(
"Workspace %s not found during deletion (may already be deleted): %s",
workspace_name,
str(e),
)
else:
success = False
error_message = f"Unsupported deletion type: {deletion_type}"
raise ValueError(error_message)
# Emit telemetry event
emit(
DeletionCompletedEvent(
workspace_name=workspace_name,
deletion_type=deletion_type,
resource_id=resource_id,
success=success,
peers_deleted=peers_deleted,
sessions_deleted=sessions_deleted,
messages_deleted=messages_deleted,
conclusions_deleted=conclusions_deleted,
error_message=error_message,
else:
success = False
error_message = f"Unsupported deletion type: {deletion_type}"
raise ValidationException(error_message)
except Exception as e:
# Catch anything that survived the per-branch `ResourceNotFoundException`
# handling above (incl. the ValueError from the unsupported-type branch).
# Record telemetry, then re-raise so the queue worker still surfaces
# the failure to its caller.
success = False
if error_message is None:
error_message = f"{type(e).__name__}: {e}"
raise
finally:
emit(
DeletionCompletedEvent(
workspace_name=workspace_name,
deletion_type=deletion_type,
resource_id=resource_id,
success=success,
peers_deleted=peers_deleted,
sessions_deleted=sessions_deleted,
messages_deleted=messages_deleted,
conclusions_deleted=conclusions_deleted,
error_message=error_message,
)
)
)
async def process_reconciler(payload: ReconcilerPayload) -> None:
@ -363,6 +387,7 @@ async def process_reconciler(payload: ReconcilerPayload) -> None:
# Emit telemetry event for cleanup stale items
emit(
CleanupStaleItemsCompletedEvent(
queue_items_cleaned=deleted_count,
total_duration_ms=duration_ms,
)
)

59
src/deriver/deriver.py
View File

@ -6,10 +6,12 @@ from src.config import ConfiguredModelSettings, settings
from src.crud.representation import RepresentationManager
from src.dependencies import tracked_db
from src.llm import honcho_llm_call
from src.llm.types import LLMTelemetryContext
from src.models import Message
from src.schemas import ResolvedConfiguration
from src.telemetry import prometheus_metrics
from src.telemetry.events import RepresentationCompletedEvent, emit
from src.telemetry.events.llm import CallPurpose
from src.telemetry.logging import accumulate_metric, log_performance_metrics
from src.telemetry.prometheus.metrics import (
DeriverComponents,
@ -39,6 +41,9 @@ async def process_representation_tasks_batch(
observers: list[str],
observed: str,
queue_item_message_ids: list[int],
hit_batch_token_cap: bool = False,
was_flush_enabled: bool = False,
batch_max_tokens: int = 0,
) -> None:
"""
Process messages with minimal overhead - single LLM call, save to multiple collections.
@ -49,6 +54,9 @@ async def process_representation_tasks_batch(
observers: List of observer peer IDs (collections to save to).
observed: The observed peer ID.
queue_item_message_ids: Message IDs from queue items being processed
hit_batch_token_cap: queue batcher clamped this batch to fit
was_flush_enabled: DERIVER.FLUSH_ENABLED snapshot at batch time
batch_max_tokens: DERIVER.REPRESENTATION_BATCH_MAX_TOKENS snapshot
"""
if not messages:
return
@ -146,6 +154,12 @@ async def process_representation_tasks_batch(
enable_retry=True,
retry_attempts=3,
trace_name="minimal_deriver",
telemetry=LLMTelemetryContext(
workspace_name=latest_message.workspace_name,
call_purpose=CallPurpose.DERIVER_REPRESENTATION.value,
parent_category="representation",
observed=observed,
),
)
llm_duration = (time.perf_counter() - llm_start) * 1000
@ -175,6 +189,7 @@ async def process_representation_tasks_batch(
latest_message.created_at,
)
successful_observer_count = 0
if observations.is_empty() or not message_ids:
logger.warning(
"Deriver generated zero observations for messages %s:%s in %s/%s!",
@ -200,6 +215,7 @@ async def process_representation_tasks_batch(
latest_message.created_at,
message_level_configuration,
)
successful_observer_count += 1
except Exception as e:
logger.error(
"Failed to save representation for observer %s: %s", observer, e
@ -234,12 +250,39 @@ async def process_representation_tasks_batch(
accumulate_metric(
f"minimal_deriver_{latest_message.id}_{observed}",
"explicit_observations",
"\n".join(f" • {obs}" for obs in observations.explicit),
"\n".join(f" • {obs}" for obs in observations.explicit),
"blob",
)
log_performance_metrics("minimal_deriver", f"{latest_message.id}_{observed}")
# token-breakdown fields derived from messages + cap snapshots.
queued_message_count = len(queue_item_message_ids)
prompt_message_count = len(messages)
prompt_message_tokens = sum(msg.token_count for msg in messages)
extra_context_message_count = max(prompt_message_count - queued_message_count, 0)
extra_context_tokens = max(prompt_message_tokens - messages_tokens, 0)
# Data-quality invariants. Best-effort — telemetry never bleeds into the
# deriver path — but log loudly when violated so analytics alerting catches
# silent estimator failures (provider tokenization drift, scaffold helper
# returning 0) at the source instead of as drift in BigQuery later.
if response.input_tokens < messages_tokens:
logger.warning(
"token-breakdown invariant violated: response.input_tokens (%d) < messages_tokens (%d) for observed=%s, latest=%s — provider tokenization drift or wrong messages_tokens computation?",
response.input_tokens,
messages_tokens,
observed,
latest_message.public_id,
)
if prompt_tokens <= 0:
logger.warning(
"prompt_scaffold_tokens estimated as %d for observed=%s, latest=%s — estimate_deriver_prompt_tokens may have failed silently",
prompt_tokens,
observed,
latest_message.public_id,
)
# Emit telemetry event
emit(
RepresentationCompletedEvent(
@ -255,6 +298,20 @@ async def process_representation_tasks_batch(
llm_call_ms=llm_duration,
total_duration_ms=overall_duration,
input_tokens=messages_tokens,
total_input_tokens=response.input_tokens,
output_tokens=response.output_tokens,
# additive fields
queued_message_count=queued_message_count,
prompt_message_count=prompt_message_count,
prompt_message_tokens=prompt_message_tokens,
extra_context_message_count=extra_context_message_count,
extra_context_tokens=extra_context_tokens,
prompt_scaffold_tokens=prompt_tokens,
batch_max_tokens=batch_max_tokens,
max_input_tokens=settings.DERIVER.MAX_INPUT_TOKENS,
was_flush_enabled=was_flush_enabled,
hit_batch_token_cap=hit_batch_token_cap,
hit_input_token_cap=response.hit_input_token_cap,
observer_count=successful_observer_count,
)
)

20
src/deriver/enqueue.py
View File

@ -399,6 +399,10 @@ def create_dream_record(
observed: str,
dream_type: schemas.DreamType,
session_name: str | None = None,
trigger_reason: str | None = None,
delay_reason: str | None = None,
documents_since_last_dream_at_schedule: int | None = None,
document_threshold: int | None = None,
) -> dict[str, Any]:
"""
Create a queue record for a dream task.
@ -409,6 +413,10 @@ def create_dream_record(
observed: Name of the observed peer
dream_type: Type of dream to execute
session_name: Name of the session to scope the dream to if specified
trigger_reason: what tripped the schedule
delay_reason: what governed when it fires
documents_since_last_dream_at_schedule: count snapshot at schedule time
document_threshold: DOCUMENT_THRESHOLD snapshot at schedule time
Returns:
Queue record dictionary with workspace_name and other fields
@ -418,6 +426,10 @@ def create_dream_record(
observer=observer,
observed=observed,
session_name=session_name,
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule,
document_threshold=document_threshold,
)
return {
@ -436,6 +448,10 @@ async def enqueue_dream(
observed: str,
dream_type: schemas.DreamType,
session_name: str | None = None,
trigger_reason: str | None = None,
delay_reason: str | None = None,
documents_since_last_dream_at_schedule: int | None = None,
document_threshold: int | None = None,
) -> None:
"""
Enqueue a dream task for immediate processing by the deriver.
@ -461,6 +477,10 @@ async def enqueue_dream(
observed=observed,
dream_type=dream_type,
session_name=session_name,
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule,
document_threshold=document_threshold,
)
work_unit_key = dream_record["work_unit_key"]

8
src/deriver/prompts.py
View File

@ -22,7 +22,9 @@ def _normalized_custom_instructions(custom_instructions: str | None) -> str | No
def _custom_instructions_section(custom_instructions: str | None) -> str:
"""Render optional custom instructions for the deriver prompt."""
normalized_custom_instructions = _normalized_custom_instructions(custom_instructions)
normalized_custom_instructions = _normalized_custom_instructions(
custom_instructions
)
if normalized_custom_instructions is None:
return ""
@ -93,7 +95,9 @@ def estimate_minimal_deriver_prompt_tokens() -> int:
def estimate_deriver_prompt_tokens(custom_instructions: str | None) -> int:
"""Estimate minimal deriver prompt tokens, including custom instructions if present."""
normalized_custom_instructions = _normalized_custom_instructions(custom_instructions)
normalized_custom_instructions = _normalized_custom_instructions(
custom_instructions
)
if normalized_custom_instructions is None:
return estimate_minimal_deriver_prompt_tokens()

183
src/deriver/queue_manager.py
View File

@ -2,6 +2,7 @@ import asyncio
import signal
from asyncio import Task
from collections.abc import Sequence
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from logging import getLogger
from typing import Any, NamedTuple, cast
@ -56,6 +57,25 @@ class WorkerOwnership(NamedTuple):
aqs_id: str # The ID of the ActiveQueueSession that the worker is processing
@dataclass(frozen=True)
class QueueBatchResult:
"""Result of `QueueManager.get_queue_item_batch`.
telemetry needs to know two things in addition to the batch
contents: whether the cumulative-token cap clamped the batch, and what
the configured cap was. These flags feed `RepresentationCompletedEvent`
so analytics can detect "we under-batched because of a flush" vs
"we hit the cap and kept going".
"""
messages_context: list[models.Message] = field(default_factory=list)
items_to_process: list["QueueItem"] = field(default_factory=list)
configuration: ResolvedConfiguration | None = None
hit_batch_token_cap: bool = False
was_flush_enabled: bool = False
batch_max_tokens: int = 0
def _detach_queue_batch_objects(
db: AsyncSession,
messages_context: list[models.Message],
@ -464,13 +484,12 @@ class QueueManager:
break
try:
if work_unit.task_type == "representation":
(
messages_context,
items_to_process,
message_level_configuration,
) = await self.get_queue_item_batch(
batch_result = await self.get_queue_item_batch(
work_unit.task_type, work_unit_key, ownership.aqs_id
)
messages_context = batch_result.messages_context
items_to_process = batch_result.items_to_process
message_level_configuration = batch_result.configuration
logger.debug(
f"Worker {worker_id} retrieved {len(messages_context)} messages and {len(items_to_process)} queue items for work unit {work_unit_key} (AQS ID: {ownership.aqs_id})"
)
@ -503,6 +522,9 @@ class QueueManager:
observers=observers,
observed=work_unit.observed,
queue_item_message_ids=queue_item_message_ids,
hit_batch_token_cap=batch_result.hit_batch_token_cap,
was_flush_enabled=batch_result.was_flush_enabled,
batch_max_tokens=batch_result.batch_max_tokens,
)
await self.mark_queue_items_as_processed(
items_to_process, work_unit_key
@ -636,13 +658,17 @@ class QueueManager:
task_type: str,
work_unit_key: str,
aqs_id: str,
) -> tuple[list[models.Message], list[QueueItem], ResolvedConfiguration | None]:
) -> "QueueBatchResult":
"""
Batch processing for representation and agent tasks.
Returns a tuple of (messages_context, items_to_process, configuration).
Returns a `QueueBatchResult` carrying:
- messages_context: unique Message rows (conversation turns) forming the context window
- items_to_process: QueueItems for the current work_unit_key within that window
- configuration: Resolved configuration for the batch
- hit_batch_token_cap: True when the cumulative-token window clamped the batch
- was_flush_enabled: snapshot of `settings.DERIVER.FLUSH_ENABLED` at fetch time
- batch_max_tokens: snapshot of the cap actually applied to this batch
"""
if task_type != "representation":
raise ValueError(
@ -650,6 +676,7 @@ class QueueManager:
)
batch_max_tokens = settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS
was_flush_enabled = settings.DERIVER.FLUSH_ENABLED
parsed_key = parse_work_unit_key(work_unit_key)
messages_context: list[models.Message] = []
items_to_process: list[QueueItem] = []
@ -663,7 +690,10 @@ class QueueManager:
.where(models.ActiveQueueSession.id == aqs_id)
)
if not ownership_check.scalar_one_or_none():
return [], [], None
return QueueBatchResult(
was_flush_enabled=was_flush_enabled,
batch_max_tokens=batch_max_tokens,
)
# Step 2: Build a single SQL query that:
# 1. Finds the earliest unprocessed message for this work_unit_key
@ -712,9 +742,16 @@ class QueueManager:
preceding_message_id_subq, min_unprocessed_message_id_subq
)
# Build CTE with ALL messages starting from effective_start_id
# This includes the preceding context message (if any) and interleaving messages
cte = (
# Build CTE in two nested selects so we can layer a second window
# function on top of `cumulative_token_count`. Postgres doesn't
# allow nesting window functions in a single select; we compute
# `cumulative_token_count` in `inner_cte`, then `cap_exceeded` as
# `bool_or(cumulative > cap) OVER ()` in the outer CTE. The flag
# is identical across every row, so reading it from any returned
# row tells us whether the SQL cap would have excluded messages —
# eliminating the separate `SELECT EXISTS` roundtrip that used to
# run post-fetch.
inner_cte = (
select(
models.Message.id.label("message_id"),
models.Message.token_count.label("token_count"),
@ -726,7 +763,20 @@ class QueueManager:
.where(models.Message.session_name == parsed_key.session_name)
.where(models.Message.workspace_name == parsed_key.workspace_name)
.where(models.Message.id >= effective_start_id)
.order_by(models.Message.id)
.subquery()
)
cte = (
select(
inner_cte.c.message_id,
inner_cte.c.token_count,
inner_cte.c.peer_name,
inner_cte.c.cumulative_token_count,
func.bool_or(inner_cte.c.cumulative_token_count > batch_max_tokens)
.over()
.label("cap_exceeded"),
)
.order_by(inner_cte.c.message_id)
.cte()
)
@ -738,7 +788,11 @@ class QueueManager:
)
query = (
select(models.Message, models.QueueItem)
select(
models.Message,
models.QueueItem,
cte.c.cap_exceeded.label("cap_exceeded"),
)
.select_from(cte)
.join(models.Message, models.Message.id == cte.c.message_id)
.outerjoin(
@ -756,31 +810,108 @@ class QueueManager:
result = await db.execute(query)
rows = result.all()
if not rows:
return [], [], None
return QueueBatchResult(
was_flush_enabled=was_flush_enabled,
batch_max_tokens=batch_max_tokens,
)
# cap_exceeded is window-aggregated over the CTE — same value on
# every row. Read once from the first row; default False if the
# cap is disabled (`batch_max_tokens == 0`).
cap_exceeded_from_query: bool = (
bool(rows[0][2]) if rows and batch_max_tokens > 0 else False
)
seen_messages: set[int] = set()
for m, qi in rows:
for m, qi, _cap in rows:
if m.id not in seen_messages:
messages_context.append(m)
seen_messages.add(m.id)
if qi is not None:
items_to_process.append(qi)
# Detach BEFORE config-filter — `_resolve_batch_configuration` is
# sync and doesn't need the session; `messages_context` is a plain
# Python list after detach and survives the rest of this block.
_detach_queue_batch_objects(db, messages_context, items_to_process)
items_to_process, resolved_config = _resolve_batch_configuration(
items_to_process
)
if items_to_process:
max_queue_item_message_id = max(
qi.message_id for qi in items_to_process if qi.message_id is not None
# The QUEUE-ITEM boundary (not the messages_context tail) is
# what matters for cap detection. messages_context includes
# non-queue interleaving context messages — if SQL kept some
# trailing context past the last queued item, the config
# filter trims that context but doesn't touch the queue
# items. Using messages_context[-1].id as a "did config
# filter shrink the batch" signal produced false negatives
# for that case.
last_queued_id_before: int | None = (
max(
qi.message_id
for qi in items_to_process
if qi.message_id is not None
)
if items_to_process
else None
)
messages_context = [
m for m in messages_context if m.id <= max_queue_item_message_id
]
return messages_context, items_to_process, resolved_config
items_to_process, resolved_config = _resolve_batch_configuration(
items_to_process
)
if items_to_process:
max_queue_item_message_id = max(
qi.message_id
for qi in items_to_process
if qi.message_id is not None
)
messages_context = [
m for m in messages_context if m.id <= max_queue_item_message_id
]
last_queued_id_after: int | None = (
max(
qi.message_id
for qi in items_to_process
if qi.message_id is not None
)
if items_to_process
else None
)
# detect if `batch_max_tokens` clamped this returned batch.
#
# `cap_exceeded_from_query` comes from the CTE's
# `bool_or(cumulative > cap) OVER ()` column — true iff the
# SQL would have excluded at least one message because of the
# cap. Combined with the queue-boundary guard below, this
# tells us the cap was binding on the returned batch:
#
# 1. Config filter didn't shrink the QUEUE-ITEM boundary
# (`last_queued_id_before == last_queued_id_after`) —
# i.e. SQL chose the trailing queue item, not config; AND
# 2. The CTE detected at least one message past the cap.
#
# Both conditions must hold; otherwise the cap wasn't the
# constraint on this specific returned batch.
#
# Previously we issued a separate `SELECT EXISTS` query for
# the second condition. Folding it into the CTE eliminates the
# roundtrip — every batch fetch is now one query, not two.
if (
batch_max_tokens > 0
and last_queued_id_before is not None
and last_queued_id_before == last_queued_id_after
):
hit_batch_token_cap = cap_exceeded_from_query
else:
hit_batch_token_cap = False
return QueueBatchResult(
messages_context=messages_context,
items_to_process=items_to_process,
configuration=resolved_config,
hit_batch_token_cap=hit_batch_token_cap,
was_flush_enabled=was_flush_enabled,
batch_max_tokens=batch_max_tokens,
)
async def mark_queue_items_as_processed(
self, items: list[QueueItem], work_unit_key: str

53
src/dialectic/core.py
View File

@ -7,10 +7,11 @@ and synthesize responses to queries about a peer.
import logging
import time
import uuid
from collections.abc import AsyncIterator, Callable
from typing import Any, cast
from nanoid import generate as generate_nanoid
from src import crud
from src.config import ConfiguredModelSettings, ReasoningLevel, settings
from src.dependencies import tracked_db
@ -21,8 +22,9 @@ from src.llm import (
StreamingResponseWithMetadata,
honcho_llm_call,
)
from src.llm.types import LLMTelemetryContext
from src.telemetry import prometheus_metrics
from src.telemetry.events import DialecticCompletedEvent, emit
from src.telemetry.events import DialecticCompletedEvent, EmbeddingCallPurpose, emit
from src.telemetry.logging import (
accumulate_metric,
log_performance_metrics,
@ -36,6 +38,7 @@ from src.utils.agent_tools import (
search_memory,
)
from src.utils.formatting import format_new_turn_with_timestamp
from src.utils.types import embedding_call_purpose
logger = logging.getLogger(__name__)
@ -99,9 +102,7 @@ class DialecticAgent:
]
self._session_history_initialized: bool = False
self._prefetched_conclusion_count: int = 0
self._run_id: str = str(uuid.uuid4())[
:8
] # Always generate for event correlation
self._run_id: str = generate_nanoid() # Always generate for event correlation
async def _initialize_session_history(self) -> None:
"""Fetch and inject session history into the system prompt if configured."""
@ -173,7 +174,13 @@ class DialecticAgent:
try:
# Pre-compute embedding once for both searches (no DB needed)
query_embedding = await embedding_client.embed(query)
with embedding_call_purpose(
EmbeddingCallPurpose.DIALECTIC_PREFETCH.value,
workspace_name=self.workspace_name,
run_id=self._run_id,
parent_category="dialectic",
):
query_embedding = await embedding_client.embed(query)
# search_memory manages its own short-lived DB sessions so no
# connection is held during external vector-store calls.
@ -200,10 +207,11 @@ class DialecticAgent:
if explicit_repr.is_empty() and derived_repr.is_empty():
return None
# Count prefetched conclusions for telemetry
explicit_count = len(explicit_repr.explicit) + len(explicit_repr.deductive)
derived_count = len(derived_repr.explicit) + len(derived_repr.deductive)
self._prefetched_conclusion_count = explicit_count + derived_count
# Count prefetched conclusions for telemetry. `Representation.len()`
# sums all four levels (explicit/deductive/inductive/contradiction);
# the previous hand-sum dropped inductive + contradiction even
# though prefetch explicitly requests them.
self._prefetched_conclusion_count = explicit_repr.len() + derived_repr.len()
# Format as two separate sections
parts: list[str] = []
@ -242,7 +250,7 @@ class DialecticAgent:
if self.metric_key:
task_name = self.metric_key
else:
run_id = str(uuid.uuid4())[:8]
run_id = generate_nanoid()
task_name = f"dialectic_chat_{run_id}"
start_time = time.perf_counter()
@ -293,6 +301,23 @@ class DialecticAgent:
return tool_executor, task_name, run_id, start_time
def _telemetry_context(self) -> LLMTelemetryContext:
"""Build the LLMTelemetryContext shared by answer() and answer_stream().
Carries the instance's `_run_id` (always set in __init__) + workspace +
peer identifiers so LLMCallCompletedEvent and 's
AgentIterationEvent can attribute every per-iteration LLM call back to
this dialectic invocation.
"""
return LLMTelemetryContext(
workspace_name=self.workspace_name,
call_purpose="dialectic.answer",
parent_category="dialectic",
agent_type="dialectic",
run_id=self._run_id,
peer_name=self.observed,
)
def _log_response_metrics(
self,
task_name: str,
@ -306,6 +331,7 @@ class DialecticAgent:
tool_calls_count: int,
thinking_content: str | None,
iterations: int,
hit_input_token_cap: bool = False,
) -> None:
"""
Log metrics common to both streaming and non-streaming responses.
@ -374,6 +400,7 @@ class DialecticAgent:
output_tokens=output_tokens,
cache_read_tokens=cache_read_input_tokens or 0,
cache_creation_tokens=cache_creation_input_tokens or 0,
hit_input_token_cap=hit_input_token_cap,
)
)
@ -422,6 +449,7 @@ class DialecticAgent:
track_name="Dialectic Agent",
max_input_tokens=settings.DIALECTIC.MAX_INPUT_TOKENS,
trace_name="dialectic_chat",
telemetry=self._telemetry_context(),
)
self._log_response_metrics(
@ -436,6 +464,7 @@ class DialecticAgent:
tool_calls_count=len(response.tool_calls_made),
thinking_content=response.thinking_content,
iterations=response.iterations,
hit_input_token_cap=response.hit_input_token_cap,
)
return response.content
@ -489,6 +518,7 @@ class DialecticAgent:
track_name="Dialectic Agent Stream",
max_input_tokens=settings.DIALECTIC.MAX_INPUT_TOKENS,
trace_name="dialectic_chat",
telemetry=self._telemetry_context(),
),
)
@ -510,4 +540,5 @@ class DialecticAgent:
tool_calls_count=len(response.tool_calls_made),
thinking_content=response.thinking_content,
iterations=response.iterations,
hit_input_token_cap=response.hit_input_token_cap,
)

46
src/dreamer/dream_scheduler.py
View File

@ -60,8 +60,17 @@ class DreamScheduler:
*,
observer: str,
observed: str,
trigger_reason: str | None = None,
delay_reason: str | None = None,
documents_since_last_dream_at_schedule: int | None = None,
document_threshold: int | None = None,
) -> None:
"""Schedule a dream for a collection after a delay."""
"""Schedule a dream for a collection after a delay.
telemetry kwargs are captured at schedule time and threaded
through the queue payload so DreamRunEvent can attribute the dream
back to its scheduling context.
"""
if not settings.DREAM.ENABLED:
return
@ -76,6 +85,10 @@ class DreamScheduler:
dream_type,
observer=observer,
observed=observed,
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule,
document_threshold=document_threshold,
)
)
self.pending_dreams[work_unit_key] = task
@ -133,6 +146,10 @@ class DreamScheduler:
*,
observer: str,
observed: str,
trigger_reason: str | None = None,
delay_reason: str | None = None,
documents_since_last_dream_at_schedule: int | None = None,
document_threshold: int | None = None,
) -> None:
try:
await asyncio.sleep(delay_minutes * 60)
@ -142,6 +159,10 @@ class DreamScheduler:
dream_type,
observer=observer,
observed=observed,
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule,
document_threshold=document_threshold,
)
logger.info("Executed dream for %s", work_unit_key)
@ -159,6 +180,10 @@ class DreamScheduler:
*,
observer: str,
observed: str,
trigger_reason: str | None = None,
delay_reason: str | None = None,
documents_since_last_dream_at_schedule: int | None = None,
document_threshold: int | None = None,
) -> None:
"""Execute the dream by enqueueing it."""
from src import crud
@ -204,6 +229,10 @@ class DreamScheduler:
observed=observed,
dream_type=dream_type,
session_name=session_name,
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule,
document_threshold=document_threshold,
)
async def shutdown(self) -> None:
@ -274,6 +303,14 @@ async def check_and_schedule_dream(
)
if documents_since_last_dream >= settings.DREAM.DOCUMENT_THRESHOLD:
# capture *why* this schedule fired (threshold) and
# *how* it will fire (idle vs immediate). The two gates were
# intentionally split — collapsing them into a single trigger_reason
# would lose the scheduling semantics.
trigger_reason = "document_threshold"
delay_reason = (
"idle_timeout" if settings.DREAM.IDLE_TIMEOUT_MINUTES > 0 else "immediate"
)
if last_dream_at:
try:
last_dream_time = datetime.fromisoformat(last_dream_at)
@ -286,6 +323,9 @@ async def check_and_schedule_dream(
f"Skipping dream for {collection.observer}/{collection.observed}: only {hours_since_last_dream:.1f} hours "
+ f"since last dream (minimum: {settings.DREAM.MIN_HOURS_BETWEEN_DREAMS})"
)
# delay_reason = "min_hours_gate" if we DID schedule, but
# we don't — return early. Telemetry only records dreams
# that actually fire.
return False
except (ValueError, TypeError) as e:
logger.warning(
@ -345,6 +385,10 @@ async def check_and_schedule_dream(
dream_type=DreamType(dream_type),
observer=collection.observer,
observed=collection.observed,
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream,
document_threshold=settings.DREAM.DOCUMENT_THRESHOLD,
)
logger.debug(
"Scheduled dream",

220
src/dreamer/orchestrator.py
View File

@ -15,12 +15,12 @@ from __future__ import annotations
import logging
import time
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any
import sentry_sdk
from nanoid import generate as generate_nanoid
from sqlalchemy import func, select
from src import crud, models
@ -28,7 +28,7 @@ from src.config import settings
from src.dependencies import tracked_db
from src.dreamer.specialists import SPECIALISTS, SpecialistResult
from src.dreamer.surprisal import SurprisalScore # type: ignore
from src.exceptions import SpecialistExecutionError, SurprisalError
from src.exceptions import SurprisalError
from src.schemas import DreamType
from src.telemetry.events import DreamRunEvent, emit
from src.telemetry.logging import (
@ -69,6 +69,12 @@ async def run_dream(
observer: str,
observed: str,
session_name: str | None = None,
*,
dream_type: str | None = None,
trigger_reason: str | None = None,
delay_reason: str | None = None,
documents_since_last_dream_at_schedule: int | None = None,
document_threshold: int | None = None,
) -> DreamResult | None:
"""
Run a full dream cycle with optional surprisal-based sampling.
@ -89,7 +95,7 @@ async def run_dream(
if not settings.DREAM.ENABLED:
return None
run_id = str(uuid.uuid4())[:8]
run_id = generate_nanoid()
task_name = f"dream_orchestrator_{run_id}"
start_time = time.perf_counter()
@ -121,12 +127,12 @@ async def run_dream(
deduction_result: SpecialistResult | None = None
induction_result: SpecialistResult | None = None
# Phase 0: Surprisal-based sampling (if enabled)
# Surprisal-based sampling (if enabled)
# Specialists are self-directed by default - hints are optional suggestions
exploration_hints: list[str] | None = None
if settings.DREAM.SURPRISAL.ENABLED:
logger.info(f"[{run_id}] Phase 0: Computing surprisal scores")
logger.info(f"[{run_id}] Computing surprisal scores")
try:
from src.dreamer.surprisal import sample_observations_with_surprisal
@ -165,91 +171,127 @@ async def run_dream(
accumulate_metric(task_name, "surprisal_error", str(e), "blob")
# Specialists will explore freely without hints
# Phase 1: Run deduction specialist (manages its own DB sessions)
logger.info(f"[{run_id}] Phase 1: Running deduction specialist")
deduction_specialist = SPECIALISTS["deduction"]
# Specialist phase wrapped in try/finally so DreamRunEvent ALWAYS emits —
# both for graceful failures (specialist raises Exception) AND unexpected
# exceptions including CancelledError. The orphaned-child-event problem
# before this fix: specialists.py:350 catches BaseException and re-raises,
# but the orchestrator's old `except SpecialistExecutionError` clauses
# didn't match anything actually raised in src/, so any real failure
# propagated past the emit at the bottom of this function. Now: the
# specialist try blocks catch Exception (CancelledError still propagates
# correctly), and the outer try/finally guarantees the parent event fires.
#
# Pre-init aggregate locals so the function-level return below and the
# finally's emit both see defined values even on an early exception.
duration_ms = 0.0
total_iterations = 0
total_input_tokens = 0
total_output_tokens = 0
try:
deduction_result = await deduction_specialist.run(
workspace_name=workspace_name,
observer=observer,
observed=observed,
session_name=session_name,
hints=exploration_hints,
configuration=configuration,
parent_run_id=run_id,
)
logger.info(
f"[{run_id}] Deduction completed: {deduction_result.content[:200]}..."
)
accumulate_metric(
task_name, "deduction_result", deduction_result.content, "blob"
)
deduction_success = deduction_result.success
except SpecialistExecutionError as e:
logger.error(f"[{run_id}] Deduction specialist failed: {e}", exc_info=True)
accumulate_metric(task_name, "deduction_error", str(e), "blob")
# Run deduction specialist (manages its own DB sessions)
logger.info(f"[{run_id}] Running deduction specialist")
deduction_specialist = SPECIALISTS["deduction"]
try:
deduction_result = await deduction_specialist.run(
workspace_name=workspace_name,
observer=observer,
observed=observed,
session_name=session_name,
hints=exploration_hints,
configuration=configuration,
parent_run_id=run_id,
)
logger.info(
f"[{run_id}] Deduction completed: {deduction_result.content[:200]}..."
)
accumulate_metric(
task_name, "deduction_result", deduction_result.content, "blob"
)
deduction_success = deduction_result.success
except Exception as e:
# `Exception` (not `BaseException`) — CancelledError must still
# propagate so the worker can shut down. SpecialistExecutionError
# is no longer raised by `src/`, but the catch is broad enough to
# cover provider/DB/tool/validation errors.
logger.error(f"[{run_id}] Deduction specialist failed: {e}", exc_info=True)
accumulate_metric(task_name, "deduction_error", str(e), "blob")
# Phase 2: Run induction specialist (after deduction so it can see new deductive obs)
logger.info(f"[{run_id}] Phase 2: Running induction specialist")
induction_specialist = SPECIALISTS["induction"]
try:
induction_result = await induction_specialist.run(
workspace_name=workspace_name,
observer=observer,
observed=observed,
session_name=session_name,
hints=exploration_hints,
configuration=configuration,
parent_run_id=run_id,
)
logger.info(
f"[{run_id}] Induction completed: {induction_result.content[:200]}..."
)
accumulate_metric(
task_name, "induction_result", induction_result.content, "blob"
)
induction_success = induction_result.success
except SpecialistExecutionError as e:
logger.error(f"[{run_id}] Induction specialist failed: {e}", exc_info=True)
accumulate_metric(task_name, "induction_error", str(e), "blob")
# Run induction specialist (after deduction so it can see new deductive obs)
logger.info(f"[{run_id}] Running induction specialist")
induction_specialist = SPECIALISTS["induction"]
try:
induction_result = await induction_specialist.run(
workspace_name=workspace_name,
observer=observer,
observed=observed,
session_name=session_name,
hints=exploration_hints,
configuration=configuration,
parent_run_id=run_id,
)
logger.info(
f"[{run_id}] Induction completed: {induction_result.content[:200]}..."
)
accumulate_metric(
task_name, "induction_result", induction_result.content, "blob"
)
induction_success = induction_result.success
except Exception as e:
logger.error(f"[{run_id}] Induction specialist failed: {e}", exc_info=True)
accumulate_metric(task_name, "induction_error", str(e), "blob")
# Log final metrics
duration_ms = (time.perf_counter() - start_time) * 1000
accumulate_metric(task_name, "total_duration", duration_ms, "ms")
# Log final metrics
duration_ms = (time.perf_counter() - start_time) * 1000
accumulate_metric(task_name, "total_duration", duration_ms, "ms")
logger.info(f"[{run_id}] Dream cycle completed in {duration_ms:.0f}ms")
log_performance_metrics("dream_orchestrator", run_id)
# Aggregate metrics from specialist results
total_iterations = (deduction_result.iterations if deduction_result else 0) + (
induction_result.iterations if induction_result else 0
)
total_input_tokens = (deduction_result.input_tokens if deduction_result else 0) + (
induction_result.input_tokens if induction_result else 0
)
total_output_tokens = (
deduction_result.output_tokens if deduction_result else 0
) + (induction_result.output_tokens if induction_result else 0)
# Emit DreamRunEvent with aggregated metrics
emit(
DreamRunEvent(
run_id=run_id,
workspace_name=workspace_name,
session_name=session_name,
observer=observer,
observed=observed,
specialists_run=["deduction", "induction"],
deduction_success=deduction_success,
induction_success=induction_success,
surprisal_enabled=settings.DREAM.SURPRISAL.ENABLED,
surprisal_conclusion_count=surprisal_observation_count,
total_iterations=total_iterations,
total_input_tokens=total_input_tokens,
total_output_tokens=total_output_tokens,
total_duration_ms=duration_ms,
)
)
logger.info(f"[{run_id}] Dream cycle completed in {duration_ms:.0f}ms")
log_performance_metrics("dream_orchestrator", run_id)
finally:
# Emit DreamRunEvent unconditionally so analytics see a parent for
# every DreamSpecialistEvent. Aggregation guards None specialist
# results. Emit-side errors are swallowed by the global telemetry
# path; a defensive try around event construction protects against
# schema-validation surprises during partial state.
if duration_ms == 0.0:
duration_ms = (time.perf_counter() - start_time) * 1000
try:
total_iterations = (
deduction_result.iterations if deduction_result else 0
) + (induction_result.iterations if induction_result else 0)
total_input_tokens = (
deduction_result.input_tokens if deduction_result else 0
) + (induction_result.input_tokens if induction_result else 0)
total_output_tokens = (
deduction_result.output_tokens if deduction_result else 0
) + (induction_result.output_tokens if induction_result else 0)
emit(
DreamRunEvent(
run_id=run_id,
workspace_name=workspace_name,
session_name=session_name,
observer=observer,
observed=observed,
specialists_run=["deduction", "induction"],
deduction_success=deduction_success,
induction_success=induction_success,
surprisal_enabled=settings.DREAM.SURPRISAL.ENABLED,
surprisal_conclusion_count=surprisal_observation_count,
total_iterations=total_iterations,
total_input_tokens=total_input_tokens,
total_output_tokens=total_output_tokens,
total_duration_ms=duration_ms,
# scheduling context threaded through the
# queue payload by check_and_schedule_dream.
dream_type=dream_type,
enabled_types_count=len(settings.DREAM.ENABLED_TYPES),
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule,
document_threshold=document_threshold,
)
)
except Exception: # pragma: no cover - telemetry must not raise
logger.debug("Failed to emit DreamRunEvent", exc_info=True)
return DreamResult(
run_id=run_id,
@ -315,6 +357,12 @@ DREAM: {payload.dream_type} documents for {workspace_name}/{payload.observer}/{p
observer=payload.observer,
observed=payload.observed,
session_name=payload.session_name,
# scheduling context — propagated to DreamRunEvent.
dream_type=payload.dream_type.value,
trigger_reason=payload.trigger_reason,
delay_reason=payload.delay_reason,
documents_since_last_dream_at_schedule=payload.documents_since_last_dream_at_schedule,
document_threshold=payload.document_threshold,
)
# Log completion (telemetry event already emitted in run_dream)

502
src/dreamer/specialists.py
View File

@ -12,17 +12,20 @@ from __future__ import annotations
import logging
import time
import uuid
from abc import ABC, abstractmethod
from collections import Counter
from collections.abc import Callable
from dataclasses import dataclass
from typing import Any
from typing import Any, cast
from nanoid import generate as generate_nanoid
from src import crud, schemas
from src.config import ConfiguredModelSettings, settings
from src.dependencies import tracked_db
from src.exceptions import ValidationException
from src.llm import HonchoLLMCallResponse, honcho_llm_call
from src.llm.types import LLMTelemetryContext
from src.schemas import ResolvedConfiguration
from src.telemetry import prometheus_metrics
from src.telemetry.events import DreamSpecialistEvent, emit
@ -72,9 +75,12 @@ class BaseSpecialist(ABC):
"""Base class for agentic specialists."""
name: str = "base"
# Whether this specialist is allowed to write to the peer card. Defaults to True;
# specialists that should never touch the card (e.g., induction) override to False.
can_update_peer_card: bool = True
# Subclasses can override to customize the peer card update instruction
peer_card_update_instruction: str = (
"Only update this with durable profile facts via `update_peer_card`."
"Only update this with durable identity markers via `update_peer_card`."
)
@abstractmethod
@ -153,154 +159,271 @@ If you update it, send the full deduplicated list and remove stale entries.
Returns:
SpecialistResult with metrics and content
"""
run_id = parent_run_id or str(uuid.uuid4())[:8]
run_id = parent_run_id or generate_nanoid()
task_name = f"dreamer_{self.name}_{run_id}"
start_time = time.perf_counter()
# Short-lived DB session for preflight operations
async with tracked_db("dream.specialist.preflight") as db:
await crud.get_peer(db, workspace_name, schemas.PeerCreate(name=observer))
if observer != observed:
# Telemetry state initialized BEFORE the try so the finally block can
# always read consistent values. Without this, a failure in preflight
# (peer lookup, peer-card preload, create_tool_executor, get_model_config,
# prompt construction) would bypass the finally entirely and the run
# would disappear from failure-path analytics — orphaning the
# downstream DreamRunEvent.
specialist_success = False
specialist_error_class: str | None = None
response: HonchoLLMCallResponse[str] | None = None
# Rollups initialized here so they're accessible from the finally
# block on the failure path (where they stay at defaults).
created_observation_count = 0
deleted_observation_count = 0
peer_card_updated = False
search_tool_calls_count = 0
duration_ms = 0.0
# Per-level rollups — accumulated from each create/delete_observations
# tool call's metadata.levels list. Counter rather than list[str] so
# the emitted dict stays compact even when the specialist produces
# many observations.
created_counts_by_level: Counter[str] = Counter()
deleted_counts_by_level: Counter[str] = Counter()
try:
# Short-lived DB session for preflight operations
async with tracked_db("dream.specialist.preflight") as db:
await crud.get_peer(
db, workspace_name, schemas.PeerCreate(name=observed)
db, workspace_name, schemas.PeerCreate(name=observer)
)
if observer != observed:
await crud.get_peer(
db, workspace_name, schemas.PeerCreate(name=observed)
)
# Determine if peer card tools should be included. Specialists that
# cannot write to the peer card (e.g., induction) skip the fetch and
# the prompt section entirely.
peer_card_enabled = self.can_update_peer_card and (
configuration is None or configuration.peer_card.create
)
# Determine if peer card tools should be included
peer_card_enabled = configuration is None or configuration.peer_card.create
# Fetch current peer card to inject into prompt (saves a tool call)
current_peer_card: list[str] | None = None
if peer_card_enabled:
current_peer_card = await crud.get_peer_card(
db,
workspace_name=workspace_name,
observer=observer,
observed=observed,
)
# DB session closed — LLM calls happen without holding a connection
# Fetch current peer card to inject into prompt (saves a tool call)
current_peer_card: list[str] | None = None
if peer_card_enabled:
current_peer_card = await crud.get_peer_card(
db,
workspace_name=workspace_name,
observer=observer,
observed=observed,
)
# DB session closed — LLM calls happen without holding a connection
# Build messages
messages: list[dict[str, str]] = [
{
"role": "system",
"content": self.build_system_prompt(
observed, peer_card_enabled=peer_card_enabled
),
},
{
"role": "user",
"content": self.build_user_prompt(hints, current_peer_card),
},
]
# Build messages
messages: list[dict[str, str]] = [
{
"role": "system",
"content": self.build_system_prompt(
observed, peer_card_enabled=peer_card_enabled
),
},
{
"role": "user",
"content": self.build_user_prompt(hints, current_peer_card),
},
]
# Create tool executor with telemetry context
tool_executor: Callable[
[str, dict[str, Any]], Any
] = await create_tool_executor(
workspace_name=workspace_name,
observer=observer,
observed=observed,
session_name=session_name,
include_observation_ids=True,
history_token_limit=settings.DREAM.HISTORY_TOKEN_LIMIT,
configuration=configuration,
run_id=run_id,
agent_type=self.name,
parent_category="dream",
)
model_config = self.get_model_config()
# Respect operator-configured max_output_tokens on the specialist's
# ModelConfig (e.g. DREAM_DEDUCTION_MODEL_CONFIG__MAX_OUTPUT_TOKENS).
# Only fall back to the specialist's hardcoded default when the
# config leaves max_output_tokens unset or non-positive.
configured_max = model_config.max_output_tokens
effective_max_tokens = (
configured_max
if configured_max and configured_max > 0
else self.get_max_tokens()
)
# Track iterations via callback
iteration_count = 0
def iteration_callback(data: Any) -> None:
nonlocal iteration_count
iteration_count = data.iteration
# Run the agent loop
response: HonchoLLMCallResponse[str] = await honcho_llm_call(
model_config=model_config,
prompt="", # Ignored since we pass messages
max_tokens=effective_max_tokens,
tools=self.get_tools(peer_card_enabled=peer_card_enabled),
tool_choice=None,
tool_executor=tool_executor,
max_tool_iterations=self.get_max_iterations(),
messages=messages,
track_name=f"Dreamer/{self.name}",
iteration_callback=iteration_callback,
)
# Log metrics
duration_ms = (time.perf_counter() - start_time) * 1000
accumulate_metric(task_name, "total_duration", duration_ms, "ms")
accumulate_metric(
task_name, "tool_calls", len(response.tool_calls_made), "count"
)
accumulate_metric(task_name, "input_tokens", response.input_tokens, "count")
accumulate_metric(task_name, "output_tokens", response.output_tokens, "count")
# Prometheus metrics
if settings.METRICS.ENABLED:
prometheus_metrics.record_dreamer_tokens(
count=response.input_tokens,
specialist_name=self.name,
token_type=TokenTypes.INPUT.value,
)
prometheus_metrics.record_dreamer_tokens(
count=response.output_tokens,
specialist_name=self.name,
token_type=TokenTypes.OUTPUT.value,
)
logger.info(
f"{self.name}: Completed in {duration_ms:.0f}ms, "
+ f"{len(response.tool_calls_made)} tool calls, "
+ f"{response.input_tokens} in / {response.output_tokens} out"
)
log_performance_metrics(f"dreamer_{self.name}", run_id)
# Emit telemetry event
emit(
DreamSpecialistEvent(
run_id=run_id,
specialist_type=self.name,
# Create tool executor with telemetry context
tool_executor: Callable[
[str, dict[str, Any]], Any
] = await create_tool_executor(
workspace_name=workspace_name,
observer=observer,
observed=observed,
iterations=iteration_count,
session_name=session_name,
include_observation_ids=True,
history_token_limit=settings.DREAM.HISTORY_TOKEN_LIMIT,
configuration=configuration,
run_id=run_id,
agent_type=self.name,
parent_category="dream",
)
model_config = self.get_model_config()
# Respect operator-configured max_output_tokens on the specialist's
# ModelConfig (e.g. DREAM_DEDUCTION_MODEL_CONFIG__MAX_OUTPUT_TOKENS).
# Only fall back to the specialist's hardcoded default when the
# config leaves max_output_tokens unset or non-positive.
configured_max = model_config.max_output_tokens
effective_max_tokens = (
configured_max
if configured_max and configured_max > 0
else self.get_max_tokens()
)
# call_purpose maps "deduction"/"induction" specialist names onto the
# closed CallPurpose enum slugs without importing the enum here.
call_purpose_slug = f"dream.{self.name}"
# Run the agent loop
response = await honcho_llm_call(
model_config=model_config,
prompt="", # Ignored since we pass messages
max_tokens=effective_max_tokens,
tools=self.get_tools(peer_card_enabled=peer_card_enabled),
tool_choice=None,
tool_executor=tool_executor,
max_tool_iterations=self.get_max_iterations(),
messages=messages,
track_name=f"Dreamer/{self.name}",
telemetry=LLMTelemetryContext(
workspace_name=workspace_name,
call_purpose=call_purpose_slug,
parent_category="dream",
agent_type=self.name,
run_id=run_id,
observer=observer,
observed=observed,
),
)
# Log metrics
duration_ms = (time.perf_counter() - start_time) * 1000
accumulate_metric(task_name, "total_duration", duration_ms, "ms")
accumulate_metric(
task_name, "tool_calls", len(response.tool_calls_made), "count"
)
accumulate_metric(task_name, "input_tokens", response.input_tokens, "count")
accumulate_metric(
task_name, "output_tokens", response.output_tokens, "count"
)
# Prometheus metrics
if settings.METRICS.ENABLED:
prometheus_metrics.record_dreamer_tokens(
count=response.input_tokens,
specialist_name=self.name,
token_type=TokenTypes.INPUT.value,
)
prometheus_metrics.record_dreamer_tokens(
count=response.output_tokens,
specialist_name=self.name,
token_type=TokenTypes.OUTPUT.value,
)
logger.info(
f"{self.name}: Completed in {duration_ms:.0f}ms, "
+ f"{len(response.tool_calls_made)} tool calls, "
+ f"{response.input_tokens} in / {response.output_tokens} out"
)
log_performance_metrics(f"dreamer_{self.name}", run_id)
# count actual observations created/deleted from the
# ToolResult.metadata that stashed on `all_tool_calls[i]`.
# Counting tool-name occurrences would mis-attribute: a single
# create_observations call can produce N (or zero) observations. The
# truth lives in the handler's returned metadata.
_search_tools = {
"search_memory",
"search_messages",
"search_messages_temporal",
}
for tc in response.tool_calls_made:
tool_name_any: Any = tc.get("tool_name") or tc.get("name")
meta_any: Any = tc.get("tool_result_metadata") or {}
if tool_name_any in _search_tools:
search_tool_calls_count += 1
if isinstance(meta_any, dict):
# `meta_any` is `dict[Unknown, Unknown]` after the isinstance
# narrow because tool_calls_made is typed list[dict[str, Any]].
# Cast to the expected dict shape to silence the partial-known
# warning without losing runtime safety.
meta_dict = cast(dict[str, Any], meta_any)
created_val: Any = meta_dict.get("created_count") or 0
deleted_val: Any = meta_dict.get("deleted_count") or 0
created_observation_count += int(created_val)
deleted_observation_count += int(deleted_val)
if meta_dict.get("peer_card_updated"):
peer_card_updated = True
# Accumulate per-level counts from create/delete observations.
# Both handlers stash `{"levels": ["explicit", "deductive", ...]}`
# in metadata (agent_tools.py:1373 + agent_tools.py:2011).
levels_any: Any = meta_dict.get("levels")
if isinstance(levels_any, list):
levels_list = cast(list[Any], levels_any)
level_strs = [
str(level) for level in levels_list if level is not None
]
# Tool-name dispatch decides which counter to update —
# create_observations metadata has `created_count`,
# delete_observations has `deleted_count`.
if "created_count" in meta_dict:
created_counts_by_level.update(level_strs)
elif "deleted_count" in meta_dict:
deleted_counts_by_level.update(level_strs)
specialist_success = True
return SpecialistResult(
run_id=run_id,
specialist_type=self.name,
iterations=response.iterations,
tool_calls_count=len(response.tool_calls_made),
input_tokens=response.input_tokens,
output_tokens=response.output_tokens,
duration_ms=duration_ms,
success=True,
content=response.content,
)
)
return SpecialistResult(
run_id=run_id,
specialist_type=self.name,
iterations=iteration_count,
tool_calls_count=len(response.tool_calls_made),
input_tokens=response.input_tokens,
output_tokens=response.output_tokens,
duration_ms=duration_ms,
success=True,
content=response.content,
)
except BaseException as e:
# BaseException (not Exception) — asyncio.CancelledError doesn't
# inherit from Exception in py3.8+, and we want the failure
# telemetry populated for cancellations too (worker shutdown,
# client disconnect). `raise` preserves cancellation semantics.
specialist_error_class = type(e).__name__
if duration_ms == 0.0:
duration_ms = (time.perf_counter() - start_time) * 1000
raise
finally:
# Emit DreamSpecialistEvent unconditionally so the success=False
# path of the schema is actually populated. Telemetry must not
# raise from inside finally during exception propagation; the
# emitter itself swallows errors but we add a defensive try
# in case event construction fails (e.g. schema validation).
try:
tool_calls_count = (
len(response.tool_calls_made) if response is not None else 0
)
input_tokens = response.input_tokens if response is not None else 0
output_tokens = response.output_tokens if response is not None else 0
iterations = response.iterations if response is not None else 0
emit(
DreamSpecialistEvent(
run_id=run_id,
specialist_type=self.name,
workspace_name=workspace_name,
observer=observer,
observed=observed,
iterations=iterations,
tool_calls_count=tool_calls_count,
input_tokens=input_tokens,
output_tokens=output_tokens,
duration_ms=duration_ms,
success=specialist_success,
error_class=specialist_error_class,
# denormalized rollups (all 0 on the failure path)
created_observation_count=created_observation_count,
deleted_observation_count=deleted_observation_count,
peer_card_updated=peer_card_updated,
search_tool_calls_count=search_tool_calls_count,
# Per-level breakdowns — `dict(Counter)` keeps the
# serialized event compact (zero-count levels are
# omitted, not enumerated).
created_counts_by_level=dict(created_counts_by_level),
deleted_counts_by_level=dict(deleted_counts_by_level),
)
)
except Exception: # pragma: no cover - telemetry must not raise
logger.debug("Failed to emit DreamSpecialistEvent", exc_info=True)
class DeductionSpecialist(BaseSpecialist):
@ -316,7 +439,7 @@ class DeductionSpecialist(BaseSpecialist):
"""
name: str = "deduction"
peer_card_update_instruction: str = "Update this with `update_peer_card` only for stable biographical/profile facts."
peer_card_update_instruction: str = "Update this with `update_peer_card` only for stable identity markers. See the PEER CARD section in the system prompt for the allowed entry kinds and rules."
def get_tools(self, *, peer_card_enabled: bool = True) -> list[dict[str, Any]]:
if peer_card_enabled:
@ -344,26 +467,61 @@ class DeductionSpecialist(BaseSpecialist):
) -> str:
peer_card_section = ""
if peer_card_enabled:
peer_card_section = """
peer_card_section = f"""
## PEER CARD (REQUIRED)
The peer card is a summary of stable biographical facts. You MUST update it when you learn:
- Name, age, location, occupation
- Family members and relationships
- Standing instructions ("call me X", "don't mention Y")
- Core preferences and traits
The peer card is {observed}'s identity store: stable identity markers that distinguish this entity from others and persist across interactions. Behavior, tendencies, transient state, and episodic facts belong in observations, not on the peer card.
Never add temporary event summaries, one-off conclusions, reasoning traces, or contradiction notes.
A peer can be anything with identity that changes over time — a human, an agent, a codebase, a team, an organization. Do not assume {observed} is human. Do not require any field; empty is the correct output when evidence is absent.
Format entries as:
- Plain facts: "Name: Alice", "Works at Google", "Lives in NYC"
- `INSTRUCTION: ...` for standing instructions
- `PREFERENCE: ...` for preferences
- `TRAIT: ...` for personality traits
### Allowed entry kinds
Call `update_peer_card` with the complete updated list when you have new biographical info.
Keep it concise (max 40 entries), deduplicated, and current."""
Each entry must start with one of these four prefixes (exact case, followed by a space):
- `IDENTITY: ...` — canonical name, kind, aliases, IDs
- `IDENTITY: Name: Alice`
- `IDENTITY: Kind: Python monorepo`
- `IDENTITY: Version: 4.2`
- `IDENTITY: Aliases: alice@example.com`
- `ATTRIBUTE: ...` — stable durable property of the entity (including explicitly stated standing preferences)
- `ATTRIBUTE: Location: NYC`
- `ATTRIBUTE: Language: Python`
- `ATTRIBUTE: Prefers tea`
- `ATTRIBUTE: Charter: ship Honcho infrastructure`
- `RELATIONSHIP: ...` — durable link to another entity
- `RELATIONSHIP: Spouse: Bob`
- `RELATIONSHIP: Maintainer: vineeth`
- `RELATIONSHIP: Members: vineeth, rajat`
- `INSTRUCTION: ...` — standing rule of engagement that {observed} has explicitly stated (do/don't for the observer). Only when explicit; never inferred from behavior.
- `INSTRUCTION: Call me Vee`
- `INSTRUCTION: Never push to main without review`
### Rules
1. **Stable.** If the value plausibly changes within six months absent a deliberate announcement, it does not belong on the card. Prefer leaving the card empty over filling it with volatile content.
2. **Subject is {observed}.** Every entry must be a fact about {observed}, not about another participant in the session. Never write facts about co-occurring peers into the card, no matter how frequently they appear in the messages.
3. **Evidence-grounded.** Only write what {observed} has explicitly stated, or what another participant has explicitly stated about {observed} with {observed}'s assent. No "general knowledge" inferences (`"co-founder"` does not imply an age; mentioning a colleague does not imply a family relationship).
4. **Type-agnostic.** {observed} may not be human. Do not require name/age/location/family/occupation fields.
5. **No behavioral content.** TRAITs, behavioral tendencies, patterns, and inferred preferences belong in observations, not on the peer card. Do not write `TRAIT:` entries or behavioral `PREFERENCE:` entries — they will be rejected.
6. **No evidence bundles.** Each entry is one concise fact. No `e.g.` clauses, no parenthetical example lists, no semicolon-separated value dumps.
### Migrating an existing peer card
The CURRENT PEER CARD shown in the user message may contain entries from an older format that do not start with an allowed prefix (e.g. `Name: Alice`, `Lives in NYC`, `TRAIT: Analytical`, `PREFERENCE: Detailed explanations`). When you call `update_peer_card`, you are responsible for re-emitting the entries you want to keep — entries you omit are dropped, and entries without an allowed prefix are silently rejected.
For each legacy entry:
- If it is still a valid identity marker, re-emit it under the correct prefix and keep the original content where reasonable. Examples:
- `Name: Alice` → `IDENTITY: Name: Alice`
- `Lives in NYC` → `ATTRIBUTE: Location: NYC`
- `Works at Google` → `ATTRIBUTE: Employer: Google`
- `INSTRUCTION: Call me Vee` → keep as is (already correctly prefixed)
- Drop entries that violate the rules above: behavioral `TRAIT:` lines, inferred behavioral `PREFERENCE:` lines, one-off events, transient state, evidence bundles. Do not re-prefix them — they are not identity markers.
When in doubt about a specific legacy entry, prefer migrating it (so valid info isn't lost) over dropping it. Splitting one dense legacy entry into multiple correctly-prefixed entries is fine and encouraged (e.g. a semicolon-separated `Tech Stack:` dump can become several `ATTRIBUTE:` lines, one per durable tool/platform).
Call `update_peer_card` with the complete deduplicated list when there is a durable identity update to record, or when the existing card needs migration. Entries that do not start with one of the four allowed prefixes will be rejected. Keep concise (max 40 entries)."""
return f"""You are a deductive reasoning agent analyzing observations about {observed}.
@ -460,20 +618,19 @@ class InductionSpecialist(BaseSpecialist):
1. Explores observations to understand what's there
2. Identifies patterns and generalizations across multiple observations
3. Creates new inductive observations with source linkage
4. Updates peer card with high-confidence traits and tendencies
Does not write to the peer card — the peer card stores stable identity markers,
which is deduction's responsibility. Inductive patterns and tendencies stay as
observations.
"""
name: str = "induction"
peer_card_update_instruction: str = "Only add highly stable profile traits/preferences; do not copy transient conclusions."
# Induction never writes to the peer card; behavioral patterns are observations.
can_update_peer_card: bool = False
def get_tools(self, *, peer_card_enabled: bool = True) -> list[dict[str, Any]]:
if peer_card_enabled:
return INDUCTION_SPECIALIST_TOOLS
return [
t
for t in INDUCTION_SPECIALIST_TOOLS
if t["name"] not in PEER_CARD_TOOL_NAMES
]
_ = peer_card_enabled
return INDUCTION_SPECIALIST_TOOLS
def get_model_config(self) -> ConfiguredModelSettings:
return _require_specialist_model_config(
@ -490,21 +647,7 @@ class InductionSpecialist(BaseSpecialist):
def build_system_prompt(
self, observed: str, *, peer_card_enabled: bool = True
) -> str:
peer_card_section = ""
if peer_card_enabled:
peer_card_section = """
## PEER CARD (REQUIRED)
After identifying patterns, only update the peer card for durable profile-level traits/preferences:
- `TRAIT: Analytical thinker`
- `TRAIT: Tends to reschedule when stressed`
- `PREFERENCE: Prefers detailed explanations`
Do NOT add temporary patterns, episode-specific conclusions, or reasoning summaries.
Call `update_peer_card` with the complete deduplicated list only when a durable profile update is warranted.
Keep it concise (max 40 entries)."""
_ = peer_card_enabled
return f"""You are an inductive reasoning agent identifying patterns about {observed}.
## YOUR JOB
@ -540,7 +683,6 @@ Create inductive observations when you see patterns:
### Temporal Patterns
- "Career goals have remained consistent"
- "Living situation changes frequently"
{peer_card_section}
## CREATING OBSERVATIONS
@ -552,8 +694,8 @@ Use `create_observations_inductive`.
"content": "The pattern or generalization",
"source_ids": ["id1", "id2", "id3"],
"sources": ["evidence 1", "evidence 2"],
"pattern_type": "tendency", // preference|behavior|personality|tendency|correlation
"confidence": "medium" // low (2 sources), medium (3-4), high (5+)
"pattern_type": "tendency", // preference|behavior|personality|tendency|correlation
"confidence": "medium" // low (2 sources), medium (3-4), high (5+)
}}]
}}
```
@ -572,11 +714,13 @@ Use `create_observations_inductive`.
hints: list[str] | None,
peer_card: list[str] | None = None,
) -> str:
peer_card_context = self._build_peer_card_context(peer_card)
# Induction does not consume peer card context — it produces inductive
# observations, not identity-marker updates.
_ = peer_card
if hints:
hints_str = "\n".join(f"- {q}" for q in hints[:5])
return f"""{peer_card_context}Explore and find patterns. These areas may be worth investigating:
return f"""Explore and find patterns. These areas may be worth investigating:
{hints_str}
@ -584,7 +728,7 @@ But follow the evidence - if you find patterns elsewhere, pursue those.
Start with `get_recent_observations`."""
return f"""{peer_card_context}Explore the observation space and identify patterns.
return """Explore the observation space and identify patterns.
Remember: patterns need 2+ sources. Look for tendencies, preferences, and behavioral regularities.

264
src/embedding_client.py
View File

@ -1,8 +1,10 @@
import asyncio
import logging
import threading
import time
from collections import defaultdict
from typing import Any, NamedTuple
from collections.abc import Awaitable, Callable
from typing import Any, Literal, NamedTuple, TypeVar
import tiktoken
from google import genai
@ -13,6 +15,115 @@ from .config import EmbeddingModelConfig, resolve_embedding_model_config, settin
logger = logging.getLogger(__name__)
_T = TypeVar("_T")
async def _emit_embedding_call(
*,
provider: str,
model: str,
texts: list[str],
input_tokens_estimate: int,
fn: Callable[[], Awaitable[_T]],
is_final_attempt: bool = True,
) -> _T:
"""time a single embedding-provider call, emit
`embedding.call.completed` on both success and exception, and return the
call's result. Errors propagate unchanged — telemetry never bleeds into
the caller's control flow.
Caller-supplied `texts` is used only for `input_count`; we don't keep the
list around for the event to avoid leaking content into telemetry.
`is_final_attempt` defaults to True so one-shot callers (`embed`,
`simple_batch_embed`) get correct semantics without changes. Retry-loop
callers (`_process_batch`) pass the real attempt index so dashboards
can distinguish exhausted retries from mid-retry failures.
"""
start = time.perf_counter()
error: BaseException | None = None
try:
return await fn()
except BaseException as exc:
error = exc
raise
finally:
if error is None:
outcome: Literal["success", "error", "cancelled"] = "success"
elif isinstance(error, asyncio.CancelledError):
outcome = "cancelled"
else:
outcome = "error"
_publish_embedding_event(
provider=provider,
model=model,
input_count=len(texts),
input_tokens_estimate=input_tokens_estimate,
duration_ms=(time.perf_counter() - start) * 1000,
outcome=outcome,
error=error,
is_final_attempt=is_final_attempt,
)
def _publish_embedding_event(
*,
provider: str,
model: str,
input_count: int,
input_tokens_estimate: int,
duration_ms: float,
outcome: Literal["success", "error", "cancelled"],
error: BaseException | None,
is_final_attempt: bool,
) -> None:
"""Build and emit the EmbeddingCallCompletedEvent. Best-effort."""
try:
from src.telemetry.events import (
EmbeddingCallCompletedEvent,
EmbeddingCallPurpose,
emit,
)
from src.utils.types import (
get_embedding_call_purpose,
get_embedding_parent_category,
get_embedding_run_id,
get_embedding_workspace_name,
)
# call_purpose travels via ContextVar so embedding callers don't have
# to thread it through every call site. Unknown values drop to None
# rather than raising — keeps telemetry resilient to drift.
purpose_slug = get_embedding_call_purpose()
call_purpose: EmbeddingCallPurpose | None = None
if purpose_slug:
try:
call_purpose = EmbeddingCallPurpose(purpose_slug)
except ValueError:
logger.debug(
"Unknown embedding_call_purpose=%r; emitting without",
purpose_slug,
)
emit(
EmbeddingCallCompletedEvent(
workspace_name=get_embedding_workspace_name(),
call_purpose=call_purpose,
parent_category=get_embedding_parent_category(),
provider=provider,
model=model,
input_count=input_count,
input_tokens_estimate=input_tokens_estimate,
duration_ms=duration_ms,
outcome=outcome,
is_final_attempt=is_final_attempt,
error_class=type(error).__name__ if error is not None else None,
run_id=get_embedding_run_id(),
)
)
except Exception: # pragma: no cover - telemetry must not raise
logger.debug("Failed to emit EmbeddingCallCompletedEvent", exc_info=True)
class BatchItem(NamedTuple):
"""A single item in a batch with its metadata."""
@ -20,6 +131,7 @@ class BatchItem(NamedTuple):
text: str
text_id: str
chunk_index: int
token_count: int
class _EmbeddingClient:
@ -93,22 +205,49 @@ class _EmbeddingClient:
f"Query exceeds maximum token limit of {self.max_embedding_tokens} tokens (got {token_count} tokens)"
)
# Bind the typed client at the dispatch site so pyright can narrow it
# for the closures without needing `assert isinstance(...)` (bandit
# B101). The closures close over the narrowed local, not `self.client`.
if isinstance(self.client, genai.Client):
response = await self.client.aio.models.embed_content(
gemini_client = self.client
async def _call_gemini() -> list[float]:
response = await gemini_client.aio.models.embed_content(
model=self.model,
contents=query,
config={"output_dimensionality": self.vector_dimensions},
)
if not response.embeddings or not response.embeddings[0].values:
raise ValueError("No embedding returned from Gemini API")
return self._validate_embedding_dimensions(
response.embeddings[0].values
)
return await _emit_embedding_call(
provider=self.transport,
model=self.model,
contents=query,
config={"output_dimensionality": self.vector_dimensions},
texts=[query],
input_tokens_estimate=token_count,
fn=_call_gemini,
)
if not response.embeddings or not response.embeddings[0].values:
raise ValueError("No embedding returned from Gemini API")
return self._validate_embedding_dimensions(response.embeddings[0].values)
else: # openai
openai_client = self.client
async def _call_openai() -> list[float]:
openai_kwargs: dict[str, Any] = {"model": self.model, "input": [query]}
if self.send_dimensions:
openai_kwargs["dimensions"] = self.vector_dimensions
response = await self.client.embeddings.create(**openai_kwargs)
response = await openai_client.embeddings.create(**openai_kwargs)
return self._validate_embedding_dimensions(response.data[0].embedding)
return await _emit_embedding_call(
provider=self.transport,
model=self.model,
texts=[query],
input_tokens_estimate=token_count,
fn=_call_openai,
)
async def simple_batch_embed(self, texts: list[str]) -> list[list[float]]:
"""
Simple batch embedding for a list of text strings.
@ -126,7 +265,11 @@ class _EmbeddingClient:
for i in range(0, len(texts), self.max_batch_size):
batch = texts[i : i + self.max_batch_size]
try:
async def _embed_batch(batch: list[str] = batch) -> list[list[float]]:
"""One provider call for one batch. Lifted into a closure so
_emit_embedding_call can time + emit + propagate errors."""
batch_embeddings: list[list[float]] = []
if isinstance(self.client, genai.Client):
# Type cast needed due to genai type signature complexity
response = await self.client.aio.models.embed_content(
@ -137,7 +280,7 @@ class _EmbeddingClient:
if response.embeddings:
for emb in response.embeddings:
if emb.values:
embeddings.append(
batch_embeddings.append(
self._validate_embedding_dimensions(emb.values)
)
else: # openai
@ -148,12 +291,26 @@ class _EmbeddingClient:
if self.send_dimensions:
openai_kwargs["dimensions"] = self.vector_dimensions
response = await self.client.embeddings.create(**openai_kwargs)
embeddings.extend(
batch_embeddings.extend(
[
self._validate_embedding_dimensions(data.embedding)
for data in response.data
]
)
return batch_embeddings
try:
# Pre-compute the tiktoken estimate ONCE for telemetry; the
# batch contents don't change between attempts.
tokens_estimate = sum(len(self.encoding.encode(t)) for t in batch)
batch_embeddings = await _emit_embedding_call(
provider=self.transport,
model=self.model,
texts=batch,
input_tokens_estimate=tokens_estimate,
fn=_embed_batch,
)
embeddings.extend(batch_embeddings)
except Exception as e:
# Check if it's a token limit error and re-raise as ValueError for consistency
if "token" in str(e).lower():
@ -248,7 +405,9 @@ class _EmbeddingClient:
current_batch = []
current_tokens = 0
current_batch.append(BatchItem(chunk_text, text_id, chunk_idx))
current_batch.append(
BatchItem(chunk_text, text_id, chunk_idx, chunk_tokens)
)
current_tokens += chunk_tokens
if current_batch:
@ -271,42 +430,53 @@ class _EmbeddingClient:
"""
last_exception: Exception | None = None
async def _call_provider() -> dict[str, dict[int, list[float]]]:
"""One provider call. Lifted out of the retry loop so
_emit_embedding_call emits a separate event per attempt — each
attempt is a distinct provider hit and shows up as its own line
item in analytics."""
result: dict[str, dict[int, list[float]]] = defaultdict(dict)
if isinstance(self.client, genai.Client):
response = await self.client.aio.models.embed_content(
model=self.model,
contents=[item.text for item in batch],
config={"output_dimensionality": self.vector_dimensions},
)
if response.embeddings:
for item, embedding in zip(batch, response.embeddings, strict=True):
if embedding.values:
result[item.text_id][item.chunk_index] = (
self._validate_embedding_dimensions(embedding.values)
)
else: # openai
openai_kwargs: dict[str, Any] = {
"model": self.model,
"input": [item.text for item in batch],
}
if self.send_dimensions:
openai_kwargs["dimensions"] = self.vector_dimensions
response = await self.client.embeddings.create(**openai_kwargs)
for item, embedding_data in zip(batch, response.data, strict=True):
result[item.text_id][item.chunk_index] = (
self._validate_embedding_dimensions(embedding_data.embedding)
)
return result
# Token counts were computed during chunk prep; reuse them here so the
# provider call doesn't re-encode every chunk just for the size proxy.
batch_tokens_estimate = sum(item.token_count for item in batch)
batch_texts = [item.text for item in batch]
for attempt in range(max_retries):
try:
# Organize embeddings by text_id and chunk_index
result: dict[str, dict[int, list[float]]] = defaultdict(dict)
if isinstance(self.client, genai.Client):
response = await self.client.aio.models.embed_content(
model=self.model,
contents=[item.text for item in batch],
config={"output_dimensionality": self.vector_dimensions},
)
if response.embeddings:
for item, embedding in zip(
batch, response.embeddings, strict=True
):
if embedding.values:
result[item.text_id][item.chunk_index] = (
self._validate_embedding_dimensions(
embedding.values
)
)
else: # openai
openai_kwargs: dict[str, Any] = {
"model": self.model,
"input": [item.text for item in batch],
}
if self.send_dimensions:
openai_kwargs["dimensions"] = self.vector_dimensions
response = await self.client.embeddings.create(**openai_kwargs)
for item, embedding_data in zip(batch, response.data, strict=True):
result[item.text_id][item.chunk_index] = (
self._validate_embedding_dimensions(
embedding_data.embedding
)
)
result = await _emit_embedding_call(
provider=self.transport,
model=self.model,
texts=batch_texts,
input_tokens_estimate=batch_tokens_estimate,
fn=_call_provider,
is_final_attempt=(attempt >= max_retries - 1),
)
return dict(result)
except Exception as e:

112
src/llm/api.py
View File

@ -38,6 +38,7 @@ from .types import (
HonchoLLMCallResponse,
HonchoLLMCallStreamChunk,
IterationCallback,
LLMTelemetryContext,
ReasoningEffortType,
StreamingResponseWithMetadata,
)
@ -73,6 +74,7 @@ async def honcho_llm_call(
max_input_tokens: int | None = None,
trace_name: str | None = None,
iteration_callback: IterationCallback | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> HonchoLLMCallResponse[M]: ...
@ -102,6 +104,7 @@ async def honcho_llm_call(
max_input_tokens: int | None = None,
trace_name: str | None = None,
iteration_callback: IterationCallback | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> HonchoLLMCallResponse[str]: ...
@ -131,6 +134,7 @@ async def honcho_llm_call(
max_input_tokens: int | None = None,
trace_name: str | None = None,
iteration_callback: IterationCallback | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> AsyncIterator[HonchoLLMCallStreamChunk] | StreamingResponseWithMetadata: ...
@ -160,6 +164,7 @@ async def honcho_llm_call(
max_input_tokens: int | None = None,
trace_name: str | None = None,
iteration_callback: IterationCallback | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> (
HonchoLLMCallResponse[Any]
| AsyncIterator[HonchoLLMCallStreamChunk]
@ -236,6 +241,8 @@ async def honcho_llm_call(
tools=tools,
tool_choice=tool_choice,
selected_config=plan.selected_config,
plan=plan,
telemetry=telemetry,
)
return await honcho_llm_call_inner(
plan.provider,
@ -254,6 +261,8 @@ async def honcho_llm_call(
tools=tools,
tool_choice=tool_choice,
selected_config=plan.selected_config,
plan=plan,
telemetry=telemetry,
)
decorated = _call_with_provider_selection
@ -306,9 +315,105 @@ async def honcho_llm_call(
# Tool-less path: call once and return.
if not tools or not tool_executor:
result: (
HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk]
) = await decorated()
# enforce `max_input_tokens` for tool-less calls too. Before
# this change, only `execute_tool_loop` consumed the kwarg — the
# deriver passed it but it was silently dropped, so the cap-hit
# signal it needed for RepresentationCompletedEvent could not be
# measured. Now we run the same message-list truncation helper
# and surface a `hit_input_token_cap` boolean on the response.
#
# The signal is purely token-based ("did the input exceed cap?")
# rather than message-count-based — the helper deliberately keeps
# the last conversation unit even when it's oversized (see
# truncate_messages_to_fit), so a single-message over-cap input
# (the deriver's prompt-only case) would otherwise silently fly
# through with hit=False. Token-based comparison catches it.
toolless_hit_input_token_cap = False
toolless_messages = messages
if max_input_tokens is not None:
from .conversation import count_message_tokens, truncate_messages_to_fit
base_messages = messages or [{"role": "user", "content": prompt}]
toolless_hit_input_token_cap = (
count_message_tokens(base_messages) > max_input_tokens
)
toolless_messages = truncate_messages_to_fit(
base_messages, max_input_tokens
)
# Re-bind the closure to use the truncated message list.
if toolless_messages is not None:
captured_messages = toolless_messages
async def _toolless_call() -> (
HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk]
):
plan = _get_attempt_plan()
# Branch on stream so each call site lands on the right
# `Literal[True]/False` overload — basedpyright won't infer
# which overload a runtime `bool` matches.
if stream:
return await honcho_llm_call_inner(
plan.provider,
plan.model,
prompt,
max_tokens,
response_model=response_model,
json_mode=json_mode,
temperature=effective_temperature(temperature),
stop_seqs=stop_seqs,
reasoning_effort=plan.reasoning_effort,
verbosity=verbosity,
thinking_budget_tokens=plan.thinking_budget_tokens,
stream=True,
client_override=plan.client,
tools=tools,
tool_choice=tool_choice,
selected_config=plan.selected_config,
plan=plan,
telemetry=telemetry,
messages=captured_messages,
)
return await honcho_llm_call_inner(
plan.provider,
plan.model,
prompt,
max_tokens,
response_model=response_model,
json_mode=json_mode,
temperature=effective_temperature(temperature),
stop_seqs=stop_seqs,
reasoning_effort=plan.reasoning_effort,
verbosity=verbosity,
thinking_budget_tokens=plan.thinking_budget_tokens,
stream=False,
client_override=plan.client,
tools=tools,
tool_choice=tool_choice,
selected_config=plan.selected_config,
plan=plan,
telemetry=telemetry,
messages=captured_messages,
)
wrapped = _toolless_call
if track_name:
wrapped = ai_track(track_name)(wrapped)
if enable_retry:
wrapped = retry(
stop=stop_after_attempt(retry_attempts),
wait=wait_exponential(multiplier=1, min=4, max=10),
before_sleep=before_retry_callback,
)(wrapped)
result: (
HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk]
) = await wrapped()
else:
result = await decorated()
if toolless_hit_input_token_cap and isinstance(result, HonchoLLMCallResponse):
result.hit_input_token_cap = True
if trace_name and isinstance(result, HonchoLLMCallResponse):
log_reasoning_trace(
task_type=trace_name,
@ -346,6 +451,7 @@ async def honcho_llm_call(
before_retry_callback=before_retry_callback,
stream_final=stream_final_only,
iteration_callback=iteration_callback,
telemetry=telemetry,
)
if trace_name and isinstance(result, HonchoLLMCallResponse):
log_reasoning_trace(

12
src/llm/backends/anthropic.py
View File

@ -34,11 +34,7 @@ class AnthropicBackend:
max_output_tokens: int | None = None,
extra_params: dict[str, Any] | None = None,
) -> CompletionResult:
del max_output_tokens
if thinking_effort is not None:
raise ValueError(
"Anthropic backend does not support thinking_effort; use thinking_budget_tokens instead"
)
del max_output_tokens, thinking_effort
request_messages, system_messages = self._extract_system(messages)
params: dict[str, Any] = {
@ -123,11 +119,7 @@ class AnthropicBackend:
extra_params: dict[str, Any] | None = None,
) -> AsyncIterator[StreamChunk]:
is_json_mode = self._json_mode(extra_params)
del max_output_tokens
if thinking_effort is not None:
raise ValueError(
"Anthropic backend does not support thinking_effort; use thinking_budget_tokens instead"
)
del max_output_tokens, thinking_effort
request_messages, system_messages = self._extract_system(messages)
params: dict[str, Any] = {

23
src/llm/backends/openai.py
View File

@ -128,11 +128,6 @@ class OpenAIBackend:
max_output_tokens: int | None = None,
extra_params: dict[str, Any] | None = None,
) -> CompletionResult:
if thinking_budget_tokens is not None:
raise ValidationException(
"OpenAI backend does not support thinking_budget_tokens; use thinking_effort instead"
)
params = self._build_params(
model=model,
messages=messages,
@ -142,6 +137,7 @@ class OpenAIBackend:
tools=tools,
tool_choice=tool_choice,
thinking_effort=thinking_effort,
thinking_budget_tokens=thinking_budget_tokens,
extra_params=extra_params,
)
@ -221,11 +217,6 @@ class OpenAIBackend:
max_output_tokens: int | None = None,
extra_params: dict[str, Any] | None = None,
) -> AsyncIterator[StreamChunk]:
if thinking_budget_tokens is not None:
raise ValidationException(
"OpenAI backend does not support thinking_budget_tokens; use thinking_effort instead"
)
params = self._build_params(
model=model,
messages=messages,
@ -235,6 +226,7 @@ class OpenAIBackend:
tools=tools,
tool_choice=tool_choice,
thinking_effort=thinking_effort,
thinking_budget_tokens=thinking_budget_tokens,
extra_params=extra_params,
)
params["stream"] = True
@ -284,6 +276,7 @@ class OpenAIBackend:
tools: list[dict[str, Any]] | None,
tool_choice: str | dict[str, Any] | None,
thinking_effort: str | None,
thinking_budget_tokens: int | None,
extra_params: dict[str, Any] | None,
) -> dict[str, Any]:
params: dict[str, Any] = {
@ -304,6 +297,16 @@ class OpenAIBackend:
if thinking_effort:
params["reasoning_effort"] = thinking_effort
# Token-budget style thinking is not part of the native OpenAI API, but
# OpenAI-compatible proxies (OpenRouter, etc.) accept a `reasoning` object
# on the request body. Pass through via extra_body so it reaches those
# backends; operators on providers that need a different shape (vLLM,
# Fireworks, ...) can override via ModelConfig.provider_params.
if thinking_budget_tokens is not None and thinking_budget_tokens > 0:
params.setdefault("extra_body", {}).setdefault("reasoning", {})[
"max_tokens"
] = thinking_budget_tokens
if stop:
params["stop"] = stop
if tools:

236
src/llm/executor.py
View File

@ -11,6 +11,9 @@ Used by:
from __future__ import annotations
import asyncio
import logging
import time
from collections.abc import AsyncIterator
from typing import Any, Literal, TypeVar, overload
@ -23,17 +26,35 @@ from .backend import StreamChunk as BackendStreamChunk
from .backend import ToolCallResult
from .registry import CLIENTS, backend_for_provider
from .request_builder import execute_completion, execute_stream
from .runtime import effective_config_for_call
from .runtime import AttemptPlan, effective_config_for_call
from .types import (
HonchoLLMCallResponse,
HonchoLLMCallStreamChunk,
LLMTelemetryContext,
ProviderClient,
ReasoningEffortType,
)
logger = logging.getLogger(__name__)
M = TypeVar("M", bound=BaseModel)
def _outcome_from_error(
err: BaseException | None,
) -> Literal["success", "error", "cancelled"]:
"""Map a finally-block error into the telemetry outcome literal.
CancelledError is a normal control-flow event (client disconnect, server
shutdown) — surface it distinctly so it doesn't pollute error-rate alerts.
"""
if err is None:
return "success"
if isinstance(err, asyncio.CancelledError):
return "cancelled"
return "error"
def _tool_call_result_to_dict(tool_call: ToolCallResult) -> dict[str, Any]:
result = {
"id": tool_call.id,
@ -45,6 +66,101 @@ def _tool_call_result_to_dict(tool_call: ToolCallResult) -> dict[str, Any]:
return result
def _emit_llm_call_completed(
*,
plan: AttemptPlan | None,
telemetry: LLMTelemetryContext | None,
provider: ModelTransport,
model: str,
max_tokens: int,
duration_ms: float,
has_tools: bool,
was_stream: bool,
outcome: Literal["success", "error", "cancelled"],
result: BackendCompletionResult | None,
error: BaseException | None,
) -> None:
"""Build and emit an LLMCallCompletedEvent. Best-effort; swallows errors so
telemetry failures never bleed into the LLM call path."""
try:
from src.telemetry.events import CallPurpose, LLMCallCompletedEvent, emit
# call_purpose is a string slug on LLMTelemetryContext; validate against
# the enum here (silent drop on unknown values keeps telemetry resilient).
call_purpose: CallPurpose | None = None
if telemetry is not None and telemetry.call_purpose:
try:
call_purpose = CallPurpose(telemetry.call_purpose)
except ValueError:
logger.debug(
"Unknown LLMTelemetryContext.call_purpose=%r; emitting without",
telemetry.call_purpose,
)
attempt = plan.attempt if plan is not None else 1
retry_attempts = plan.retry_attempts if plan is not None else 1
was_fallback = plan.is_fallback if plan is not None else False
emit(
LLMCallCompletedEvent(
workspace_name=(telemetry.workspace_name if telemetry else None),
call_purpose=call_purpose,
parent_category=(telemetry.parent_category if telemetry else None),
transport=provider,
provider_label=_infer_provider_label(provider, model, plan),
model=model,
effective_max_output_tokens=max_tokens,
provider_input_tokens=(result.input_tokens if result else 0),
provider_output_tokens=(result.output_tokens if result else 0),
cache_read_tokens=(result.cache_read_input_tokens if result else 0),
cache_creation_tokens=(
result.cache_creation_input_tokens if result else 0
),
finish_reason=(result.finish_reason if result else None),
outcome=outcome,
is_final_attempt=(attempt >= retry_attempts),
error_class=(type(error).__name__ if error else None),
attempt=attempt,
retry_attempts=retry_attempts,
was_fallback=was_fallback,
duration_ms=duration_ms,
has_tools=has_tools,
tool_call_count=(len(result.tool_calls) if result else 0),
was_stream=was_stream,
run_id=(telemetry.run_id if telemetry else None),
iteration=(telemetry.iteration if telemetry else None),
)
)
except Exception: # pragma: no cover - telemetry must not raise
logger.debug("Failed to emit LLMCallCompletedEvent", exc_info=True)
def _infer_provider_label(
_transport: ModelTransport, model: str, plan: AttemptPlan | None
) -> str | None:
"""Best-effort vendor inference for relay setups.
When the model name carries a vendor prefix (OpenRouter convention:
"anthropic/claude-..." routed through the openai transport), surface that
as the provider label so analytics can distinguish "openai-the-vendor"
from "openai-the-transport-pointing-at-openrouter".
`_transport` is currently unused but kept on the signature so callers stay
explicit about which transport produced the call — future inference rules
(e.g. anthropic-direct vs anthropic-via-relay) may need it.
"""
if "/" in model:
return model.split("/", 1)[0]
# Defensive getattr — selected_config may be a stub in tests or a config
# without an explicit base_url. Either way the inference is best-effort.
base_url = (
getattr(plan.selected_config, "base_url", None) if plan is not None else None
)
if base_url and "openrouter" in base_url.lower():
return "openrouter"
return None
def completion_result_to_response(
result: BackendCompletionResult,
) -> HonchoLLMCallResponse[Any]:
@ -92,6 +208,8 @@ async def honcho_llm_call_inner(
tool_choice: str | dict[str, Any] | None = None,
messages: list[dict[str, Any]] | None = None,
selected_config: ModelConfig | None = None,
plan: AttemptPlan | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> HonchoLLMCallResponse[M]: ...
@ -114,6 +232,8 @@ async def honcho_llm_call_inner(
tool_choice: str | dict[str, Any] | None = None,
messages: list[dict[str, Any]] | None = None,
selected_config: ModelConfig | None = None,
plan: AttemptPlan | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> HonchoLLMCallResponse[str]: ...
@ -136,6 +256,8 @@ async def honcho_llm_call_inner(
tool_choice: str | dict[str, Any] | None = None,
messages: list[dict[str, Any]] | None = None,
selected_config: ModelConfig | None = None,
plan: AttemptPlan | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> AsyncIterator[HonchoLLMCallStreamChunk]: ...
@ -157,11 +279,22 @@ async def honcho_llm_call_inner(
tool_choice: str | dict[str, Any] | None = None,
messages: list[dict[str, Any]] | None = None,
selected_config: ModelConfig | None = None,
plan: AttemptPlan | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> HonchoLLMCallResponse[Any] | AsyncIterator[HonchoLLMCallStreamChunk]:
"""One backend call. No retry, no fallback, no tool loop.
The outer src/llm/api.py `honcho_llm_call` handles retry + fallback +
tool orchestration on top of this.
Emits one LLMCallCompletedEvent per call. On the stream path, setup
runs inside the awaited coroutine (so it sits inside any outer retry
wrapper) and emits its own event on failure; the wrapping generator
emits a second event from its finally block after drain completes or
raises. `was_stream` is True for streamed calls. Token counts are
zero on the stream path because provider token totals aren't surfaced
post-stream at this layer; aggregate envelopes (DialecticCompletedEvent
etc.) carry the accurate totals.
"""
client = client_override or CLIENTS.get(provider)
if client is None:
@ -187,8 +320,18 @@ async def honcho_llm_call_inner(
call_extras: dict[str, Any] = {"json_mode": json_mode, "verbosity": verbosity}
if stream:
async def _stream() -> AsyncIterator[HonchoLLMCallStreamChunk]:
# Stream path: setup must run inside the awaited coroutine so it
# sits inside the outer retry wrapper (tool_loop.stream_final_response
# wraps `await honcho_llm_call_inner(stream=True)` with tenacity).
# If we deferred `execute_stream` into the generator body, a transient
# setup failure (rate-limit, auth, network) would surface at first
# iteration — outside retry — and crash the request.
#
# Drain failures stay unretried by design (chunks may have already
# been sent to the client) and report via the wrapper's finally.
# Token counts are 0 on this path; aggregate envelopes carry totals.
stream_start = time.perf_counter()
try:
stream_iter = await execute_stream(
backend,
effective_config,
@ -200,23 +343,80 @@ async def honcho_llm_call_inner(
cache_policy=effective_config.cache_policy,
extra_params=call_extras,
)
async for chunk in stream_iter:
yield stream_chunk_to_response_chunk(chunk)
except BaseException as exc:
_emit_llm_call_completed(
plan=plan,
telemetry=telemetry,
provider=provider,
model=model,
max_tokens=max_tokens,
duration_ms=(time.perf_counter() - stream_start) * 1000,
has_tools=bool(tools),
was_stream=True,
outcome=_outcome_from_error(exc),
result=None,
error=exc,
)
raise
return _stream()
async def _wrap_stream() -> AsyncIterator[HonchoLLMCallStreamChunk]:
stream_error: BaseException | None = None
try:
async for chunk in stream_iter:
yield stream_chunk_to_response_chunk(chunk)
except BaseException as exc:
stream_error = exc
raise
finally:
_emit_llm_call_completed(
plan=plan,
telemetry=telemetry,
provider=provider,
model=model,
max_tokens=max_tokens,
duration_ms=(time.perf_counter() - stream_start) * 1000,
has_tools=bool(tools),
was_stream=True,
outcome=_outcome_from_error(stream_error),
result=None,
error=stream_error,
)
result = await execute_completion(
backend,
effective_config,
messages=messages,
max_tokens=max_tokens,
tools=tools,
tool_choice=tool_choice,
response_format=response_model,
cache_policy=effective_config.cache_policy,
extra_params=call_extras,
)
return completion_result_to_response(result)
return _wrap_stream()
start = time.perf_counter()
backend_result: BackendCompletionResult | None = None
error: BaseException | None = None
try:
backend_result = await execute_completion(
backend,
effective_config,
messages=messages,
max_tokens=max_tokens,
tools=tools,
tool_choice=tool_choice,
response_format=response_model,
cache_policy=effective_config.cache_policy,
extra_params=call_extras,
)
return completion_result_to_response(backend_result)
except BaseException as exc:
error = exc
raise
finally:
_emit_llm_call_completed(
plan=plan,
telemetry=telemetry,
provider=provider,
model=model,
max_tokens=max_tokens,
duration_ms=(time.perf_counter() - start) * 1000,
has_tools=bool(tools),
was_stream=False,
outcome=_outcome_from_error(error),
result=backend_result,
error=error,
)
__all__ = [

6
src/llm/runtime.py
View File

@ -75,6 +75,9 @@ class AttemptPlan:
thinking_budget_tokens: int | None
reasoning_effort: ReasoningEffortType
selected_config: ModelConfig
attempt: int
retry_attempts: int
is_fallback: bool
def resolve_runtime_model_config(
@ -166,6 +169,9 @@ def plan_attempt(
thinking_budget_tokens=attempt_thinking_budget,
reasoning_effort=attempt_reasoning_effort,
selected_config=selected,
attempt=attempt,
retry_attempts=retry_attempts,
is_fallback=not is_primary,
)

195
src/llm/tool_loop.py
View File

@ -11,16 +11,24 @@
from __future__ import annotations
import dataclasses
import functools
import logging
from collections.abc import AsyncIterator, Callable
from typing import Any
from collections.abc import AsyncIterator, Awaitable, Callable
from typing import Any, ParamSpec, TypeVar
from pydantic import BaseModel
from tenacity import retry, stop_after_attempt, wait_exponential
from src.config import ModelTransport
from src.exceptions import ValidationException
from src.utils.types import set_current_iteration
from src.utils.types import (
get_last_tool_metadata,
iteration_scope,
set_current_iteration,
set_current_tool_call_seq,
set_last_tool_metadata,
)
from .executor import honcho_llm_call_inner
from .registry import history_adapter_for_provider
@ -34,10 +42,106 @@ from .types import (
HonchoLLMCallStreamChunk,
IterationCallback,
IterationData,
LLMTelemetryContext,
StreamingResponseWithMetadata,
VerbosityType,
)
_P = ParamSpec("_P")
_R = TypeVar("_R")
def _with_iteration_scope(
fn: Callable[_P, Awaitable[_R]],
) -> Callable[_P, Awaitable[_R]]:
"""Wrap an async tool-loop entry point in `iteration_scope()` so the
per-iteration ContextVars (iteration, tool_call_seq, provider id, last
tool metadata) are reset to their pre-call values on exit. Defensive
against subsequent loops in the same asyncio Task observing stale state.
"""
@functools.wraps(fn)
async def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _R:
with iteration_scope():
return await fn(*args, **kwargs)
return wrapper
def _telemetry_for_iteration(
base: LLMTelemetryContext | None, iteration: int
) -> LLMTelemetryContext | None:
"""Return a copy of `base` with `iteration` set, or None if no base.
We always copy rather than mutate the caller-supplied context so callers
that pass the same context into multiple `honcho_llm_call` invocations
don't see drift across concurrent runs.
"""
if base is None:
return None
return LLMTelemetryContext(
workspace_name=base.workspace_name,
call_purpose=base.call_purpose,
parent_category=base.parent_category,
run_id=base.run_id,
iteration=iteration,
observer=base.observer,
observed=base.observed,
peer_name=base.peer_name,
agent_type=base.agent_type,
)
def _emit_agent_iteration(
telemetry: LLMTelemetryContext | None,
iteration: int,
response: HonchoLLMCallResponse[Any],
) -> None:
"""emit AgentIterationEvent after each per-iteration LLM response.
Fired immediately after `response = await call_func()` in the per-iteration
loop AND after the max-iteration synthesis call. Emitted regardless of
whether the model requested tool calls — the no-tool terminating iteration
still counts as an iteration for cost calibration.
Skipped when telemetry context is missing or lacks the required agent
identifiers (no agent → no agent.iteration event).
"""
if telemetry is None or not telemetry.run_id:
return
if not telemetry.parent_category or not telemetry.agent_type:
# Without agent_type / parent_category we can't fill the event's
# required fields. Skip rather than emit a half-populated event.
return
if not telemetry.workspace_name:
return
try:
# Local import: keeps src/llm/ free of a hard dependency on telemetry
# at import time so the LLM layer remains usable in unit tests that
# don't initialize the telemetry stack.
from src.telemetry.events import AgentIterationEvent, emit
emit(
AgentIterationEvent(
run_id=telemetry.run_id,
parent_category=telemetry.parent_category,
agent_type=telemetry.agent_type,
workspace_name=telemetry.workspace_name,
observer=telemetry.observer,
observed=telemetry.observed,
peer_name=telemetry.peer_name,
iteration=iteration,
tool_calls=[tc["name"] for tc in response.tool_calls_made],
input_tokens=response.input_tokens,
output_tokens=response.output_tokens,
cache_read_tokens=response.cache_read_input_tokens or 0,
cache_creation_tokens=response.cache_creation_input_tokens or 0,
)
)
except Exception: # pragma: no cover - telemetry must not raise
logger.debug("Failed to emit AgentIterationEvent", exc_info=True)
logger = logging.getLogger(__name__)
# Bounds for max_tool_iterations to prevent runaway loops.
@ -98,6 +202,7 @@ async def stream_final_response(
enable_retry: bool,
retry_attempts: int,
before_retry_callback: Callable[[Any], None],
telemetry: LLMTelemetryContext | None = None,
) -> AsyncIterator[HonchoLLMCallStreamChunk]:
"""Stream the final response after tool execution is complete.
@ -110,7 +215,24 @@ async def stream_final_response(
streaming call against the same pinned model for transient errors.
"""
# Bump the per-retry attempt index inside `_setup_stream`. The pinned
# `winning_plan.attempt` is frozen from before retries started; without
# this counter, every retried stream-setup emit reports the same attempt
# value — telemetry can't tell the retry sequence apart.
stream_attempt = 0
async def _setup_stream() -> AsyncIterator[HonchoLLMCallStreamChunk]:
nonlocal stream_attempt
stream_attempt += 1
# `dataclasses.replace` produces a per-attempt plan with the bumped
# `attempt` and the real `retry_attempts` budget so the executor's
# LLMCallCompletedEvent reports attempt=1/2/3 and is_final_attempt
# correctly across the retry sequence.
plan_for_attempt = dataclasses.replace(
winning_plan,
attempt=stream_attempt,
retry_attempts=retry_attempts,
)
return await honcho_llm_call_inner(
winning_plan.provider,
winning_plan.model,
@ -129,6 +251,8 @@ async def stream_final_response(
tool_choice=None,
messages=conversation_messages,
selected_config=winning_plan.selected_config,
plan=plan_for_attempt,
telemetry=telemetry,
)
if enable_retry:
@ -145,6 +269,7 @@ async def stream_final_response(
yield chunk
@_with_iteration_scope
async def execute_tool_loop(
*,
prompt: str,
@ -166,6 +291,7 @@ async def execute_tool_loop(
before_retry_callback: Callable[[Any], None],
stream_final: bool = False,
iteration_callback: IterationCallback | None = None,
telemetry: LLMTelemetryContext | None = None,
) -> HonchoLLMCallResponse[Any] | StreamingResponseWithMetadata:
"""Run the iterative tool calling loop for agentic LLM interactions.
@ -179,7 +305,7 @@ async def execute_tool_loop(
Final HonchoLLMCallResponse with accumulated token counts and tool call
history, or a StreamingResponseWithMetadata if stream_final=True.
"""
from .conversation import truncate_messages_to_fit
from .conversation import count_message_tokens, truncate_messages_to_fit
if not MIN_TOOL_ITERATIONS <= max_tool_iterations <= MAX_TOOL_ITERATIONS:
raise ValidationException(
@ -199,6 +325,14 @@ async def execute_tool_loop(
total_cache_creation_tokens = 0
total_cache_read_tokens = 0
empty_response_retries = 0
# Latch — set when any iteration's input exceeded `max_input_tokens`.
# Token-based rather than message-count-based: catches both "messages
# got dropped" and "couldn't drop the last unit but still over cap."
# Stamped onto the final response so
# RepresentationCompletedEvent.hit_input_token_cap and
# DialecticCompletedEvent.hit_input_token_cap reflect the cap hit
# (the toolless path tracks this in src/llm/api.py:325-340).
hit_input_token_cap = False
# Track effective tool_choice — switches from "required"/"any" to "auto" after iter 1.
effective_tool_choice = tool_choice
@ -208,6 +342,8 @@ async def execute_tool_loop(
logger.debug(f"Tool execution iteration {iteration + 1}/{max_tool_iterations}")
if max_input_tokens is not None:
if count_message_tokens(conversation_messages) > max_input_tokens:
hit_input_token_cap = True
conversation_messages = truncate_messages_to_fit(
conversation_messages, max_input_tokens
)
@ -215,6 +351,7 @@ async def execute_tool_loop(
async def _call_with_messages(
effective_tool_choice: str | dict[str, Any] | None = effective_tool_choice,
conversation_messages: list[dict[str, Any]] = conversation_messages,
iteration_for_call: int = iteration + 1,
) -> HonchoLLMCallResponse[Any]:
plan = get_attempt_plan()
return await honcho_llm_call_inner(
@ -235,6 +372,8 @@ async def execute_tool_loop(
tool_choice=effective_tool_choice,
messages=conversation_messages,
selected_config=plan.selected_config,
plan=plan,
telemetry=_telemetry_for_iteration(telemetry, iteration_for_call),
)
if enable_retry:
@ -253,6 +392,11 @@ async def execute_tool_loop(
total_cache_creation_tokens += response.cache_creation_input_tokens
total_cache_read_tokens += response.cache_read_input_tokens
# emit one AgentIterationEvent per LLM response BEFORE the
# no-tool early return. The terminating iteration counts too — it has
# an empty tool_calls list and is essential for cost calibration.
_emit_agent_iteration(telemetry, iteration + 1, response)
if not response.tool_calls_made:
logger.debug("No tool calls in response, finishing")
@ -293,6 +437,7 @@ async def execute_tool_loop(
enable_retry=enable_retry,
retry_attempts=retry_attempts,
before_retry_callback=before_retry_callback,
telemetry=_telemetry_for_iteration(telemetry, iteration + 1),
)
return StreamingResponseWithMetadata(
stream=stream,
@ -303,6 +448,7 @@ async def execute_tool_loop(
cache_read_input_tokens=total_cache_read_tokens,
thinking_content=response.thinking_content,
iterations=iteration + 1,
hit_input_token_cap=hit_input_token_cap,
)
response.tool_calls_made = all_tool_calls
@ -311,6 +457,9 @@ async def execute_tool_loop(
response.cache_creation_input_tokens = total_cache_creation_tokens
response.cache_read_input_tokens = total_cache_read_tokens
response.iterations = iteration + 1
response.hit_input_token_cap = (
response.hit_input_token_cap or hit_input_token_cap
)
return response
current_provider = get_attempt_plan().provider
@ -328,15 +477,27 @@ async def execute_tool_loop(
set_current_iteration(iteration + 1)
tool_results: list[dict[str, Any]] = []
for tool_call in response.tool_calls_made:
for seq, tool_call in enumerate(response.tool_calls_made):
tool_name = tool_call["name"]
tool_input = tool_call["input"]
tool_id = tool_call.get("id", "")
logger.debug(f"Executing tool: {tool_name}")
# the executor closure reads these from
# ContextVars to populate AgentToolCallCompletedEvent. Set BEFORE
# the executor call so two calls to the same tool in one iteration
# get distinct seq values. Reset last-tool metadata so we never
# observe stale state from a prior call.
set_current_tool_call_seq(seq, tool_id or None)
set_last_tool_metadata({})
try:
tool_result = await tool_executor(tool_name, tool_input)
# Stash ToolResult.metadata on all_tool_calls so
# specialist rollups can read created/deleted observation
# counts without round-tripping through the event store.
tool_result_metadata = get_last_tool_metadata()
tool_results.append(
{
"tool_id": tool_id,
@ -349,6 +510,7 @@ async def execute_tool_loop(
"tool_name": tool_name,
"tool_input": tool_input,
"tool_result": tool_result,
"tool_result_metadata": tool_result_metadata,
}
)
except Exception as e:
@ -391,6 +553,10 @@ async def execute_tool_loop(
f"Tool execution loop reached max iterations ({max_tool_iterations})"
)
# The max-iteration synthesis call gets iteration N+1 in telemetry so 's
# AgentIterationEvent and this LLMCallCompletedEvent line up sequentially.
synthesis_iteration = iteration + 1
synthesis_prompt = (
"You have reached the maximum number of tool calls. "
"Based on all the information you have gathered, provide your final response now. "
@ -401,6 +567,8 @@ async def execute_tool_loop(
# Truncate again — the per-iteration truncate ran before the last tool
# call, so appending synthesis_prompt could nudge us back over the cap.
if max_input_tokens is not None:
if count_message_tokens(conversation_messages) > max_input_tokens:
hit_input_token_cap = True
conversation_messages = truncate_messages_to_fit(
conversation_messages, max_input_tokens
)
@ -422,6 +590,7 @@ async def execute_tool_loop(
enable_retry=enable_retry,
retry_attempts=retry_attempts,
before_retry_callback=before_retry_callback,
telemetry=_telemetry_for_iteration(telemetry, synthesis_iteration),
)
return StreamingResponseWithMetadata(
stream=stream,
@ -432,6 +601,7 @@ async def execute_tool_loop(
cache_read_input_tokens=total_cache_read_tokens,
thinking_content=None,
iterations=iteration + 1,
hit_input_token_cap=hit_input_token_cap,
)
current_attempt.set(1)
@ -456,6 +626,8 @@ async def execute_tool_loop(
tool_choice=None,
messages=conversation_messages,
selected_config=plan.selected_config,
plan=plan,
telemetry=_telemetry_for_iteration(telemetry, synthesis_iteration),
)
if enable_retry:
@ -468,6 +640,16 @@ async def execute_tool_loop(
final_call_func = _final_call
final_response = await final_call_func()
# emit the synthesis-call iteration event BEFORE merging cumulative
# totals onto final_response below — otherwise the event's per-iteration
# token counts would double-count the running totals.
_emit_agent_iteration(
_telemetry_for_iteration(telemetry, synthesis_iteration),
synthesis_iteration,
final_response,
)
final_response.tool_calls_made = all_tool_calls
final_response.iterations = iteration + 1
final_response.input_tokens = total_input_tokens + final_response.input_tokens
@ -478,6 +660,9 @@ async def execute_tool_loop(
final_response.cache_read_input_tokens = (
total_cache_read_tokens + final_response.cache_read_input_tokens
)
final_response.hit_input_token_cap = (
final_response.hit_input_token_cap or hit_input_token_cap
)
return final_response

74
src/llm/types.py
View File

@ -45,6 +45,38 @@ class IterationData:
"""Tokens written to cache in this iteration."""
@dataclass
class LLMTelemetryContext:
"""Context threaded through honcho_llm_call → honcho_llm_call_inner so the
LLMCallCompletedEvent emitter (and AgentIterationEvent emitter)
can attribute calls to the right workspace / agent / iteration without
re-deriving any of it from ambient state.
Iteration is mutable: tool_loop updates this field before each inner call.
NOT read from set_current_iteration ContextVar — that fires after the LLM
call returns, so reading it from the executor would yield stale values.
"""
workspace_name: str | None = None
# call_purpose carries the same string as src.telemetry.events.llm.CallPurpose values.
# Stored as str rather than importing the enum here to keep src/llm/ free of
# telemetry imports — the emitter validates against the enum.
call_purpose: str | None = None
parent_category: str | None = None
run_id: str | None = None
iteration: int | None = None
# Optional peer context (dream agents pass observer/observed; dialectic
# passes peer_name). Kept here so AgentIterationEvent can populate
# them without a separate threading path.
observer: str | None = None
observed: str | None = None
peer_name: str | None = None
# Tool-related context: agent_type is the human-readable identifier of the
# agent — dialectic/deduction/induction. Used by agent iteration
# event and tool call event.
agent_type: str | None = None
IterationCallback = Callable[[IterationData], None]
@ -71,6 +103,12 @@ class HonchoLLMCallResponse(BaseModel, Generic[T]):
thinking_blocks: list[dict[str, Any]] = Field(default_factory=list)
# OpenRouter reasoning_details for Gemini models — must be preserved across turns.
reasoning_details: list[dict[str, Any]] = Field(default_factory=list)
# True when the original input exceeded `max_input_tokens` — covers
# both "messages were dropped" and "couldn't drop the last unit and
# remaining tokens still exceeded the cap" (the deriver's prompt-only
# case). Maps 1:1 to `RepresentationCompletedEvent.hit_input_token_cap`
# and `DialecticCompletedEvent.hit_input_token_cap`.
hit_input_token_cap: bool = False
class HonchoLLMCallStreamChunk(BaseModel):
@ -87,6 +125,15 @@ class StreamingResponseWithMetadata:
Lets callers read tool_calls_made / token counts / thinking_content from
the tool-execution phase while still iterating the final streamed answer.
`output_tokens` is updated AS THE STREAM DRAINS — `__aiter__` wraps the
underlying iterator and accumulates the latest non-None `output_tokens`
value reported by chunk usage. Providers like OpenAI (with
`stream_options.include_usage`) and Anthropic emit a final usage chunk
with the cumulative count, so the post-drain `output_tokens` value
reflects tool-loop output + final-stream output. Callers that read
`output_tokens` AFTER fully iterating the stream get the true total;
callers that read it before drain see only the tool-loop portion.
"""
_stream: AsyncIterator[HonchoLLMCallStreamChunk]
@ -97,6 +144,7 @@ class StreamingResponseWithMetadata:
cache_read_input_tokens: int
thinking_content: str | None
iterations: int
hit_input_token_cap: bool
def __init__(
self,
@ -108,6 +156,7 @@ class StreamingResponseWithMetadata:
cache_read_input_tokens: int,
thinking_content: str | None = None,
iterations: int = 0,
hit_input_token_cap: bool = False,
):
self._stream = stream
self.tool_calls_made = tool_calls_made
@ -117,9 +166,31 @@ class StreamingResponseWithMetadata:
self.cache_read_input_tokens = cache_read_input_tokens
self.thinking_content = thinking_content
self.iterations = iterations
self.hit_input_token_cap = hit_input_token_cap
def __aiter__(self) -> AsyncIterator[HonchoLLMCallStreamChunk]:
return self._stream.__aiter__()
# Wrap the underlying iterator to capture final-stream output_tokens
# from chunks as they arrive. Providers emit a usage chunk at end-of-
# stream with the cumulative output_tokens count; we fold it into
# self.output_tokens (which carries the tool-loop running total at
# construction) so the post-drain value reflects the true cost.
return self._iterate_with_usage_capture()
async def _iterate_with_usage_capture(
self,
) -> AsyncIterator[HonchoLLMCallStreamChunk]:
final_stream_output_tokens = 0
async for chunk in self._stream:
if chunk.output_tokens is not None:
# Take the LATEST value, not the sum — providers report
# the cumulative usage in the final chunk, not deltas.
final_stream_output_tokens = chunk.output_tokens
yield chunk
# Stream drained — fold the final-stream output tokens into the
# tool-loop totals so DialecticCompletedEvent / downstream readers
# see the true cost.
if final_stream_output_tokens > 0:
self.output_tokens += final_stream_output_tokens
async def __anext__(self) -> HonchoLLMCallStreamChunk:
return await self._stream.__anext__()
@ -130,6 +201,7 @@ __all__ = [
"HonchoLLMCallStreamChunk",
"IterationCallback",
"IterationData",
"LLMTelemetryContext",
"ProviderClient",
"ReasoningEffortType",
"StreamingResponseWithMetadata",

3
src/main.py
View File

@ -15,6 +15,7 @@ from pydantic import ValidationError
from sentry_sdk.integrations.fastapi import FastApiIntegration
from sentry_sdk.integrations.starlette import StarletteIntegration
from src._version import HONCHO_VERSION
from src.cache.client import close_cache, init_cache
from src.config import settings
from src.db import engine, request_context
@ -161,7 +162,7 @@ app = FastAPI(
title="Honcho API",
summary="The Identity Layer for the Agentic World",
description="""Honcho is a platform for giving agents user-centric memory and social cognition.""",
version="3.0.6",
version=HONCHO_VERSION,
contact={
"name": "Plastic Labs",
"url": "https://honcho.dev",

14
src/reconciler/sync_vectors.py
View File

@ -22,6 +22,8 @@ from src.config import settings
from src.dependencies import tracked_db
from src.embedding_client import embedding_client
from src.exceptions import VectorStoreError
from src.telemetry.events import EmbeddingCallPurpose
from src.utils.types import embedding_call_purpose
from src.vector_store import VectorRecord, VectorStore, get_external_vector_store
logger = logging.getLogger(__name__)
@ -210,7 +212,11 @@ async def _sync_documents(
if docs_needing_embed:
try:
contents = [doc.content for doc in docs_needing_embed]
new_embeddings = await embedding_client.simple_batch_embed(contents)
with embedding_call_purpose(
EmbeddingCallPurpose.VECTOR_SYNC.value,
parent_category="reconciliation",
):
new_embeddings = await embedding_client.simple_batch_embed(contents)
if len(new_embeddings) != len(docs_needing_embed):
logger.warning(
@ -332,7 +338,11 @@ async def _sync_message_embeddings(
if embs_needing_embed:
try:
contents = [emb.content for emb in embs_needing_embed]
new_embeddings = await embedding_client.simple_batch_embed(contents)
with embedding_call_purpose(
EmbeddingCallPurpose.VECTOR_SYNC.value,
parent_category="reconciliation",
):
new_embeddings = await embedding_client.simple_batch_embed(contents)
if len(new_embeddings) != len(embs_needing_embed):
logger.warning(

25
src/routers/conclusions.py
View File

@ -9,6 +9,8 @@ from src import crud, schemas
from src.dependencies import db
from src.exceptions import ResourceNotFoundException, ValidationException
from src.security import require_auth
from src.telemetry.events import EmbeddingCallPurpose
from src.utils.types import embedding_call_purpose
logger = logging.getLogger(__name__)
@ -111,16 +113,21 @@ async def query_conclusions(
"observer and observed must be specified for semantic search"
)
documents = await crud.query_documents(
db,
with embedding_call_purpose(
EmbeddingCallPurpose.GENERIC_DOCUMENT_SEARCH.value,
workspace_name=workspace_id,
query=body.query,
observer=observer,
observed=observed,
filters=body.filters,
max_distance=body.distance,
top_k=body.top_k,
)
parent_category="api",
):
documents = await crud.query_documents(
db,
workspace_name=workspace_id,
query=body.query,
observer=observer,
observed=observed,
filters=body.filters,
max_distance=body.distance,
top_k=body.top_k,
)
return [schemas.Conclusion.model_validate(doc) for doc in documents]

41
src/routers/messages.py
View File

@ -23,6 +23,7 @@ from src.deriver import enqueue
from src.exceptions import FileTooLargeError, ResourceNotFoundException
from src.security import require_auth
from src.telemetry import prometheus_metrics
from src.telemetry.events import FileUploadedEvent, MessageCreatedEvent, emit
from src.utils.files import process_file_uploads_for_messages
logger = logging.getLogger(__name__)
@ -107,6 +108,17 @@ async def create_messages_for_session(
workspace_name=workspace_id,
)
emit(
MessageCreatedEvent(
workspace_name=workspace_id,
session_name=session_id,
message_count=len(created_messages),
total_tokens=sum(message.token_count for message in created_messages),
source="api",
last_message_id=created_messages[-1].public_id,
)
)
# Enqueue for processing (existing logic)
payloads = [
{
@ -206,6 +218,35 @@ async def create_messages_with_file(
workspace_name=workspace_id,
)
# An empty extracted file (no chunks) leaves both lists empty. Skip the
# telemetry in that case rather than indexing into [].
if all_message_data and created_messages:
file_metadata = all_message_data[0]["file_metadata"]
total_tokens = sum(message.token_count for message in created_messages)
emit(
FileUploadedEvent(
workspace_name=workspace_id,
session_name=session_id,
peer_name=form_data.peer_id,
file_id=str(file_metadata["file_id"]),
filename=file.filename,
content_type=file.content_type,
file_size_bytes=file.size,
message_count=len(created_messages),
total_tokens=total_tokens,
)
)
emit(
MessageCreatedEvent(
workspace_name=workspace_id,
session_name=session_id,
message_count=len(created_messages),
total_tokens=total_tokens,
source="file_upload",
last_message_id=created_messages[-1].public_id,
)
)
return created_messages

64
src/routers/peers.py
View File

@ -1,6 +1,10 @@
"""FastAPI routes for peer resources and peer-scoped operations."""
import json
import logging
from collections.abc import AsyncIterator
from contextlib import suppress
from time import perf_counter
from fastapi import APIRouter, Body, Depends, Path, Query, Response
from fastapi.responses import StreamingResponse
@ -12,10 +16,13 @@ from src import crud, schemas
from src.config import settings
from src.dependencies import db, tracked_db
from src.dialectic.chat import agentic_chat, agentic_chat_stream
from src.embedding_client import embedding_client
from src.exceptions import AuthenticationException, ResourceNotFoundException
from src.security import JWTParams, require_auth
from src.telemetry import prometheus_metrics
from src.telemetry.events import EmbeddingCallPurpose, GetContextEvent, emit
from src.utils.search import search
from src.utils.types import embedding_call_purpose
logger = logging.getLogger(__name__)
@ -35,6 +42,7 @@ async def get_peers(
options: schemas.PeerGet | None = Body(
None, description="Filtering options for the peers list"
),
reverse: bool = Query(False, description="Whether to reverse the order of results"),
db: AsyncSession = db,
):
"""Get all Peers for a Workspace, paginated with optional filters."""
@ -46,7 +54,11 @@ async def get_peers(
return await apaginate(
db,
await crud.get_peers(workspace_name=workspace_id, filters=filter_param),
await crud.get_peers(
workspace_name=workspace_id,
filters=filter_param,
reverse=reverse,
),
)
@ -121,6 +133,7 @@ async def get_sessions_for_peer(
options: schemas.SessionGet | None = Body(
None, description="Filtering options for the sessions list"
),
reverse: bool = Query(False, description="Whether to reverse the order of results"),
db: AsyncSession = db,
):
"""Get all Sessions for a Peer, paginated with optional filters."""
@ -137,6 +150,7 @@ async def get_sessions_for_peer(
workspace_name=workspace_id,
peer_name=peer_id,
filters=filter_param,
reverse=reverse,
),
)
@ -252,6 +266,18 @@ async def get_representation(
If no target is provided, we get the omniscient Honcho Representation of the Peer.
"""
try:
embedding: list[float] | None = None
if options.search_query:
with (
suppress(Exception),
embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MEMORY.value,
workspace_name=workspace_id,
parent_category="api",
),
):
embedding = await embedding_client.embed(options.search_query)
# If no target specified, get global representation (omniscient Honcho perspective)
representation = await crud.get_working_representation(
workspace_id,
@ -259,6 +285,7 @@ async def get_representation(
observed=options.target if options.target is not None else peer_id,
session_name=options.session_id,
include_semantic_query=options.search_query,
embedding=embedding,
semantic_search_top_k=options.search_top_k,
semantic_search_max_distance=options.search_max_distance,
include_most_derived=options.include_most_frequent
@ -267,6 +294,7 @@ async def get_representation(
max_observations=options.max_conclusions
if options.max_conclusions is not None
else settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS,
parent_category="api",
)
return schemas.RepresentationResponse(
representation=representation.format_as_markdown()
@ -399,8 +427,21 @@ async def get_peer_context(
"""
# If no target specified, get the peer's own context (self-observation)
observed = target if target is not None else peer_id
context_started = perf_counter()
try:
embedding: list[float] | None = None
if search_query:
with (
suppress(Exception),
embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MEMORY.value,
workspace_name=workspace_id,
parent_category="api",
),
):
embedding = await embedding_client.embed(search_query)
# Get the working representation
representation = await crud.get_working_representation(
workspace_id,
@ -408,12 +449,14 @@ async def get_peer_context(
observed=observed,
session_name=None, # Peer context is global, not session-scoped
include_semantic_query=search_query,
embedding=embedding,
semantic_search_top_k=search_top_k,
semantic_search_max_distance=search_max_distance,
include_most_derived=include_most_frequent,
max_observations=max_conclusions
if max_conclusions is not None
else settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS,
parent_category="api",
)
# Get the peer card
@ -421,12 +464,29 @@ async def get_peer_context(
db, workspace_id, observer=peer_id, observed=observed
)
return schemas.PeerContext(
response = schemas.PeerContext(
peer_id=peer_id,
target_id=observed,
representation=representation.format_as_markdown(),
peer_card=peer_card,
)
emit(
GetContextEvent(
workspace_name=workspace_id,
context_scope="peer",
peer_name=peer_id,
target_name=observed,
has_representation=bool(response.representation),
has_peer_card=peer_card is not None,
search_query_provided=search_query is not None,
search_top_k=search_top_k,
search_max_distance=search_max_distance,
include_most_frequent=include_most_frequent,
max_conclusions=max_conclusions,
total_duration_ms=(perf_counter() - context_started) * 1000,
)
)
return response
except ValueError as e:
logger.warning(f"Failed to get context for peer {peer_id}: {str(e)}")
raise ResourceNotFoundException("Peer not found") from e

72
src/routers/sessions.py
View File

@ -1,5 +1,8 @@
"""FastAPI routes for session resources and session-scoped operations."""
import logging
from contextlib import suppress
from time import perf_counter
from fastapi import APIRouter, Body, Depends, Path, Query, Response
from fastapi_pagination import Page
@ -18,10 +21,12 @@ from src.exceptions import (
ValidationException,
)
from src.security import JWTParams, require_auth
from src.telemetry.events import EmbeddingCallPurpose, GetContextEvent, emit
from src.utils import summarizer
from src.utils.representation import Representation
from src.utils.search import search
from src.utils.tokens import estimate_tokens
from src.utils.types import embedding_call_purpose
logger = logging.getLogger(__name__)
@ -78,6 +83,8 @@ async def _get_working_representation_task(
max_observations=max_observations
if max_observations is not None
else config.settings.DERIVER.WORKING_REPRESENTATION_MAX_OBSERVATIONS,
parent_category="api",
embedding_purpose=EmbeddingCallPurpose.SESSION_CONTEXT_SEARCH,
)
@ -243,6 +250,7 @@ async def get_sessions(
options: schemas.SessionGet | None = Body(
None, description="Filtering and pagination options for the sessions list"
),
reverse: bool = Query(False, description="Whether to reverse the order of results"),
db: AsyncSession = db,
):
"""Get all Sessions for a Workspace, paginated with optional filters."""
@ -254,7 +262,12 @@ async def get_sessions(
filter_param = None
return await apaginate(
db, await crud.get_sessions(workspace_name=workspace_id, filters=filter_param)
db,
await crud.get_sessions(
workspace_name=workspace_id,
filters=filter_param,
reverse=reverse,
),
)
@ -669,6 +682,7 @@ async def get_session_context(
token_limit = (
tokens if tokens is not None else config.settings.GET_CONTEXT_MAX_TOKENS
)
context_started = perf_counter()
if peer_perspective and not peer_target:
raise ValidationException(
@ -680,11 +694,30 @@ async def get_session_context(
summary, messages = await _get_session_context_task(
db, workspace_id, session_id, token_limit, include_summary
)
return schemas.SessionContext(
response = schemas.SessionContext(
name=session_id,
messages=messages,
summary=summary,
)
emit(
GetContextEvent(
workspace_name=workspace_id,
context_scope="session",
session_name=session_id,
tokens_requested=tokens,
message_count=len(messages),
has_summary=summary is not None,
search_query_provided=search_query is not None,
search_top_k=search_top_k,
search_max_distance=search_max_distance,
include_most_frequent=include_most_frequent,
max_conclusions=max_conclusions,
include_summary=include_summary,
limit_to_session=limit_to_session,
total_duration_ms=(perf_counter() - context_started) * 1000,
)
)
return response
observer = peer_perspective or peer_target
observed = peer_target
@ -692,7 +725,14 @@ async def get_session_context(
# Pre-compute embedding outside the DB session (best-effort)
embedding: list[float] | None = None
if search_query:
with suppress(Exception):
with (
suppress(Exception),
embedding_call_purpose(
EmbeddingCallPurpose.SESSION_CONTEXT_SEARCH.value,
workspace_name=workspace_id,
parent_category="api",
),
):
embedding = await embedding_client.embed(search_query)
# Sequential calls on shared DB session
@ -731,13 +771,37 @@ async def get_session_context(
db, workspace_id, session_id, messages_start_id, messages_budget
)
return schemas.SessionContext(
response = schemas.SessionContext(
name=session_id,
messages=messages,
summary=summary,
peer_representation=representation.format_as_markdown(),
peer_card=card,
)
emit(
GetContextEvent(
workspace_name=workspace_id,
context_scope="session",
session_name=session_id,
peer_name=observer,
target_name=observed,
tokens_requested=tokens,
message_count=len(messages),
has_summary=summary is not None,
has_representation=bool(response.peer_representation),
has_peer_card=card is not None,
search_query_provided=search_query is not None,
search_top_k=search_top_k,
search_max_distance=search_max_distance,
include_most_frequent=include_most_frequent,
max_conclusions=max_conclusions,
include_summary=include_summary,
limit_to_session=limit_to_session,
peer_perspective_provided=peer_perspective is not None,
total_duration_ms=(perf_counter() - context_started) * 1000,
)
)
return response
@router.get(

12
src/routers/workspaces.py
View File

@ -1,3 +1,5 @@
"""FastAPI routes for workspace resources and workspace-scoped operations."""
import logging
from fastapi import APIRouter, Body, Depends, HTTPException, Path, Query, Response
@ -64,6 +66,7 @@ async def get_all_workspaces(
options: schemas.WorkspaceGet | None = Body(
None, description="Filtering and pagination options for the workspaces list"
),
reverse: bool = Query(False, description="Whether to reverse the order of results"),
db: AsyncSession = db,
):
"""Get all Workspaces, paginated with optional filters."""
@ -75,7 +78,7 @@ async def get_all_workspaces(
return await apaginate(
db,
await crud.get_all_workspaces(filters=filter_param),
await crud.get_all_workspaces(filters=filter_param, reverse=reverse),
)
@ -228,6 +231,13 @@ async def schedule_dream(
observed=observed,
dream_type=dream_type,
session_name=request.session_id,
# Manual route — explicit sentinels for the DreamRunEvent
# scheduling-context fields. Auto-schedule threads concrete
# threshold/delay reasons (see src/dreamer/dream_scheduler.py);
# without these, manual dreams arrive with both null and break
# analytics joins on `trigger_reason`.
trigger_reason="manual",
delay_reason="immediate",
)
logger.info(

212
src/telemetry/emitter.py
View File

@ -13,18 +13,64 @@ import contextlib
import json
import logging
from collections import deque
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Any
import httpx
from cloudevents.conversion import to_json # pyright: ignore[reportUnknownVariableType]
from cloudevents.http import CloudEvent
from src._version import HONCHO_VERSION
if TYPE_CHECKING:
from src.telemetry.events.base import BaseEvent
logger = logging.getLogger(__name__)
def _should_sample(
event: "BaseEvent", rate: object, *, event_id: str | None = None
) -> bool:
"""Trace-coherent deterministic sampler for high-volume events.
`rate` is typed as `object` (rather than `float`) because the caller
reads it straight from `settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE`,
which in tests gets MagicMock'd. A MagicMock comparison against 1.0
raises TypeError, so we validate at the boundary and fall back to
passthrough on anything non-numeric.
When the event carries a `run_id`, sampling decisions hash on that id —
so every event in an agent run either passes or fails the sampler, and
join queries downstream don't see half-traces. Events without `run_id`
(summarizer, deriver — non-agentic call sites) sample independently per
event using the event's deterministic id. Callers that have already
computed `event.generate_id()` can pass it as `event_id` to avoid the
redundant sha256 hash.
"""
if not isinstance(rate, int | float):
return True
rate_f = float(rate)
if rate_f >= 1.0:
return True
if rate_f <= 0.0:
return False
run_id = getattr(event, "run_id", None)
if isinstance(run_id, str) and run_id:
key = run_id
elif event_id is not None:
key = event_id
else:
key = event.generate_id()
# Stable hash → 0..9999 → compare against rate * 10000.
bucket = int.from_bytes(_stable_hash(key)[:4], "big") % 10_000
return bucket < int(rate_f * 10_000)
def _stable_hash(value: str) -> bytes:
import hashlib
return hashlib.sha256(value.encode("utf-8")).digest()
class TelemetryEmitter:
"""Buffered, async CloudEvents emitter with retry logic.
@ -59,6 +105,8 @@ class TelemetryEmitter:
_client: httpx.AsyncClient | None
_running: bool
_lock: asyncio.Lock
_capacity_warning_active: bool
_pending_flush_tasks: set[asyncio.Task[None]]
def __init__(
self,
@ -97,6 +145,8 @@ class TelemetryEmitter:
self._client = None
self._running = False
self._lock = asyncio.Lock()
self._capacity_warning_active = False
self._pending_flush_tasks = set()
async def start(self) -> None:
"""Start the emitter background tasks.
@ -121,8 +171,14 @@ class TelemetryEmitter:
async def shutdown(self) -> None:
"""Gracefully shutdown the emitter.
Stops the periodic flush task, flushes remaining events,
and closes the HTTP client.
Stops the periodic flush task, drains any in-flight threshold
flushes, flushes remaining events, and closes the HTTP client.
Threshold flushes are spawned from emit() and pop their batch
under lock before releasing it for the HTTP send. If we don't
await those tasks first, the final flush() can see an empty
buffer and return while the in-flight task is still mid-send —
closing the HTTP client then orphans that batch.
"""
if not self.enabled:
return
@ -135,6 +191,12 @@ class TelemetryEmitter:
with contextlib.suppress(asyncio.CancelledError):
await self._flush_task
# Drain in-flight threshold flushes. Snapshot the set first because
# the done-callback mutates it. Exceptions here mustn't block shutdown.
pending = list(self._pending_flush_tasks)
if pending:
await asyncio.gather(*pending, return_exceptions=True)
# Final flush of remaining events
await self.flush()
@ -159,9 +221,45 @@ class TelemetryEmitter:
return
from src.config import settings
from src.telemetry.prometheus.metrics import prometheus_metrics
# Generate deterministic event ID
event_id = event.generate_id()
# High-volume events are subject to HIGH_VOLUME_SAMPLE_RATE. Aggregate
# envelopes (representation.completed, dialectic.completed, dream.run,
# etc.) declare _volume_class="ground_truth" and skip the sampler.
# Sampling is deterministic on run_id when available so an entire
# agentic trace is either fully kept or fully dropped — never a
# half-sampled run that breaks join queries downstream.
#
# Trade-off: at rate < 1.0, ground_truth aggregates still emit but
# their high-volume children get sampled out. Downstream JOIN ... ON
# run_id sees orphaned parents; aggregates carry totals so this is
# intentional, but per-call analytics rebuilt from the sampled
# children alone will undercount. See HIGH_VOLUME_SAMPLE_RATE
# docstring in src/config.py for the full implications.
# Lazy event_id generation. The sampler only needs it for high-volume
# events without a run_id (run_id events sample on run_id directly).
# For sampled-out children of an agent run, deferring saves a sha256
# hash per event; for events that pass the sampler or skip it, we
# still only compute the id once and reuse it for the CloudEvent.
event_id: str | None = None
if event.volume_class() == "high_volume":
run_id = getattr(event, "run_id", None)
has_run_id = isinstance(run_id, str) and bool(run_id)
if not has_run_id:
event_id = event.generate_id()
if not _should_sample(
event,
settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE,
event_id=event_id,
):
prometheus_metrics.record_telemetry_event_sampled_out(
event_type=event.event_type()
)
return
if event_id is None:
event_id = event.generate_id()
# Build source with namespace for tenant routing
# Format: /honcho/{namespace}/{category} or /honcho/{category}
@ -179,28 +277,64 @@ class TelemetryEmitter:
"dataschema": f"https://honcho.dev/schemas/{event.event_type()}/v{event.schema_version()}",
}
# Build body and inject envelope-level identity. We do NOT mutate the event
# instance — tests and callers that observe the event after emit() see it
# unchanged. Only the serialized body that hits the wire carries the extras.
body: dict[str, Any] = event.model_dump(mode="json")
body["honcho_version"] = HONCHO_VERSION
# Buffer-full check happens here because deque(maxlen=) silently evicts.
# Detect by length-before-append; if at capacity, the append will displace
# the oldest event — that's a drop.
will_drop_oldest = len(self._buffer) >= self.max_buffer_size
# Create CloudEvent
cloud_event = CloudEvent(attributes, event.model_dump(mode="json"))
cloud_event = CloudEvent(attributes, body)
if will_drop_oldest:
prometheus_metrics.record_telemetry_event_dropped(reason="buffer_full")
self._buffer.append(cloud_event)
buffer_size = len(self._buffer)
prometheus_metrics.record_telemetry_event_emitted(event_type=event.event_type())
prometheus_metrics.set_telemetry_buffer_size(size=buffer_size)
logger.debug("Queued event %s (buffer size: %d)", event_id, buffer_size)
# Warning logs as buffer approaches max capacity
# Warning logs as buffer approaches max capacity. Edge-triggered so
# sustained backpressure doesn't spam thousands of WARN lines per
# second — exactly when log pipelines are most fragile.
capacity_ratio = buffer_size / self.max_buffer_size
if capacity_ratio >= 0.8:
logger.warning(
"Telemetry buffer at %.0f%% capacity (%d/%d events)",
capacity_ratio * 100,
buffer_size,
self.max_buffer_size,
)
if not self._capacity_warning_active:
self._capacity_warning_active = True
logger.warning(
"Telemetry buffer at %.0f%% capacity (%d/%d events)",
capacity_ratio * 100,
buffer_size,
self.max_buffer_size,
)
else:
self._capacity_warning_active = False
logger.debug("Event added to emitter (buffer size: %d)", buffer_size)
# Threshold-based flush trigger
# Threshold-based flush trigger. emit() is sync — guard against the
# case where a future caller invokes it from outside an event loop;
# the periodic flush task will still pick up the buffered events.
# Track spawned tasks so shutdown() can await them before closing the
# HTTP client — otherwise an in-flight threshold flush (which pops its
# batch under lock, then sends without the lock) can be orphaned and
# lose its batch when the client closes underneath it.
if buffer_size >= self.flush_threshold and self._running:
logger.debug("Triggering flush (buffer size: %d)", buffer_size)
asyncio.create_task(self.flush())
try:
asyncio.get_running_loop()
flush_task = asyncio.create_task(self.flush())
self._pending_flush_tasks.add(flush_task)
flush_task.add_done_callback(self._pending_flush_tasks.discard)
except RuntimeError:
logger.debug(
"emit() called outside an event loop; deferring flush to periodic task"
)
async def flush(self) -> None:
"""Flush buffered events to the endpoint.
@ -208,31 +342,47 @@ class TelemetryEmitter:
Sends events in batches up to batch_size. Uses exponential
backoff retry on failure. Events are returned to the buffer
on permanent failure.
The lock is held only for buffer mutations (pop batch / restore on
failure) — never across the HTTP send. A failing endpoint can spend
tens of seconds in retry + backoff; keeping that out of the lock
lets concurrent flushers make progress on disjoint batches.
"""
if not self.enabled or not self._buffer or self._client is None:
return
async with self._lock:
while self._buffer:
# Extract a batch
while True:
async with self._lock:
if not self._buffer:
return
batch: list[CloudEvent] = []
while self._buffer and len(batch) < self.batch_size:
batch.append(self._buffer.popleft())
if not batch:
break
if not batch:
return
# Try to send the batch
success = await self._send_batch(batch)
if not success:
# Put events back at the front of the buffer
for event in reversed(batch):
self._buffer.appendleft(event)
logger.warning(
"Failed to send batch of %d events, returned to buffer",
len(batch),
)
break
success = await self._send_batch(batch)
if success:
continue
from src.telemetry.prometheus.metrics import prometheus_metrics
async with self._lock:
# Put events back at the front of the buffer. If the buffer is
# already full, deque.appendleft silently evicts from the right
# — those events are lost. Count the eviction as send_failed.
for event in reversed(batch):
if len(self._buffer) >= self.max_buffer_size:
prometheus_metrics.record_telemetry_event_dropped(
reason="send_failed"
)
self._buffer.appendleft(event)
logger.warning(
"Failed to send batch of %d events, returned to buffer",
len(batch),
)
return
async def _send_batch(self, batch: list[CloudEvent]) -> bool:
"""Send a batch of events to the endpoint with retry logic.

27
src/telemetry/events/__init__.py
View File

@ -24,6 +24,11 @@ Event Categories:
- AgentToolPeerCardUpdatedEvent: Peer card updated by agent
- AgentToolSummaryCreatedEvent: Summary created
api: User-facing API operations
- MessageCreatedEvent: Message batch created
- FileUploadedEvent: File converted into messages
- GetContextEvent: Context retrieved for a session or peer
deletion: Resource removal
- DeletionCompletedEvent: Resource deletion completed (with cascade counts)
@ -55,11 +60,17 @@ import logging
from src.telemetry.events.agent import (
AgentIterationEvent,
AgentToolCallCompletedEvent,
AgentToolConclusionsCreatedEvent,
AgentToolConclusionsDeletedEvent,
AgentToolPeerCardUpdatedEvent,
AgentToolSummaryCreatedEvent,
)
from src.telemetry.events.api import (
FileUploadedEvent,
GetContextEvent,
MessageCreatedEvent,
)
from src.telemetry.events.base import BaseEvent, generate_event_id
from src.telemetry.events.deletion import DeletionCompletedEvent
from src.telemetry.events.dialectic import DialecticCompletedEvent
@ -67,6 +78,12 @@ from src.telemetry.events.dream import (
DreamRunEvent,
DreamSpecialistEvent,
)
from src.telemetry.events.llm import (
CallPurpose,
EmbeddingCallCompletedEvent,
EmbeddingCallPurpose,
LLMCallCompletedEvent,
)
from src.telemetry.events.reconciliation import (
CleanupStaleItemsCompletedEvent,
SyncVectorsCompletedEvent,
@ -89,10 +106,20 @@ __all__ = [
"DialecticCompletedEvent",
# Agent events
"AgentIterationEvent",
"AgentToolCallCompletedEvent",
"AgentToolConclusionsCreatedEvent",
"AgentToolConclusionsDeletedEvent",
"AgentToolPeerCardUpdatedEvent",
"AgentToolSummaryCreatedEvent",
# API events
"MessageCreatedEvent",
"FileUploadedEvent",
"GetContextEvent",
# LLM events
"LLMCallCompletedEvent",
"CallPurpose",
"EmbeddingCallCompletedEvent",
"EmbeddingCallPurpose",
# Reconciliation events
"SyncVectorsCompletedEvent",
"CleanupStaleItemsCompletedEvent",

146
src/telemetry/events/agent.py
View File

@ -29,11 +29,12 @@ class AgentIterationEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "agent.iteration"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "agent"
_volume_class: ClassVar[str] = "high_volume"
# Run identification
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
# Context
parent_category: str = Field(
@ -82,11 +83,11 @@ class AgentToolConclusionsCreatedEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "agent.tool.conclusions.created"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "agent"
# Run identification
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
iteration: int = Field(..., description="Iteration number when this occurred")
# Context
@ -121,11 +122,11 @@ class AgentToolConclusionsDeletedEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "agent.tool.conclusions.deleted"
_schema_version: ClassVar[int] = 2
_schema_version: ClassVar[int] = 3
_category: ClassVar[str] = "agent"
# Run identification
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
iteration: int = Field(..., description="Iteration number when this occurred")
# Context
@ -158,11 +159,11 @@ class AgentToolPeerCardUpdatedEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "agent.tool.peer_card.updated"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "agent"
# Run identification
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
iteration: int = Field(..., description="Iteration number when this occurred")
# Context
@ -190,11 +191,11 @@ class AgentToolSummaryCreatedEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "agent.tool.summary.created"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "agent"
# Run identification (may be placeholder if not from an agentic loop)
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
iteration: int = Field(..., description="Iteration number when this occurred")
# Context
@ -219,17 +220,140 @@ class AgentToolSummaryCreatedEvent(BaseEvent):
# Token usage
input_tokens: int = Field(
..., description="Input tokens used for summary generation"
...,
description=(
"Provider-side input tokens for the summary LLM call "
"(equivalent to HonchoLLMCallResponse.input_tokens). "
"keeps this field unchanged — adding a duplicate "
"`provider_input_tokens` would only churn analytics queries."
),
)
output_tokens: int = Field(..., description="Output tokens (summary token count)")
# ---- Additive fields ----
# Breakdown of what *went into* the summary prompt. Lets calibration
# answer "how much of a summary call's cost was the previous-summary
# rollup vs. the new messages vs. the scaffold/instructions" without
# re-deriving from the message corpus.
previous_summary_tokens: int = Field(
default=0,
description=(
"Token count of the previous summary text fed back in as context. "
"0 when this is the first summary for the session."
),
)
message_tokens: int = Field(
default=0,
description=(
"Sum of `Message.token_count` across the messages being "
"summarized (excludes scaffold and previous_summary)."
),
)
prompt_scaffold_tokens: int = Field(
default=0,
description=(
"Estimated tokens for the static scaffold portion of the prompt "
"(from estimate_short/long_summary_prompt_tokens)."
),
)
def get_resource_id(self) -> str:
"""Resource ID includes run_id and iteration for uniqueness."""
return f"{self.run_id}:{self.iteration}:summary_created"
class AgentToolCallCompletedEvent(BaseEvent):
"""generic tool-call event: fires once per tool invocation.
Complements the four state-changer events (conclusions_created/deleted,
peer_card_updated, summary_created), which carry semantic information
about specific tools, with a lightweight per-call telemetry record that
covers every tool — including read-only tools (`search_memory`,
`get_recent_history`, etc.) that have no dedicated event today.
Resource id includes `tool_call_seq` so the model can legitimately call
the same tool twice in one iteration (it does) without colliding event
ids — without seq, both calls would deterministically hash to the same
id and dedupe would drop one.
"""
_event_type: ClassVar[str] = "agent.tool.call.completed"
_schema_version: ClassVar[int] = 1
_category: ClassVar[str] = "agent"
_volume_class: ClassVar[str] = "high_volume"
# Run identification
run_id: str = Field(..., description="Run id for correlation")
iteration: int = Field(..., description="Iteration number (1-indexed)")
tool_call_seq: int = Field(
...,
description="0-indexed position within the iteration's tool batch. Disambiguates two calls to the same tool in one iteration.",
)
provider_tool_call_id: str | None = Field(
default=None,
description="Provider-supplied tool call id (e.g. Anthropic's toolu_*) when available; lets analytics cross-reference provider logs",
)
# Context
parent_category: str = Field(
..., description="Parent category: 'dream' or 'dialectic'"
)
agent_type: str = Field(
..., description="Agent type: 'deduction', 'induction', or 'dialectic'"
)
workspace_name: str = Field(..., description="Workspace name")
# What ran
tool_name: str = Field(..., description="Tool name as invoked")
duration_ms: float = Field(..., description="Wall-clock duration of the handler")
is_error: bool = Field(default=False, description="True if the handler raised")
# Result shape
result_chars: int = Field(
..., description="Length of the result string returned to the LLM"
)
result_chars_before_truncation: int | None = Field(
default=None,
description="Original result size when the handler truncated; None when no truncation occurred. Pair with was_truncated for the delta.",
)
result_tokens_estimate: int = Field(
default=0,
description="tiktoken-based size proxy for the result string; estimate only",
)
was_truncated: bool = Field(
default=False,
description="True when the handler clamped the result to fit a size budget",
)
# Search-specific fields (None for non-search tools). Populated by search
# handlers via the ToolResult.metadata bridge.
query_tokens: int | None = Field(
default=None, description="tiktoken estimate of the search query text"
)
top_k: int | None = Field(
default=None, description="Caller-supplied top_k for the search"
)
results_count: int | None = Field(
default=None, description="Number of results returned by the search"
)
used_embedding: bool | None = Field(
default=None,
description="True when the search ran a vector lookup (vs. metadata-only filter)",
)
embedding_query_count: int = Field(
default=0,
description="Number of embedding API calls the handler made for this invocation",
)
def get_resource_id(self) -> str:
"""{run_id}:{iteration}:{tool_call_seq} so duplicate tool calls within
one iteration produce distinct deterministic ids."""
return f"{self.run_id}:{self.iteration}:{self.tool_call_seq}"
__all__ = [
"AgentIterationEvent",
"AgentToolCallCompletedEvent",
"AgentToolConclusionsCreatedEvent",
"AgentToolConclusionsDeletedEvent",
"AgentToolPeerCardUpdatedEvent",

167
src/telemetry/events/api.py Normal file
View File

@ -0,0 +1,167 @@
"""
API events for Honcho telemetry.
These events track user-facing API operations
"""
from typing import ClassVar, Literal
from pydantic import Field
from src.telemetry.events.base import BaseEvent
class MessageCreatedEvent(BaseEvent):
"""Emitted when one or more messages are created.
This is the canonical API event for counting created messages, including
messages created from file uploads. The resource_id keys on
`last_message_id` (the trailing message's public nanoid) so two batches
of the same size in the same session+source produce distinct event ids.
"""
_event_type: ClassVar[str] = "message.created"
_schema_version: ClassVar[int] = 1
_category: ClassVar[str] = "api"
workspace_name: str = Field(..., description="Workspace name")
session_name: str = Field(..., description="Session name")
message_count: int = Field(..., description="Number of messages created")
total_tokens: int = Field(..., description="Total tokens across created messages")
source: Literal["api", "file_upload"] = Field(
default="api", description="Source of the created messages"
)
last_message_id: str = Field(
...,
description="public_id (nanoid) of the trailing message in the batch — used as the stable unique key for this emission",
)
def get_resource_id(self) -> str:
"""Resource ID keys on the trailing message's public_id so two batches
of the same size in the same session+source produce distinct event ids.
"""
return (
f"{self.workspace_name}:{self.session_name}:"
f"{self.source}:{self.last_message_id}"
)
class FileUploadedEvent(BaseEvent):
"""Emitted when an uploaded file is converted into messages.
This captures file-side metadata. Message creation counts should use
MessageCreatedEvent to avoid double-counting file uploads.
"""
_event_type: ClassVar[str] = "file.uploaded"
_schema_version: ClassVar[int] = 1
_category: ClassVar[str] = "api"
workspace_name: str = Field(..., description="Workspace name")
session_name: str = Field(..., description="Session name")
peer_name: str = Field(..., description="Peer that uploaded the file")
file_id: str = Field(..., description="Generated file identifier")
filename: str | None = Field(default=None, description="Uploaded filename")
content_type: str | None = Field(default=None, description="Uploaded content type")
file_size_bytes: int | None = Field(
default=None, description="Uploaded file size in bytes"
)
message_count: int = Field(
..., description="Number of messages created from the file"
)
total_tokens: int = Field(
..., description="Total tokens across messages created from the file"
)
def get_resource_id(self) -> str:
"""Resource ID includes workspace, session, and generated file ID."""
return f"{self.workspace_name}:{self.session_name}:{self.file_id}"
class GetContextEvent(BaseEvent):
"""Emitted when context is retrieved for a session or peer."""
_event_type: ClassVar[str] = "context.retrieved"
_schema_version: ClassVar[int] = 1
_category: ClassVar[str] = "api"
workspace_name: str = Field(..., description="Workspace name")
context_scope: Literal["session", "peer"] = Field(
..., description="Context endpoint scope"
)
session_name: str | None = Field(
default=None, description="Session name for session-scoped context"
)
peer_name: str | None = Field(default=None, description="Observer peer name")
target_name: str | None = Field(default=None, description="Observed peer name")
tokens_requested: int | None = Field(
default=None,
description="Caller-supplied tokens query parameter (None = endpoint default applied)",
)
message_count: int = Field(
default=0, description="Number of messages returned in context"
)
has_summary: bool = Field(
default=False, description="Whether a summary was returned"
)
has_representation: bool = Field(
default=False, description="Whether a representation was returned"
)
has_peer_card: bool = Field(
default=False, description="Whether a peer card was returned"
)
search_query_provided: bool = Field(
default=False, description="Whether semantic search query text was provided"
)
search_top_k: int | None = Field(
default=None,
description="Caller-supplied search_top_k (None = endpoint default)",
)
search_max_distance: float | None = Field(
default=None,
description="Caller-supplied search_max_distance (None = endpoint default)",
)
include_most_frequent: bool | None = Field(
default=None,
description="Caller-supplied include_most_frequent (None = endpoint default; defaults differ between peer and session endpoints)",
)
max_conclusions: int | None = Field(
default=None,
description="Caller-supplied max_conclusions (None = endpoint default)",
)
include_summary: bool | None = Field(
default=None,
description="Whether summary inclusion was requested; None when unsupported by endpoint",
)
limit_to_session: bool = Field(
default=False,
description="Whether representation retrieval was session-limited",
)
peer_perspective_provided: bool = Field(
default=False,
description="Whether peer_perspective was supplied for session context",
)
total_duration_ms: float = Field(..., description="Total processing time")
def get_resource_id(self) -> str:
"""Resource ID identifies the requested context scope.
Uses empty string (illegal in peer names — nanoid-derived) as the
absent-peer sentinel so that a peer literally named "none" does not
collide with the absent-peer case.
"""
peer_name = self.peer_name if self.peer_name is not None else ""
target_name = self.target_name if self.target_name is not None else ""
if self.context_scope == "session":
return (
f"{self.workspace_name}:session:{self.session_name}:"
f"{peer_name}:{target_name}"
)
return f"{self.workspace_name}:peer:{peer_name}:{target_name}"
__all__ = [
"FileUploadedEvent",
"GetContextEvent",
"MessageCreatedEvent",
]

32
src/telemetry/events/base.py
View File

@ -18,21 +18,28 @@ def generate_event_id(
event_type: str,
timestamp: datetime,
resource_id: str,
honcho_version: str | None = None,
) -> str:
"""Generate a deterministic event ID for idempotency.
Same inputs always produce the same ID, so retries are automatically
deduplicated on the receiving end.
deduplicated on the receiving end. `honcho_version` is folded into the
payload so two deploys emitting the same logical event produce distinct
IDs — protects against silently merging events whose payload shape may
have shifted between versions.
Args:
event_type: The CloudEvents type (e.g., "honcho.work.representation.completed")
timestamp: When the event occurred
resource_id: A unique identifier for the resource (can include workspace_id if relevant)
honcho_version: The honcho package version emitting the event;
included in the hash so cross-deploy events don't dedupe.
Returns:
A deterministic event ID in the format "evt_{base64_hash}"
"""
payload = f"{event_type}:{resource_id}:{timestamp.isoformat()}"
version_segment = honcho_version or ""
payload = f"{event_type}:{resource_id}:{timestamp.isoformat()}:{version_segment}"
hash_bytes = hashlib.sha256(payload.encode()).digest()[:16]
# Use URL-safe base64 encoding, strip padding
encoded = base64.urlsafe_b64encode(hash_bytes).decode().rstrip("=")
@ -57,6 +64,12 @@ class BaseEvent(BaseModel):
_schema_version: ClassVar[int]
_category: ClassVar[str] # "work", "activity", or "resource"
# Volume class for sampling decisions:
# - "ground_truth": always emitted at rate 1.0 (aggregates, calibration keys)
# - "high_volume": subject to TELEMETRY.HIGH_VOLUME_SAMPLE_RATE
# Default is ground_truth so existing events keep firing unconditionally.
_volume_class: ClassVar[str] = "ground_truth"
# Common timestamp field present in all events
timestamp: datetime = Field(
default_factory=lambda: datetime.now(UTC),
@ -78,6 +91,11 @@ class BaseEvent(BaseModel):
"""Return the event category (work, activity, or resource)."""
return cls._category
@classmethod
def volume_class(cls) -> str:
"""Return the volume class for sampling: 'ground_truth' or 'high_volume'."""
return cls._volume_class
def get_resource_id(self) -> str:
"""Return the resource ID for idempotency key generation.
@ -89,9 +107,17 @@ class BaseEvent(BaseModel):
raise NotImplementedError("Subclasses must implement get_resource_id()")
def generate_id(self) -> str:
"""Generate a deterministic event ID for this event instance."""
"""Generate a deterministic event ID for this event instance.
Folds in the honcho package version so the same logical event from
two different deploys produces distinct ids — downstream dedupe by
id won't silently merge events whose body shape may have shifted.
"""
from src._version import HONCHO_VERSION
return generate_event_id(
event_type=self.event_type(),
timestamp=self.timestamp,
resource_id=self.get_resource_id(),
honcho_version=HONCHO_VERSION,
)

14
src/telemetry/events/dialectic.py
View File

@ -25,11 +25,11 @@ class DialecticCompletedEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "dialectic.completed"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "dialectic"
# Run identification (for correlating with iteration/tool events)
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
# Workspace context
workspace_name: str = Field(..., description="Workspace name")
@ -67,6 +67,16 @@ class DialecticCompletedEvent(BaseEvent):
default=0, description="Tokens written to prompt cache"
)
# Cap hit flag
hit_input_token_cap: bool = Field(
default=False,
description=(
"True when an iteration's input exceeded "
"settings.DIALECTIC.MAX_INPUT_TOKENS. Token-based — fires for the "
"single-oversized-message case too, not just message-list shrinkage."
),
)
def get_resource_id(self) -> str:
"""Resource ID is the run_id for uniqueness."""
return self.run_id

91
src/telemetry/events/dream.py
View File

@ -22,11 +22,11 @@ class DreamRunEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "dream.run"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "dream"
# Run identification (for correlating with specialist/iteration/tool events)
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
# Workspace context
workspace_name: str = Field(..., description="Workspace name")
@ -69,6 +69,43 @@ class DreamRunEvent(BaseEvent):
)
total_duration_ms: float = Field(..., description="Total processing time")
# ---- Additive fields ----
dream_type: str | None = Field(
default=None,
description="DreamType slug (currently 'omni'; future: 'deductive'/'inductive')",
)
enabled_types_count: int = Field(
default=0,
description="len(settings.DREAM.ENABLED_TYPES) at run start — how many dream types this deploy was producing",
)
trigger_reason: str | None = Field(
default=None,
description=(
"What tripped the schedule: 'document_threshold' | 'manual' | 'surprisal'. "
"Captured at schedule time and threaded through the queue payload."
),
)
delay_reason: str | None = Field(
default=None,
description=(
"What governed when this dream actually fired: 'idle_timeout' | "
"'immediate' | 'min_hours_gate'. Disambiguates from trigger_reason "
"to preserve the two-gate scheduler semantics in analytics."
),
)
documents_since_last_dream_at_schedule: int | None = Field(
default=None,
description=(
"Document count at the moment check_and_schedule_dream made the decision. "
"Named _at_schedule because the live count changes between schedule and fire "
"(idle delay) — this is the snapshot, not the current value."
),
)
document_threshold: int | None = Field(
default=None,
description="settings.DREAM.DOCUMENT_THRESHOLD snapshot at schedule time",
)
def get_resource_id(self) -> str:
"""Resource ID is the run_id for uniqueness."""
return self.run_id
@ -82,11 +119,11 @@ class DreamSpecialistEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "dream.specialist"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "dream"
# Run identification (correlates with parent dream.run)
run_id: str = Field(..., description="8-char UUID prefix for run correlation")
run_id: str = Field(..., description="Nanoid for run correlation")
# Specialist info
specialist_type: str = Field(
@ -108,6 +145,52 @@ class DreamSpecialistEvent(BaseEvent):
duration_ms: float = Field(..., description="Processing time")
success: bool = Field(..., description="Whether the specialist succeeded")
# ---- Additive fields ----
# Denormalized rollups so analytics can answer "how many observations did
# this specialist actually produce" without re-aggregating per-tool events.
# Sourced from ToolResult.metadata via tool_loop's all_tool_calls, NOT
# from tool-name counting — `create_observations` calls can produce zero
# observations when all entries fail validation.
created_observation_count: int = Field(
default=0,
description="Actual observations created across all create_observations calls (from ToolResult.metadata.created_count)",
)
deleted_observation_count: int = Field(
default=0,
description="Actual observations deleted across all delete_observations calls (from ToolResult.metadata.deleted_count)",
)
created_counts_by_level: dict[str, int] = Field(
default_factory=dict,
description=(
"Counts of created observations per level (explicit / deductive / "
"inductive / contradiction), aggregated across all "
"create_observations tool calls in this specialist run. Levels "
"with zero count may be omitted; queries should treat missing "
"keys as 0. Dict-of-counts rather than list[str] because dream "
"specialists can produce 10-20+ observations per run — a flat "
"list becomes noisy at that scale."
),
)
deleted_counts_by_level: dict[str, int] = Field(
default_factory=dict,
description=(
"Counts of deleted observations per level, aggregated across all "
"delete_observations tool calls in this specialist run."
),
)
peer_card_updated: bool = Field(
default=False,
description="True when at least one update_peer_card tool call succeeded",
)
search_tool_calls_count: int = Field(
default=0,
description="Number of search_memory / search_messages / search_messages_temporal invocations",
)
error_class: str | None = Field(
default=None,
description="Exception class name when success=False; None on success.",
)
def get_resource_id(self) -> str:
"""Resource ID includes run_id and specialist type for uniqueness."""
return f"{self.run_id}:{self.specialist_type}"

264
src/telemetry/events/llm.py Normal file
View File

@ -0,0 +1,264 @@
"""LLM-call events for Honcho telemetry.
These events fire once per provider hit (each iteration of an agentic tool loop,
each deriver/summarizer LLM call, etc.) and carry the full cost-attribution
context: model/provider/transport, token counts with cache breakdown, finish
reason, outcome (success/error), retry/fallback state, and run correlation.
Unlike the existing aggregate `*Completed` events (representation, dialectic,
dream), this event is high-volume. It participates in the
`settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE` sampler so per-iteration emission
can be tuned against a budget.
"""
from __future__ import annotations
from enum import Enum
from typing import ClassVar, Literal
from pydantic import Field
from src.config import ModelTransport
from src.telemetry.events.base import BaseEvent
class CallPurpose(str, Enum):
"""Closed taxonomy for LLM call purposes.
The schema lint enforces that all `LLMCallCompletedEvent` emissions use a
value from this enum. Adding a new call site requires adding a value here
first — keeps the analytics taxonomy stable.
"""
DERIVER_REPRESENTATION = "deriver.representation"
DIALECTIC_ANSWER = "dialectic.answer"
DREAM_DEDUCTION = "dream.deduction"
DREAM_INDUCTION = "dream.induction"
SUMMARY_SHORT = "summary.short"
SUMMARY_LONG = "summary.long"
class LLMCallCompletedEvent(BaseEvent):
"""Emitted once per provider hit by `honcho_llm_call_inner`.
Covers success, failure, and cancellation via `outcome`. The last attempt
of a tenacity retry chain is flagged with `is_final_attempt=True` regardless
of outcome — calibration queries for "exhausted" use
`outcome='error' AND is_final_attempt`. Cancellations (typically client
disconnect mid-stream or server shutdown) are distinct from errors and
should not feed error-rate alerting.
Streaming note: when `was_stream=True`, the token counts are placeholders
(0) because token totals aren't knowable until the stream drains. Use the
aggregate envelopes (`DialecticCompletedEvent` etc.) for streamed-call
accuracy until streaming completion is wired through.
"""
_event_type: ClassVar[str] = "llm.call.completed"
_schema_version: ClassVar[int] = 1
_category: ClassVar[str] = "llm"
_volume_class: ClassVar[str] = "high_volume"
# Context (None for system calls without workspace context)
workspace_name: str | None = Field(default=None, description="Workspace name")
call_purpose: CallPurpose | None = Field(
default=None,
description="Closed enum identifying the call site (deriver, dialectic, etc.)",
)
parent_category: str | None = Field(
default=None,
description="Parent category for analytics joins: 'representation' | 'dialectic' | 'dream' | 'summary'",
)
# Provider info
transport: ModelTransport = Field(
..., description="SDK transport: 'anthropic' | 'openai' | 'gemini'"
)
provider_label: str | None = Field(
default=None,
description="Best-effort vendor inference for relay setups (e.g. 'anthropic' when an OpenRouter base_url + 'anthropic/claude-...' model is used); None when not reliably inferable",
)
model: str = Field(..., description="Model identifier as sent to the provider")
effective_max_output_tokens: int = Field(
..., description="max_tokens value used for this call"
)
# Token usage (zero on was_stream=True placeholder)
provider_input_tokens: int = Field(default=0, description="Provider input_tokens")
provider_output_tokens: int = Field(default=0, description="Provider output_tokens")
cache_read_tokens: int = Field(
default=0, description="Tokens read from prompt cache"
)
cache_creation_tokens: int = Field(
default=0, description="Tokens written to prompt cache"
)
# Outcome
finish_reason: str | None = Field(
default=None,
description="First finish reason from the response (None on error)",
)
outcome: Literal["success", "error", "cancelled"] = Field(
...,
description="'success' when the provider returned a result, 'error' when it raised, 'cancelled' when the awaitable was cancelled (client disconnect, server shutdown). Cancellations should be excluded from error-rate alerting.",
)
is_final_attempt: bool = Field(
...,
description="True when this is the last allowed attempt (attempt == retry_attempts). Combine with outcome='error' to identify retry-exhausted calls. Cancellations are not retried so this reflects the attempt at cancellation time.",
)
error_class: str | None = Field(
default=None,
description="Exception class name when outcome is 'error' or 'cancelled' (e.g. 'CancelledError')",
)
# Retry/fallback state
attempt: int = Field(..., description="1-indexed tenacity attempt number")
retry_attempts: int = Field(..., description="Total attempts allowed by caller")
was_fallback: bool = Field(
..., description="True when this attempt used the fallback ModelConfig"
)
# Timing
duration_ms: float = Field(
..., description="Wall-clock duration of the provider call"
)
# Shape
has_tools: bool = Field(default=False, description="True if tools were provided")
tool_call_count: int = Field(
default=0, description="Number of tool calls the model requested"
)
was_stream: bool = Field(
default=False,
description="True for the stream_final_response path. Token counts are 0 placeholders — see class docstring.",
)
# Agent correlation (None for non-agent calls like summarizer / deriver)
run_id: str | None = Field(
default=None,
description="Agent run id (ULID when widened in follow-up)",
)
iteration: int | None = Field(
default=None,
description="1-indexed iteration within an agentic tool loop. Passed explicitly via LLMTelemetryContext — NOT read from set_current_iteration (that fires after the LLM call)",
)
def get_resource_id(self) -> str:
"""Resource id includes run_id + iteration + attempt + transport/model
so multi-attempt retries within one iteration get distinct ids."""
run = self.run_id or "none"
iteration = self.iteration if self.iteration is not None else 0
return f"{run}:{iteration}:{self.attempt}:{self.transport}:{self.model}"
class EmbeddingCallPurpose(str, Enum):
"""Closed taxonomy for embedding call purposes.
Mirrors `CallPurpose` for LLM calls. Adding a new embedding call site
requires adding a value here first — keeps the analytics taxonomy stable
and prevents free-form `track_name` drift from leaking into queries.
"""
SEARCH_MEMORY = "search_memory"
SEARCH_MESSAGES = "search_messages"
CREATE_OBSERVATIONS = "create_observations"
VECTOR_SYNC = "vector_sync"
SUMMARY = "summary"
MESSAGE_CREATE = "message_create"
# Added so previously-unattributed call sites land on a distinct slug
# instead of None. Closed taxonomy — coordinate with analytics before
# adding more.
DIALECTIC_PREFETCH = "dialectic_prefetch"
SESSION_CONTEXT_SEARCH = "session_context_search"
PREFERENCE_EXTRACTION = "preference_extraction"
GENERIC_DOCUMENT_SEARCH = "generic_document_search"
class EmbeddingCallCompletedEvent(BaseEvent):
"""Emitted once per embedding-provider call.
Embedding calls are real provider spend (per-token like LLM calls).
Search tools, observation creation, the message-embedding sync, and
the deriver/summarizer paths all hit the embedding API; this event
captures cost-attribution context for all of them.
Volume note: this event is high-volume. Interactive paths
(`search_memory` / `search_messages`) emit one event per query, so under
a search-heavy dialectic load this can match or exceed the LLM call
rate. The shared `HIGH_VOLUME_SAMPLE_RATE` covers both.
"""
_event_type: ClassVar[str] = "embedding.call.completed"
_schema_version: ClassVar[int] = 1
_category: ClassVar[str] = "llm"
_volume_class: ClassVar[str] = "high_volume"
workspace_name: str | None = Field(default=None, description="Workspace name")
call_purpose: EmbeddingCallPurpose | None = Field(
default=None,
description=(
"Closed enum identifying the call site. Set by callers via the "
"`embedding_call_purpose` ContextVar; None when the call originated "
"outside an instrumented path."
),
)
parent_category: str | None = Field(
default=None,
description="Parent category for analytics joins (e.g. 'dialectic', 'representation')",
)
provider: str = Field(..., description="'openai' | 'gemini'")
model: str = Field(..., description="Model identifier")
input_count: int = Field(
..., description="Number of texts embedded in this call (batch size)"
)
input_tokens_estimate: int = Field(
default=0,
description=(
"tiktoken-based size proxy for the embedded text. ESTIMATE only — "
"the embedding client uses encoding_for_model() with a cl100k_base "
"fallback (see embedding_client.py:68-71), which is exact for "
"older OpenAI models, an approximation for newer ones, and a "
"rough proxy for Gemini (which has its own tokenizer)."
),
)
duration_ms: float = Field(
..., description="Wall-clock duration of the provider call"
)
outcome: Literal["success", "error", "cancelled"] = Field(
...,
description="'success' when the provider returned a result, 'error' when it raised, 'cancelled' when the awaitable was cancelled. Cancellations should be excluded from error-rate alerting.",
)
is_final_attempt: bool = Field(
default=False,
description=(
"True on the last retry attempt. Mirrors LLMCallCompletedEvent's "
"convention: combine with outcome='error' to identify exhausted "
"embedding calls. Cancellations are not retried."
),
)
error_class: str | None = Field(
default=None,
description="Exception class name when outcome is 'error' or 'cancelled'",
)
run_id: str | None = Field(
default=None,
description="Agent run id when called from an agentic loop; None for sync/CRUD paths",
)
def get_resource_id(self) -> str:
"""Resource id includes timestamp-derived components implicitly via
generate_id(); we just stake out a non-empty identifier scope."""
run = self.run_id or "none"
purpose = self.call_purpose.value if self.call_purpose else "unknown"
return f"{run}:{purpose}:{self.provider}:{self.model}:{self.input_count}"
__all__ = [
"CallPurpose",
"EmbeddingCallCompletedEvent",
"EmbeddingCallPurpose",
"LLMCallCompletedEvent",
]

74
src/telemetry/events/representation.py
View File

@ -22,7 +22,7 @@ class RepresentationCompletedEvent(BaseEvent):
"""
_event_type: ClassVar[str] = "representation.completed"
_schema_version: ClassVar[int] = 1
_schema_version: ClassVar[int] = 2
_category: ClassVar[str] = "representation"
# Workspace context
@ -57,9 +57,79 @@ class RepresentationCompletedEvent(BaseEvent):
total_duration_ms: float = Field(..., description="Total processing time")
# Token usage
input_tokens: int = Field(..., description="Input tokens used")
input_tokens: int = Field(
...,
description=(
"Queued-message tokens (the ones we're actually reasoning ABOUT). "
"This field is the downstream metering key for "
"representation.completed — DO NOT rename or repurpose without "
"coordinating with downstream consumers."
),
)
total_input_tokens: int = Field(
...,
description="Total tokens sent to the LLM (queued + extra context + scaffold)",
)
output_tokens: int = Field(..., description="Output tokens generated")
# ---- Additive fields ----
# Token breakdown beyond `input_tokens` (queued-message tokens already
# captured above). These break out what made up the LLM prompt so analytics
# can answer "how much did extra context cost us per call".
queued_message_count: int = Field(
default=0,
description="Number of messages in this batch that were the actual queue items being reasoned about",
)
prompt_message_count: int = Field(
default=0,
description="Total messages in the prompt — queued + extra interleaving context",
)
prompt_message_tokens: int = Field(
default=0,
description="Sum of token_count across all messages in the prompt",
)
extra_context_message_count: int = Field(
default=0,
description="prompt_message_count - queued_message_count: the extra-context messages we pulled in",
)
extra_context_tokens: int = Field(
default=0,
description="prompt_message_tokens - input_tokens: token cost of the extra context",
)
prompt_scaffold_tokens: int = Field(
default=0,
description="Estimated tokens for the system/scaffold portion of the prompt",
)
# Cap configuration + hit flags ()
batch_max_tokens: int = Field(
default=0,
description="settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS at fetch time",
)
max_input_tokens: int = Field(
default=0, description="settings.DERIVER.MAX_INPUT_TOKENS at call time"
)
was_flush_enabled: bool = Field(
default=False,
description="settings.DERIVER.FLUSH_ENABLED snapshot at batch time",
)
hit_batch_token_cap: bool = Field(
default=False,
description="True when the queue batcher clamped the batch to fit batch_max_tokens",
)
hit_input_token_cap: bool = Field(
default=False,
description=(
"True when the LLM call truncated input messages to fit max_input_tokens."
),
)
# Observer fanout
observer_count: int = Field(
default=0,
description="Number of observers this representation was saved against",
)
def get_resource_id(self) -> str:
"""Resource ID includes workspace, session, and latest message for uniqueness."""
return f"{self.workspace_name}:{self.session_name}:{self.latest_message_id}"

58
src/telemetry/prometheus/metrics.py
View File

@ -10,6 +10,7 @@ from prometheus_client import (
CONTENT_TYPE_LATEST,
REGISTRY,
Counter,
Gauge,
disable_created_metrics,
generate_latest,
)
@ -29,6 +30,12 @@ class NamespacedCounter(Counter):
return super().labels(**kwargs) # type: ignore[return-value]
class NamespacedGauge(Gauge):
def labels(self, **kwargs: str) -> NamespacedGauge:
kwargs["namespace"] = cast(str, settings.METRICS.NAMESPACE)
return super().labels(**kwargs) # type: ignore[return-value]
class TokenTypes(Enum):
INPUT = "input"
OUTPUT = "output"
@ -92,6 +99,32 @@ dreamer_tokens_processed_counter = NamespacedCounter(
["namespace", "specialist_name", "token_type"],
)
# CloudEvents emitter health metrics. Split intentional (sampled out) vs unintentional
# (dropped) so the dropped counter remains a real alert signal.
telemetry_events_emitted_counter = NamespacedCounter(
"telemetry_events_emitted",
"CloudEvents successfully placed on the emitter buffer",
["namespace", "type"],
)
telemetry_events_sampled_out_counter = NamespacedCounter(
"telemetry_events_sampled_out",
"CloudEvents intentionally dropped by HIGH_VOLUME_SAMPLE_RATE",
["namespace", "type"],
)
telemetry_events_dropped_counter = NamespacedCounter(
"telemetry_events_dropped",
"CloudEvents lost unintentionally (buffer_full or send_failed)",
["namespace", "reason"],
)
telemetry_buffer_size_gauge = NamespacedGauge(
"telemetry_buffer_size",
"Current size of the CloudEvents emitter buffer",
["namespace"],
)
@final
class PrometheusMetrics:
@ -217,6 +250,31 @@ class PrometheusMetrics:
except Exception as e:
self._handle_metric_error("record_dreamer_tokens", e)
def record_telemetry_event_emitted(self, *, event_type: str) -> None:
try:
telemetry_events_emitted_counter.labels(type=event_type).inc()
except Exception as e:
self._handle_metric_error("record_telemetry_event_emitted", e)
def record_telemetry_event_sampled_out(self, *, event_type: str) -> None:
try:
telemetry_events_sampled_out_counter.labels(type=event_type).inc()
except Exception as e:
self._handle_metric_error("record_telemetry_event_sampled_out", e)
def record_telemetry_event_dropped(self, *, reason: str) -> None:
# Reason is one of "buffer_full" | "send_failed".
try:
telemetry_events_dropped_counter.labels(reason=reason).inc()
except Exception as e:
self._handle_metric_error("record_telemetry_event_dropped", e)
def set_telemetry_buffer_size(self, *, size: int) -> None:
try:
telemetry_buffer_size_gauge.labels().set(size)
except Exception as e:
self._handle_metric_error("set_telemetry_buffer_size", e)
prometheus_metrics = PrometheusMetrics()

602
src/utils/agent_tools.py
View File

@ -20,18 +20,49 @@ from src.telemetry.events import (
AgentToolConclusionsCreatedEvent,
AgentToolConclusionsDeletedEvent,
AgentToolPeerCardUpdatedEvent,
EmbeddingCallPurpose,
emit,
)
from src.utils import summarizer
from src.utils.formatting import format_new_turn_with_timestamp, utc_now_iso
from src.utils.representation import Representation
from src.utils.types import get_current_iteration
from src.utils.types import ToolResult, embedding_call_purpose, get_current_iteration
logger = logging.getLogger(__name__)
# Hard cap to prevent unbounded peer card growth from repeated agent updates.
MAX_PEER_CARD_FACTS = 40
# Identity-marker prefixes allowed on the peer card. Anything else is rejected
# structurally — see `_validate_peer_card_entry`.
PEER_CARD_ALLOWED_PREFIXES: tuple[str, ...] = (
"IDENTITY:",
"ATTRIBUTE:",
"RELATIONSHIP:",
"INSTRUCTION:",
)
# Per-entry character cap to block evidence-bundle dumps and runaway lines.
MAX_PEER_CARD_ENTRY_LENGTH = 200
def _validate_peer_card_entry(line: str) -> bool:
"""Structural validation for a single peer card entry.
Returns True when the line starts with one of the allowed prefixes followed
by a space, has a non-empty body after the prefix, and fits within the per-
entry length cap. Subject-substance correctness (is this actually about the
observed peer?) is left to the prompt — this is form-only.
"""
if not line or len(line) > MAX_PEER_CARD_ENTRY_LENGTH:
return False
for prefix in PEER_CARD_ALLOWED_PREFIXES:
prefix_with_space = f"{prefix} "
if line.startswith(prefix_with_space):
body = line[len(prefix_with_space) :].strip()
return bool(body)
return False
def _normalized_observation_input(
obs: schemas.ObservationInput,
@ -313,16 +344,46 @@ class ObservationsCreatedResult:
failed: list[ObservationFailure]
def _truncate_tool_output(output: str, max_chars: int | None = None) -> str:
"""Truncate tool output to prevent token explosion."""
def _truncate_tool_output(
output: str, max_chars: int | None = None
) -> tuple[str, int, bool]:
"""Truncate tool output to prevent token explosion.
Returns (text, original_chars, was_truncated). Callers thread the
truncation signal into `ToolResult.metadata` so
`AgentToolCallCompletedEvent` can report `was_truncated` and
`result_chars_before_truncation` instead of always emitting them as
None/False.
"""
if max_chars is None:
max_chars = settings.LLM.MAX_TOOL_OUTPUT_CHARS
if len(output) <= max_chars:
return output
truncated = output[:max_chars]
return (
truncated
+ f"\n\n[OUTPUT TRUNCATED - showing {max_chars:,} of {len(output):,} characters]"
original_chars = len(output)
if original_chars <= max_chars:
return output, original_chars, False
truncated = (
output[:max_chars]
+ f"\n\n[OUTPUT TRUNCATED - showing {max_chars:,} of {original_chars:,} characters]"
)
return truncated, original_chars, True
def _maybe_truncated_result(output: str) -> "str | ToolResult":
"""Run `_truncate_tool_output` and wrap in `ToolResult` only when the
output was actually clamped, so the truncation signal reaches the
`AgentToolCallCompletedEvent` emitter (which reads `was_truncated` /
`result_chars_before_truncation` from `ToolResult.metadata`). Returns a
bare `str` in the common no-truncation case to keep the handler
contract unchanged.
"""
content, original_chars, was_truncated = _truncate_tool_output(output)
if not was_truncated:
return content
return ToolResult(
content=content,
metadata={
"was_truncated": True,
"result_chars_before_truncation": original_chars,
},
)
@ -429,9 +490,16 @@ TOOLS: dict[str, dict[str, Any]] = {
"update_peer_card": {
"name": "update_peer_card",
"description": (
"Update the peer card with durable profile facts about the observed peer. "
+ "Only include stable biographical facts, standing instructions, and long-lived preferences/traits. "
+ "Do not include one-off conclusions, temporary events, or duplicate entries."
"Update the peer card with stable identity markers about the observed peer. "
"An identity marker distinguishes the peer from others of its kind and persists across interactions. "
"The peer may be any entity with identity that changes over time (human, agent, codebase, team, organization) — do not assume the peer is human. "
"Each entry must start with one of four prefixes: `IDENTITY:` (canonical name, kind, aliases, IDs), "
"`ATTRIBUTE:` (stable durable property, including explicitly stated standing preferences), "
"`RELATIONSHIP:` (durable link to another entity), or "
"`INSTRUCTION:` (standing rule of engagement the peer has explicitly stated). "
"Do not write `TRAIT:` or behavioral `PREFERENCE:` entries, one-off observations, transient state, "
"inferred facts not directly supported by evidence, evidence bundles / `e.g.` clauses, or entries about co-occurring peers. "
"Entries without an allowed prefix or that exceed the per-entry length cap are rejected."
),
"input_schema": {
"type": "object",
@ -440,7 +508,9 @@ TOOLS: dict[str, dict[str, Any]] = {
"type": "array",
"description": (
"Complete deduplicated peer card list (max 40 entries). "
+ "Each entry should be a concise standalone profile fact."
"Each entry must start with one of the allowed prefixes "
"(`IDENTITY: `, `ATTRIBUTE: `, `RELATIONSHIP: `, `INSTRUCTION: `) "
"followed by one concise identity marker. Entries without an allowed prefix are rejected."
),
"items": {"type": "string"},
},
@ -765,7 +835,7 @@ DEDUCTION_SPECIALIST_TOOLS: list[dict[str, Any]] = [
# Tools for the induction specialist (dreamer phase 2)
# Creates inductive observations from explicit and deductive observations
# Includes message access for context and self-directed exploration
# Note: get_peer_card is not included - peer card is injected into the prompt directly
# Induction does not write to the peer card — that is deduction's responsibility.
INDUCTION_SPECIALIST_TOOLS: list[dict[str, Any]] = [
# Discovery tools
TOOLS["get_recent_observations"],
@ -773,7 +843,6 @@ INDUCTION_SPECIALIST_TOOLS: list[dict[str, Any]] = [
TOOLS["search_messages"],
# Action tools
TOOLS["create_observations_inductive"],
TOOLS["update_peer_card"],
]
@ -785,6 +854,8 @@ async def create_observations(
workspace_name: str,
message_ids: list[int],
message_created_at: str,
run_id: str | None = None,
parent_category: str | None = None,
) -> ObservationsCreatedResult:
"""
Create multiple observations (documents) in the memory system in a single call.
@ -799,6 +870,9 @@ async def create_observations(
workspace_name: Workspace identifier
message_ids: List of message IDs these observations are based on
message_created_at: Timestamp of the message that triggered these observations
run_id: Agent run id, threaded onto the embedding-call ContextVar so
EmbeddingCallCompletedEvents emitted here can be joined back to
the originating agent run.
Returns:
ObservationsCreatedResult with created count and any per-observation failures
@ -808,13 +882,15 @@ async def create_observations(
return ObservationsCreatedResult(created_count=0, created_levels=[], failed=[])
normalized_observations = [
_normalized_observation_input(obs) for obs in observations if obs.content.strip()
_normalized_observation_input(obs)
for obs in observations
if obs.content.strip()
]
if not normalized_observations:
logger.info("No non-empty observations to create")
return ObservationsCreatedResult(created_count=0, created_levels=[], failed=[])
# Phase 1: Ensure collection exists (short DB scope)
# Ensure collection exists (short DB scope)
async with tracked_db("create_observations.collection") as db:
await crud.get_or_create_collection(
db,
@ -823,11 +899,17 @@ async def create_observations(
observed=observed,
)
# Phase 2: Compute embeddings (no DB needed)
# Compute embeddings (no DB needed)
contents = [obs.content for obs in normalized_observations]
embeddings_by_index: dict[int, list[float]] | None = None
try:
embeddings = await embedding_client.simple_batch_embed(contents)
with embedding_call_purpose(
EmbeddingCallPurpose.CREATE_OBSERVATIONS.value,
workspace_name=workspace_name,
run_id=run_id,
parent_category=parent_category,
):
embeddings = await embedding_client.simple_batch_embed(contents)
embeddings_by_index = dict(
zip(range(len(normalized_observations)), embeddings, strict=True)
)
@ -846,7 +928,13 @@ async def create_observations(
embedding = embeddings_by_index[i]
else:
try:
embedding = await embedding_client.embed(obs.content)
with embedding_call_purpose(
EmbeddingCallPurpose.CREATE_OBSERVATIONS.value,
workspace_name=workspace_name,
run_id=run_id,
parent_category=parent_category,
):
embedding = await embedding_client.embed(obs.content)
except Exception as e:
logger.warning(
"Error embedding observation content for level '%s': %s",
@ -890,7 +978,7 @@ async def create_observations(
)
documents.append(doc)
# Phase 3: Bulk create all documents (short DB scope)
# Bulk create all documents (short DB scope)
accepted: list[schemas.DocumentCreate] = []
if documents:
async with tracked_db("create_observations.save") as db:
@ -1207,7 +1295,7 @@ async def _handle_create_observations_impl(
tool_input: dict[str, Any],
*,
forced_level: str | None = None,
) -> str:
) -> "str | ToolResult":
"""Handle create_observations tool."""
raw_observations = tool_input.get("observations", [])
@ -1271,6 +1359,8 @@ async def _handle_create_observations_impl(
workspace_name=ctx.workspace_name,
message_ids=message_ids,
message_created_at=message_created_at,
run_id=ctx.run_id,
parent_category=ctx.parent_category,
)
# Merge validation and embedding failures
@ -1311,18 +1401,27 @@ async def _handle_create_observations_impl(
)
response += f"\nFailed {len(all_failures)}: {failure_details}"
return response
# +5: surface created_count so DreamSpecialistEvent can sum actual
# observations across the run rather than just counting create_observations
# calls (which would conflate "1 call that made 5 observations" with
# "5 calls that each made 1").
from src.utils.types import ToolResult
return ToolResult(
content=response,
metadata={"created_count": result.created_count, "levels": levels},
)
async def _handle_create_observations(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
return await _handle_create_observations_impl(ctx, tool_input)
async def _handle_create_observations_deductive(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
return await _handle_create_observations_impl(
ctx,
tool_input,
@ -1332,7 +1431,7 @@ async def _handle_create_observations_deductive(
async def _handle_create_observations_inductive(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
return await _handle_create_observations_impl(
ctx,
tool_input,
@ -1340,7 +1439,9 @@ async def _handle_create_observations_inductive(
)
async def _handle_update_peer_card(ctx: ToolContext, tool_input: dict[str, Any]) -> str:
async def _handle_update_peer_card(
ctx: ToolContext, tool_input: dict[str, Any]
) -> "str | ToolResult":
"""Handle update_peer_card tool."""
# Check if peer card creation is disabled via configuration
if ctx.configuration is not None and not ctx.configuration.peer_card.create:
@ -1362,9 +1463,16 @@ async def _handle_update_peer_card(ctx: ToolContext, tool_input: dict[str, Any])
)
return "Peer card content was empty, no update performed."
# Normalize and deduplicate to keep peer cards bounded and stable.
# Normalize, validate structure, and deduplicate to keep peer cards bounded
# and on-spec.
normalized_peer_card: list[str] = []
seen: set[str] = set()
rejected_count = 0
# Keep a small sample of rejected entries to surface back to the model so it
# can self-correct on a retry. Capped to avoid bloating the tool response.
rejected_samples: list[str] = []
_REJECTED_SAMPLE_CAP = 3
_REJECTED_SAMPLE_LINE_LIMIT = 120
items = (
cast(list[str], raw_peer_card_content)
if isinstance(raw_peer_card_content, list)
@ -1375,6 +1483,16 @@ async def _handle_update_peer_card(ctx: ToolContext, tool_input: dict[str, Any])
if not line:
continue
if not _validate_peer_card_entry(line):
rejected_count += 1
if len(rejected_samples) < _REJECTED_SAMPLE_CAP:
rejected_samples.append(line[:_REJECTED_SAMPLE_LINE_LIMIT])
logger.info(
"Rejecting peer card entry (no allowed prefix, empty body, or over length cap): %r",
line[:80],
)
continue
# Case-insensitive dedupe with whitespace normalization.
normalized_key = " ".join(line.lower().split())
if normalized_key in seen:
@ -1382,12 +1500,44 @@ async def _handle_update_peer_card(ctx: ToolContext, tool_input: dict[str, Any])
seen.add(normalized_key)
normalized_peer_card.append(line)
# Don't clear the peer card if all content normalized to empty.
if rejected_count:
logger.info(
"Peer card update for %s/%s/%s rejected %d structurally invalid entries",
ctx.workspace_name,
ctx.observer,
ctx.observed,
rejected_count,
)
def _format_rejection_feedback(scope: str) -> str:
"""Build a self-correction hint for the model. `scope` is grammar glue:
either "all" (every entry rejected) or e.g. "3 of 12" (partial)."""
samples_block = ""
if rejected_samples:
sample_lines = "\n".join(f" - {s!r}" for s in rejected_samples)
extra = (
f" (+{rejected_count - len(rejected_samples)} more)"
if rejected_count > len(rejected_samples)
else ""
)
samples_block = f" Examples of rejected entries{extra}:\n{sample_lines}"
return (
f"Rejected {scope} entries for failing structural validation. "
"Each entry must start with one of `IDENTITY: `, `ATTRIBUTE: `, "
"`RELATIONSHIP: `, or `INSTRUCTION: ` and stay under the per-entry "
f"length cap.{samples_block}"
)
# Don't clear the peer card if all content normalized to empty or every
# entry was structurally invalid.
if not normalized_peer_card:
logger.warning(
"Peer card update normalized to empty for %s, keeping existing card",
"Peer card update normalized to empty for %s (rejected=%d), keeping existing card",
ctx.workspace_name,
rejected_count,
)
if rejected_count:
return _format_rejection_feedback(f"all {rejected_count}")
return "Peer card content was empty after normalization, no update performed."
if len(normalized_peer_card) > MAX_PEER_CARD_FACTS:
@ -1427,12 +1577,34 @@ async def _handle_update_peer_card(ctx: ToolContext, tool_input: dict[str, Any])
)
)
return f"Updated peer card for {ctx.observed} by {ctx.observer}"
# signal a successful peer_card update so DreamSpecialistEvent
# can set its `peer_card_updated` flag without name-counting.
from src.utils.types import ToolResult
success_content = (
f"Updated peer card for {ctx.observed} by {ctx.observer} "
f"with {len(normalized_peer_card)} entries."
)
if rejected_count:
# Partial reject: surface the rejection so the model can re-emit the
# dropped entries (with correct prefixes) on a retry instead of
# silently losing them.
accepted = len(normalized_peer_card)
total = accepted + rejected_count
success_content = f"{success_content} {_format_rejection_feedback(f'{rejected_count} of {total}')}"
return ToolResult(
content=success_content,
metadata={
"peer_card_updated": True,
"facts_count": len(normalized_peer_card),
"rejected_count": rejected_count,
},
)
async def _handle_get_recent_history(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
"""Handle get_recent_history tool."""
_ = tool_input
async with tracked_db("tool.get_recent_history") as db:
@ -1454,21 +1626,39 @@ async def _handle_get_recent_history(
else f"from {ctx.observed} across sessions"
)
output = f"Conversation history ({len(history)} messages {scope}):\n{history_text}"
return _truncate_tool_output(output)
return _maybe_truncated_result(output)
async def _handle_search_memory(ctx: ToolContext, tool_input: dict[str, Any]) -> str:
async def _handle_search_memory(
ctx: ToolContext, tool_input: dict[str, Any]
) -> "str | ToolResult":
"""Handle search_memory tool."""
from src.utils.types import ToolResult
top_k = min(_safe_int(tool_input.get("top_k"), 20), 40)
query = tool_input["query"]
try:
query_embedding = await embedding_client.embed(query)
with embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MEMORY.value,
workspace_name=ctx.workspace_name,
run_id=ctx.run_id,
parent_category=ctx.parent_category,
):
query_embedding = await embedding_client.embed(query)
except ValueError:
return (
"ERROR: Query exceeds maximum token limit of "
+ f"{settings.EMBEDDING.MAX_INPUT_TOKENS}. Please use a shorter query."
)
# Base telemetry metadata; results_count gets filled in below.
search_meta: dict[str, Any] = {
"top_k": top_k,
"used_embedding": True,
"embedding_query_count": 1,
"query_tokens": _estimate_tokens_safe(query),
}
documents = await crud.query_documents(
db=None,
workspace_name=ctx.workspace_name,
@ -1481,11 +1671,13 @@ async def _handle_search_memory(ctx: ToolContext, tool_input: dict[str, Any]) ->
mem = Representation.from_documents(documents)
total_count = mem.len()
if total_count == 0:
# fallback behavior: if the memory is *empty*, that means we're quite
# early in a workspace/peer/session -- in order to give good answers in
# this stage, and be efficient with tool calls, and make sure the model
# doesn't short-circuit and think there's nothing here, we
# automatically search the message history for relevant information.
# Empty-memory fallback: if the memory is *empty*, that means we're
# quite early in a workspace/peer/session -- in order to give good
# answers in this stage, and be efficient with tool calls, and make
# sure the model doesn't short-circuit and think there's nothing
# here, we automatically search the message history for relevant
# information.
zero_hit_meta = {**search_meta, "results_count": 0}
if ctx.agent_type == "dialectic":
limit = min(_safe_int(tool_input.get("top_k"), 20), 20)
message_output = None
@ -1503,21 +1695,33 @@ async def _handle_search_memory(ctx: ToolContext, tool_input: dict[str, Any]) ->
snippets, f"for query '{query}'"
)
if message_output:
return (
f"No observations yet. Message search results:\n\n{message_output}"
fallback_meta = {**zero_hit_meta, "results_count": len(snippets)}
return ToolResult(
content=f"No observations yet. Message search results:\n\n{message_output}",
metadata=fallback_meta,
)
return (
f"No observations found for query '{query}', and no messages found in "
"history. Try a different phrasing or use grep_messages for exact text."
return ToolResult(
content=(
f"No observations found for query '{query}', and no messages found in "
"history. Try a different phrasing or use grep_messages for exact text."
),
metadata=zero_hit_meta,
)
return f"No observations found for query '{query}'"
return ToolResult(
content=f"No observations found for query '{query}'",
metadata=zero_hit_meta,
)
mem_str = mem.str_with_ids() if ctx.include_observation_ids else str(mem)
return f"Found {total_count} observations for query '{query}':\n\n{mem_str}"
search_meta["results_count"] = total_count
return ToolResult(
content=f"Found {total_count} observations for query '{query}':\n\n{mem_str}",
metadata=search_meta,
)
async def _handle_get_observation_context(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
"""Handle get_observation_context tool."""
async with tracked_db("tool.get_observation_context") as db:
messages = await get_observation_context(
@ -1540,16 +1744,26 @@ async def _handle_get_observation_context(
]
)
output = f"Retrieved {len(messages)} messages with context:\n{messages_text}"
return _truncate_tool_output(output)
return _maybe_truncated_result(output)
async def _handle_search_messages(ctx: ToolContext, tool_input: dict[str, Any]) -> str:
async def _handle_search_messages(
ctx: ToolContext, tool_input: dict[str, Any]
) -> "str | ToolResult":
"""Handle search_messages tool."""
from src.utils.types import ToolResult
query = tool_input["query"]
limit = min(_safe_int(tool_input.get("limit"), 10), 20) # Cap at 20
# Pre-compute embedding outside DB session to avoid holding a connection
# during the external API call (same pattern as _handle_search_memory).
query_embedding = await embedding_client.embed(query)
with embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MESSAGES.value,
workspace_name=ctx.workspace_name,
run_id=ctx.run_id,
parent_category=ctx.parent_category,
):
query_embedding = await embedding_client.embed(query)
snippets = await crud.search_messages(
workspace_name=ctx.workspace_name,
session_name=ctx.session_name,
@ -1559,13 +1773,25 @@ async def _handle_search_messages(ctx: ToolContext, tool_input: dict[str, Any])
embedding=query_embedding,
observer=ctx.observer,
)
search_meta: dict[str, Any] = {
"top_k": limit,
"used_embedding": True,
"embedding_query_count": 1,
"query_tokens": _estimate_tokens_safe(query),
"results_count": len(snippets),
}
if not snippets:
return f"No messages found for query '{query}'"
return ToolResult(
content=f"No messages found for query '{query}'",
metadata=search_meta,
)
formatted = _format_message_snippets(snippets, f"for query '{query}'")
return formatted
return ToolResult(content=formatted, metadata=search_meta)
async def _handle_grep_messages(ctx: ToolContext, tool_input: dict[str, Any]) -> str:
async def _handle_grep_messages(
ctx: ToolContext, tool_input: dict[str, Any]
) -> "str | ToolResult":
"""Handle grep_messages tool."""
text = tool_input.get("text", "")
if not text:
@ -1606,7 +1832,7 @@ async def _handle_grep_messages(ctx: ToolContext, tool_input: dict[str, Any]) ->
f"Found {total_matches} messages containing '{text}' in {len(snippets)} conversation snippets:\n\n"
+ "\n\n".join(snippet_texts)
)
return _truncate_tool_output(output)
return _maybe_truncated_result(output)
def _parse_date(date_str: str | None, param_name: str) -> datetime | None | str:
@ -1621,7 +1847,7 @@ def _parse_date(date_str: str | None, param_name: str) -> datetime | None | str:
async def _handle_get_messages_by_date_range(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
"""Handle get_messages_by_date_range tool."""
after_date_str = tool_input.get("after_date")
before_date_str = tool_input.get("before_date")
@ -1677,12 +1903,12 @@ async def _handle_get_messages_by_date_range(
output = (
f"Found {msg_count} messages ({range_desc}, {order_desc}):\n\n{messages_text}"
)
return _truncate_tool_output(output)
return _maybe_truncated_result(output)
async def _handle_search_messages_temporal(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
"""Handle search_messages_temporal tool."""
query = tool_input.get("query", "")
if not query:
@ -1703,7 +1929,13 @@ async def _handle_search_messages_temporal(
# Pre-compute embedding outside DB session to avoid holding a connection
# during the external API call.
query_embedding = await embedding_client.embed(query)
with embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MESSAGES.value,
workspace_name=ctx.workspace_name,
run_id=ctx.run_id,
parent_category=ctx.parent_category,
):
query_embedding = await embedding_client.embed(query)
snippets = await crud.search_messages_temporal(
workspace_name=ctx.workspace_name,
session_name=ctx.session_name,
@ -1722,11 +1954,25 @@ async def _handle_search_messages_temporal(
date_filter.append(f"before {before_date_str}")
filter_desc = f" ({' and '.join(date_filter)})" if date_filter else ""
# Matches the search_messages metadata shape so analytics can filter
# AgentToolCallCompletedEvent uniformly across all embedding-backed
# search tools (search_memory / search_messages / search_messages_temporal).
search_meta: dict[str, Any] = {
"top_k": limit,
"used_embedding": True,
"embedding_query_count": 1,
"query_tokens": _estimate_tokens_safe(query),
"results_count": len(snippets),
}
if not snippets:
return f"No messages found for query '{query}'{filter_desc}"
return ToolResult(
content=f"No messages found for query '{query}'{filter_desc}",
metadata=search_meta,
)
formatted = _format_message_snippets(snippets, f"for query '{query}'{filter_desc}")
return formatted
return ToolResult(content=formatted, metadata=search_meta)
async def _handle_get_recent_observations(
@ -1820,7 +2066,7 @@ async def _handle_get_peer_card(ctx: ToolContext, tool_input: dict[str, Any]) ->
async def _handle_delete_observations(
ctx: ToolContext, tool_input: dict[str, Any]
) -> str:
) -> "str | ToolResult":
"""Handle delete_observations tool."""
observation_ids = tool_input.get("observation_ids", [])
if not observation_ids:
@ -1859,7 +2105,16 @@ async def _handle_delete_observations(
)
)
return f"Deleted {deleted_count} observations"
# +5: surface deleted_count + levels for DreamSpecialistEvent rollups.
from src.utils.types import ToolResult
return ToolResult(
content=f"Deleted {deleted_count} observations",
metadata={
"deleted_count": deleted_count,
"levels": [level for _, level in deleted],
},
)
async def _handle_finish_consolidation(
@ -1876,12 +2131,20 @@ async def _handle_extract_preferences(
) -> str:
"""Handle extract_preferences tool."""
_ = tool_input
results = await extract_preferences(
# Wrap so the batch-embed + downstream search_messages embedding calls
# all carry preference-extraction attribution.
with embedding_call_purpose(
EmbeddingCallPurpose.PREFERENCE_EXTRACTION.value,
workspace_name=ctx.workspace_name,
session_name=ctx.session_name,
observed=ctx.observed,
observer=ctx.observer,
)
run_id=ctx.run_id,
parent_category=ctx.parent_category,
):
results = await extract_preferences(
workspace_name=ctx.workspace_name,
session_name=ctx.session_name,
observed=ctx.observed,
observer=ctx.observer,
)
messages = results.get("messages", [])
@ -1901,7 +2164,13 @@ async def _handle_extract_preferences(
def _format_message_snippets(
snippets: list[tuple[list[models.Message], list[models.Message]]], desc: str
) -> str:
"""Format message snippets for output."""
"""Format message snippets for output.
Returns bare `str` because callers concatenate it into other strings
or place it into `ToolResult.content`. Callers that need the
truncation telemetry signal route their own output through
`_maybe_truncated_result` themselves.
"""
snippet_texts: list[str] = []
total_matches = sum(len(matches) for matches, _ in snippets)
for i, (matches, context) in enumerate(snippets, 1):
@ -1921,7 +2190,10 @@ def _format_message_snippets(
f"Found {total_matches} matching messages in {len(snippets)} conversation snippets {desc}:\n\n"
+ "\n\n".join(snippet_texts)
)
return _truncate_tool_output(output)
# `[0]` extracts the truncated text — telemetry signal is discarded here
# because callers wrap the result into ToolResult themselves (and so any
# downstream truncation telemetry should come from the caller's path).
return _truncate_tool_output(output)[0]
async def _handle_get_reasoning_chain(
@ -1960,9 +2232,7 @@ async def _handle_get_reasoning_chain(
premise_lines: list[Any] = []
for p in premises:
p_level = p.level or "explicit"
premise_lines.append(
f" - [id:{p.id}] ({p_level}): {p.content}"
)
premise_lines.append(f" - [id:{p.id}] ({p_level}): {p.content}")
output_parts.append(
f"\n**Premises ({len(premises)}):**\n"
+ "\n".join(premise_lines)
@ -1979,7 +2249,7 @@ async def _handle_get_reasoning_chain(
source_lines: list[Any] = []
for s in sources:
s_level = s.level or "explicit"
source_lines.append(f" - [id:{s.id}] ({s_level}): {s.content}")
source_lines.append(f" - [id:{s.id}] ({s_level}): {s.content}")
output_parts.append(
f"\n**Sources ({len(sources)}):**\n" + "\n".join(source_lines)
)
@ -2007,7 +2277,7 @@ async def _handle_get_reasoning_chain(
child_lines: list[Any] = []
for c in children:
c_level = c.level or "explicit"
child_lines.append(f" - [id:{c.id}] ({c_level}): {c.content}")
child_lines.append(f" - [id:{c.id}] ({c_level}): {c.content}")
output_parts.append(
f"\n**Derived Conclusions ({len(children)}):**\n"
+ "\n".join(child_lines)
@ -2110,33 +2380,187 @@ async def create_tool_executor(
Returns:
String result describing what was done
"""
logger.info("[tool call] %s %s", tool_name, tool_input)
import time
from src.utils.types import (
ToolResult,
get_current_iteration,
get_current_provider_tool_call_id,
get_current_tool_call_seq,
set_last_tool_metadata,
)
# Log nondisclosive call shape only. Raw `tool_input` can carry user
# content (search queries, peer-card text, etc.); the param keys are
# enough to reconstruct the call shape from telemetry without leaking
# content to log sinks.
logger.info("[tool call] %s keys=%s", tool_name, sorted(tool_input.keys()))
start = time.perf_counter()
# Defaults populated even on early returns / error paths so the
# AgentToolCallCompletedEvent emission below can fire consistently.
result_str: str = ""
metadata: dict[str, Any] = {}
is_error: bool = False
try:
handler = _TOOL_HANDLERS.get(tool_name)
if handler:
result = await handler(ctx, tool_input)
logger.info("[tool result] %s %s", tool_name, result)
return result
return f"Unknown tool: {tool_name}"
handler_result = await handler(ctx, tool_input)
# Handlers return either a plain str (existing contract) or a
# ToolResult(content, metadata) carrying structured fields for
# telemetry and specialist rollups.
if isinstance(handler_result, ToolResult):
result_str = handler_result.content
metadata = handler_result.metadata
else:
result_str = handler_result
# Log shape, not contents — `result_str` can carry retrieved
# observations, message snippets, peer-card text, etc. The
# AgentToolCallCompletedEvent telemetry captures the
# structured metadata for analytics.
logger.info(
"[tool result] %s len=%d metadata_keys=%s",
tool_name,
len(result_str),
sorted(metadata.keys()),
)
else:
result_str = f"Unknown tool: {tool_name}"
is_error = True
logger.warning(result_str)
except asyncio.CancelledError:
# Cancellation (client disconnect, server shutdown) — populate
# telemetry fields so the finally-block emit records an accurate
# event, then re-raise so cancellation propagates to the caller.
# CancelledError extends BaseException, so the broader except
# clauses below do not catch it.
result_str = f"Tool {tool_name} cancelled"
is_error = True
raise
except ValueError as e:
# Recoverable errors (bad input, validation failures) - return to LLM
error_msg = f"Tool {tool_name} failed with invalid input: {e}"
logger.warning(error_msg)
return error_msg
result_str = f"Tool {tool_name} failed with invalid input: {e}"
is_error = True
logger.warning(result_str)
except KeyError as e:
# Missing required parameters - return to LLM
error_msg = f"Tool {tool_name} missing required parameter: {e}"
logger.warning(error_msg)
return error_msg
result_str = f"Tool {tool_name} missing required parameter: {e}"
is_error = True
logger.warning(result_str)
except Exception as e:
# Unexpected errors - log with full traceback but still return to LLM
# We don't re-raise because the LLM should be able to continue with other tools
error_msg = f"Tool {tool_name} failed unexpectedly: {type(e).__name__}: {e}"
logger.error(error_msg, exc_info=True)
result_str = (
f"Tool {tool_name} failed unexpectedly: {type(e).__name__}: {e}"
)
is_error = True
logger.error(result_str, exc_info=True)
# No explicit rollback needed — each handler uses tracked_db() which
# handles rollback in its finally block
return error_msg
finally:
# Emit in finally so CancelledError (and any other BaseException)
# still produces an AgentToolCallCompletedEvent before propagating.
duration_ms = (time.perf_counter() - start) * 1000
# Publish ToolResult.metadata for tool_loop to stash on all_tool_calls.
# Reset to {} (rather than leaving stale metadata) so a non-ToolResult
# handler doesn't appear to have leaked metadata from a prior call.
set_last_tool_metadata(metadata)
_emit_agent_tool_call_completed(
ctx=ctx,
tool_name=tool_name,
duration_ms=duration_ms,
result_str=result_str,
metadata=metadata,
is_error=is_error,
iteration=get_current_iteration(),
tool_call_seq=get_current_tool_call_seq(),
provider_tool_call_id=get_current_provider_tool_call_id(),
)
return result_str
return execute_tool
def _emit_agent_tool_call_completed(
*,
ctx: "ToolContext",
tool_name: str,
duration_ms: float,
result_str: str,
metadata: dict[str, Any],
is_error: bool,
iteration: int,
tool_call_seq: int,
provider_tool_call_id: str | None,
) -> None:
"""Build and emit AgentToolCallCompletedEvent. Best-effort; swallows errors.
Skipped when the executor was constructed without agent identifiers
(run_id / agent_type / parent_category) — telemetry attribution requires
all three.
"""
if not (ctx.run_id and ctx.agent_type and ctx.parent_category):
return
try:
from src.telemetry.events import AgentToolCallCompletedEvent, emit
emit(
AgentToolCallCompletedEvent(
run_id=ctx.run_id,
iteration=iteration,
tool_call_seq=tool_call_seq,
provider_tool_call_id=provider_tool_call_id,
parent_category=ctx.parent_category,
agent_type=ctx.agent_type,
workspace_name=ctx.workspace_name,
tool_name=tool_name,
duration_ms=duration_ms,
is_error=is_error,
result_chars=len(result_str),
result_chars_before_truncation=metadata.get(
"result_chars_before_truncation"
),
result_tokens_estimate=_estimate_tokens(result_str),
was_truncated=bool(metadata.get("was_truncated", False)),
query_tokens=metadata.get("query_tokens"),
top_k=metadata.get("top_k"),
results_count=metadata.get("results_count"),
used_embedding=metadata.get("used_embedding"),
embedding_query_count=int(metadata.get("embedding_query_count") or 0),
)
)
except Exception: # pragma: no cover - telemetry must not raise
logger.debug("Failed to emit AgentToolCallCompletedEvent", exc_info=True)
def _estimate_tokens(text: str) -> int:
"""Tiktoken-based size proxy for tool result strings. Best-effort."""
if not text:
return 0
try:
import tiktoken
# Use cl100k_base as a stable default — matches the embedding-client
# fallback. Exact accuracy isn't required; this is a size proxy.
encoding = tiktoken.get_encoding("cl100k_base")
return len(encoding.encode(text))
except Exception:
# Fall back to a rough char→token ratio so the field is always populated.
return max(1, len(text) // 4)
def _estimate_tokens_safe(text: str | None) -> int | None:
"""Wrapper around `_estimate_tokens` that returns None on falsy input.
Used by search-handler metadata where we want `query_tokens=None`
when the query is empty rather than 0 (which could be confused with a
real measurement).
"""
if not text:
return None
return _estimate_tokens(text)

17
src/utils/queue_payload.py
View File

@ -57,6 +57,15 @@ class DreamPayload(BasePayload):
observer: str
observed: str
session_name: str | None = None
# scheduling context captured at schedule time so the
# eventual DreamRunEvent can attribute the cycle back to *why* it was
# scheduled (which threshold tripped) and *what* governed when it fired
# (idle delay vs. immediate vs. min-hours gate). Defaults preserve
# backward compat for any in-flight payloads from older producers.
trigger_reason: str | None = None
delay_reason: str | None = None
documents_since_last_dream_at_schedule: int | None = None
document_threshold: int | None = None
class DeletionPayload(BasePayload):
@ -90,6 +99,10 @@ def create_dream_payload(
observer: str,
observed: str,
session_name: str | None = None,
trigger_reason: str | None = None,
delay_reason: str | None = None,
documents_since_last_dream_at_schedule: int | None = None,
document_threshold: int | None = None,
) -> dict[str, Any]:
"""Create a dream payload."""
return DreamPayload(
@ -97,6 +110,10 @@ def create_dream_payload(
observer=observer,
observed=observed,
session_name=session_name,
trigger_reason=trigger_reason,
delay_reason=delay_reason,
documents_since_last_dream_at_schedule=documents_since_last_dream_at_schedule,
document_threshold=document_threshold,
).model_dump(mode="json", exclude_none=True)

9
src/utils/search.py
View File

@ -17,8 +17,10 @@ from src.dependencies import tracked_db
from src.embedding_client import embedding_client
from src.exceptions import ValidationException
from src.models import session_peers_table
from src.telemetry.events import EmbeddingCallPurpose
from src.utils.filter import apply_filter
from src.utils.formatting import ILIKE_ESCAPE_CHAR, escape_ilike_pattern
from src.utils.types import embedding_call_purpose
from src.vector_store import get_external_vector_store
T = TypeVar("T")
@ -380,7 +382,12 @@ async def search(
if settings.EMBED_MESSAGES and isinstance(workspace_name, str):
try:
query_embedding = await embedding_client.embed(query)
with embedding_call_purpose(
EmbeddingCallPurpose.SEARCH_MESSAGES.value,
workspace_name=workspace_name,
parent_category="api",
):
query_embedding = await embedding_client.embed(query)
except ValueError as e:
raise ValidationException(
f"Query exceeds maximum token limit of {settings.EMBEDDING.MAX_INPUT_TOKENS}."

49
src/utils/summarizer.py
View File

@ -16,9 +16,11 @@ from src.crud.session import session_cache_key
from src.dependencies import tracked_db
from src.exceptions import ResourceNotFoundException
from src.llm import HonchoLLMCallResponse, honcho_llm_call
from src.llm.types import LLMTelemetryContext
from src.models import Message
from src.telemetry import prometheus_metrics
from src.telemetry.events import AgentToolSummaryCreatedEvent, emit
from src.telemetry.events.llm import CallPurpose
from src.telemetry.logging import accumulate_metric, conditional_observe
from src.telemetry.prometheus.metrics import (
DeriverComponents,
@ -198,6 +200,8 @@ async def create_short_summary(
formatted_messages: str,
input_tokens: int,
previous_summary: str | None = None,
*,
workspace_name: str | None = None,
) -> HonchoLLMCallResponse[str]:
# input_tokens indicates how many tokens the message list + previous summary take up
# we want to optimize short summaries to be smaller than the actual content being summarized
@ -219,6 +223,11 @@ async def create_short_summary(
model_config=_get_summary_model_config(),
prompt=prompt,
max_tokens=settings.SUMMARY.MAX_TOKENS_SHORT,
telemetry=LLMTelemetryContext(
workspace_name=workspace_name,
call_purpose=CallPurpose.SUMMARY_SHORT.value,
parent_category="summary",
),
)
@ -226,6 +235,8 @@ async def create_short_summary(
async def create_long_summary(
formatted_messages: str,
previous_summary: str | None = None,
*,
workspace_name: str | None = None,
) -> HonchoLLMCallResponse[str]:
# the word/token ratio is roughly 4:3 so we multiply by 0.75.
# LLMs *seem* to respond better to getting asked for a word count but should workshop this.
@ -244,6 +255,11 @@ async def create_long_summary(
model_config=_get_summary_model_config(),
prompt=prompt,
max_tokens=settings.SUMMARY.MAX_TOKENS_LONG,
telemetry=LLMTelemetryContext(
workspace_name=workspace_name,
call_purpose=CallPurpose.SUMMARY_LONG.value,
parent_category="summary",
),
)
@ -437,16 +453,19 @@ async def _create_and_save_summary(
last_message_id=last_message_id,
last_message_content_preview=last_message_content_preview,
message_count=message_count,
workspace_name=workspace_name,
)
# Compute scaffold tokens up front (cheap + idempotent) so both the
# save-summary path and the telemetry emit below can use it
# without basedpyright tripping on a possibly-unbound name.
if summary_type == SummaryType.SHORT:
prompt_tokens = estimate_short_summary_prompt_tokens()
else:
prompt_tokens = estimate_long_summary_prompt_tokens()
# Step 3: Save to database with new transaction
if not is_fallback:
# Get base prompt tokens based on summary type
if summary_type == SummaryType.SHORT:
prompt_tokens = estimate_short_summary_prompt_tokens()
else:
prompt_tokens = estimate_long_summary_prompt_tokens()
track_deriver_input_tokens(
task_type=DeriverTaskTypes.SUMMARY,
components={
@ -499,6 +518,9 @@ async def _create_and_save_summary(
# Note: Using AgentToolSummaryCreatedEvent with dummy run_id/iteration since
# this is called from the deriver, not from an agentic loop
if not is_fallback:
# `prompt_tokens` is set in the `if not is_fallback` block above for
# both SHORT and LONG summary types — we're inside the same branch, so
# it's guaranteed bound here.
emit(
AgentToolSummaryCreatedEvent(
run_id="deriver", # Placeholder - not from an agentic run
@ -513,6 +535,10 @@ async def _create_and_save_summary(
summary_type="short" if summary_type == SummaryType.SHORT else "long",
input_tokens=llm_input_tokens,
output_tokens=llm_output_tokens,
# additive token-breakdown fields
previous_summary_tokens=previous_summary_tokens,
message_tokens=messages_tokens,
prompt_scaffold_tokens=prompt_tokens,
)
)
@ -526,6 +552,8 @@ async def _create_summary(
last_message_id: int,
last_message_content_preview: str,
message_count: int,
*,
workspace_name: str | None = None,
) -> tuple[Summary, bool, int, int]:
"""
Generate a summary of the provided messages using an LLM.
@ -554,11 +582,16 @@ async def _create_summary(
try:
if summary_type == SummaryType.SHORT:
response = await create_short_summary(
formatted_messages, input_tokens, previous_summary_text
formatted_messages,
input_tokens,
previous_summary_text,
workspace_name=workspace_name,
)
else:
response = await create_long_summary(
formatted_messages, previous_summary_text
formatted_messages,
previous_summary_text,
workspace_name=workspace_name,
)
summary_text = response.content

203
src/utils/types.py
View File

@ -1,7 +1,8 @@
from collections.abc import Awaitable, Callable
from collections.abc import Awaitable, Callable, Generator
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass, field
from typing import Generic, Literal, TypeVar
from typing import Any, Generic, Literal, TypeVar
T = TypeVar("T")
@ -20,6 +21,204 @@ def get_current_iteration() -> int:
return _current_iteration.get()
# ordinal of the tool call within its iteration. Two calls
# to the same tool in one iteration (the model can do this) need distinct
# resource ids on AgentToolCallCompletedEvent — seq disambiguates.
_current_tool_call_seq: ContextVar[int] = ContextVar("current_tool_call_seq", default=0)
# Optional provider-supplied tool-call id (e.g. Anthropic's `toolu_*`) so the
# emitted event can be cross-referenced with provider logs.
_current_provider_tool_call_id: ContextVar[str | None] = ContextVar(
"current_provider_tool_call_id", default=None
)
def set_current_tool_call_seq(seq: int, provider_tool_call_id: str | None) -> None:
"""Set the current tool-call ordinal + provider id for telemetry context.
Called by tool_loop before invoking the tool_executor closure. The seq
starts at 0 within each iteration's tool batch and increments per call.
"""
_current_tool_call_seq.set(seq)
_current_provider_tool_call_id.set(provider_tool_call_id)
def get_current_tool_call_seq() -> int:
return _current_tool_call_seq.get()
def get_current_provider_tool_call_id() -> str | None:
return _current_provider_tool_call_id.get()
# After `execute_tool` finishes, the metadata dict from the handler's
# ToolResult is published here so tool_loop can stash it on the
# `all_tool_calls` entry (which DreamSpecialistEvent reads for rollups).
# Default is None (not {}) per ruff B039 — mutable defaults on
# ContextVars are foot-guns; the getter normalizes None → {}.
_last_tool_metadata: ContextVar[dict[str, Any] | None] = ContextVar(
"last_tool_metadata", default=None
)
def set_last_tool_metadata(metadata: dict[str, Any]) -> None:
"""Publish the just-finished tool call's ToolResult metadata."""
_last_tool_metadata.set(metadata)
def get_last_tool_metadata() -> dict[str, Any]:
"""Read the last tool call's metadata. Returns {} when no ToolResult was returned."""
return _last_tool_metadata.get() or {}
@contextmanager
def iteration_scope() -> Generator[None]:
"""Reset per-tool-loop ContextVars on exit.
Wrap the body of `tool_loop.run_tool_loop` so a subsequent loop in the
same asyncio Task (worker batches, tests using TestClient) starts with
fresh iteration / tool-call state instead of inheriting stale values
from a prior loop. ContextVars are per-Task in asyncio, so cross-request
leakage is unlikely under normal FastAPI use — but this is defensive and
cheap.
"""
iter_token = _current_iteration.set(0)
seq_token = _current_tool_call_seq.set(0)
pid_token = _current_provider_tool_call_id.set(None)
meta_token = _last_tool_metadata.set(None)
try:
yield
finally:
_current_iteration.reset(iter_token)
_current_tool_call_seq.reset(seq_token)
_current_provider_tool_call_id.reset(pid_token)
_last_tool_metadata.reset(meta_token)
# embedding-call purpose ContextVar. Callers wrap embedding-driving
# operations in `with embedding_call_purpose("search_memory"): ...` so the
# embedding client can stamp every provider call with the originating intent
# without changing the call signature. None = caller didn't instrument; the
# event still emits but with call_purpose unset.
_embedding_call_purpose: ContextVar[str | None] = ContextVar(
"embedding_call_purpose", default=None
)
# Companion ContextVars so the event can also carry workspace + run
# correlation without threading kwargs through every embedding call site.
_embedding_workspace_name: ContextVar[str | None] = ContextVar(
"embedding_workspace_name", default=None
)
_embedding_run_id: ContextVar[str | None] = ContextVar("embedding_run_id", default=None)
# Parent category for joining EmbeddingCallCompletedEvent against the
# workflow that drove the call (e.g. "dialectic", "deriver",
# "reconciliation", "api"). Optional — None when uninstrumented.
_embedding_parent_category: ContextVar[str | None] = ContextVar(
"embedding_parent_category", default=None
)
def get_embedding_call_purpose() -> str | None:
"""Read the current embedding call purpose. None when uninstrumented."""
return _embedding_call_purpose.get()
def get_embedding_workspace_name() -> str | None:
"""Read the workspace name attached to the current embedding call scope."""
return _embedding_workspace_name.get()
def get_embedding_run_id() -> str | None:
"""Read the run_id attached to the current embedding call scope."""
return _embedding_run_id.get()
def get_embedding_parent_category() -> str | None:
"""Read the parent category attached to the current embedding call scope."""
return _embedding_parent_category.get()
@contextmanager
def embedding_call_purpose(
purpose: str,
*,
workspace_name: str | None = None,
run_id: str | None = None,
parent_category: str | None = None,
) -> Generator[None]:
"""Tag any embedding calls made inside this `with` block.
`purpose` should match an `EmbeddingCallPurpose` enum value (see
src/telemetry/events/llm.py). Unknown values pass through silently and
land as None on the event — the emitter validates against the enum.
`workspace_name` and `run_id` let the emitted event correlate back to
a specific workspace and agent run. Both are optional; callers that
don't have one (or have it set further up the stack via a wider
`with` block) can omit it.
`parent_category` joins the event back to the originating workflow —
typically the same category used by the calling LLM agent ("dialectic",
"deriver", "reconciliation", "api"). Lets analytics pivot embedding
cost/latency by workflow without per-purpose joins.
"""
purpose_token = _embedding_call_purpose.set(purpose)
workspace_token = (
_embedding_workspace_name.set(workspace_name)
if workspace_name is not None
else None
)
run_id_token = _embedding_run_id.set(run_id) if run_id is not None else None
parent_category_token = (
_embedding_parent_category.set(parent_category)
if parent_category is not None
else None
)
try:
yield
finally:
_embedding_call_purpose.reset(purpose_token)
if workspace_token is not None:
_embedding_workspace_name.reset(workspace_token)
if run_id_token is not None:
_embedding_run_id.reset(run_id_token)
if parent_category_token is not None:
_embedding_parent_category.reset(parent_category_token)
@dataclass
class ToolResult:
"""Internal return shape used by tool handlers.
Handlers may continue to return a plain `str` (existing contract). When
they need to carry structured metadata for downstream events — search
`top_k`/`results_count` for AgentToolCallCompletedEvent, or
`created_count`/`deleted_count` for specialist rollups — they
return `ToolResult(content=..., metadata={...})` instead. The
`execute_tool` closure in `create_tool_executor` unwraps the dataclass
before returning the string to `tool_loop`.
Treat this as a private contract between agent_tools.py and tool_loop.py.
Public callers see only the string content.
The `__contains__` and `__str__` overrides exist so direct handler-unit
tests that predate the dataclass — e.g. `assert "Created 2" in result`
— keep working without churning every assertion. Anything beyond
substring / str() (like `.lower()`) should access `.content` explicitly.
"""
content: str
metadata: dict[str, Any] = field(default_factory=dict)
def __contains__(self, item: object) -> bool:
# Only meaningful for substring checks; matches the legacy str-return
# contract used by handler unit tests.
if not isinstance(item, str):
return False
return item in self.content
def __str__(self) -> str:
return self.content
@dataclass
class GetOrCreateResult(Generic[T]):
"""Result of a get_or_create operation indicating whether the resource was created."""

25
src/vector_store/turbopuffer.py
View File

@ -11,7 +11,7 @@ from typing import Any, Literal, cast
from turbopuffer import AsyncTurbopuffer, InternalServerError, NotFoundError
from turbopuffer.lib.namespace import AsyncNamespace
from turbopuffer.types import Filter
from turbopuffer.types import Filter, RowParam
from src.config import settings
from src.exceptions import VectorStoreError
@ -77,12 +77,23 @@ class TurbopufferVectorStore(VectorStore):
ns = self._get_namespace(namespace)
rows: list[dict[str, Any]] = [
{
"id": v.id,
"vector": v.embedding,
**(v.metadata or {}),
}
# The dict literal carries arbitrary metadata fields, which RowParam supports
# via extra_items=object. basedpyright can't see through the spread, so cast
# via object per its reportInvalidCast guidance.
# Spread metadata first so a caller-supplied "id" or "vector" key
# can never clobber the required upsert fields.
rows: list[RowParam] = [
cast(
RowParam,
cast(
object,
{
**(v.metadata or {}),
"id": v.id,
"vector": v.embedding,
},
),
)
for v in vectors
]

18
tests/bench/molecular.py
View File

@ -634,7 +634,7 @@ class MolecularJudge:
"required": ["analyses"],
}
props_text = "\n".join(f'{i+1}. "{p}"' for i, p in enumerate(propositions))
props_text = "\n".join(f'{i + 1}. "{p}"' for i, p in enumerate(propositions))
result = await self._call_llm(
AMBIGUITY_DETECTION_PROMPT,
@ -717,7 +717,7 @@ class MolecularJudge:
"required": ["analyses"],
}
props_text = "\n".join(f'{i+1}. "{p}"' for i, p in enumerate(propositions))
props_text = "\n".join(f'{i + 1}. "{p}"' for i, p in enumerate(propositions))
result = await self._call_llm(
DECONTEXTUALITY_PROMPT,
@ -814,7 +814,7 @@ class MolecularJudge:
"required": ["analyses"],
}
props_text = "\n".join(f'{i+1}. "{p}"' for i, p in enumerate(propositions))
props_text = "\n".join(f'{i + 1}. "{p}"' for i, p in enumerate(propositions))
result = await self._call_llm(
MINIMALITY_PROMPT,
@ -996,16 +996,16 @@ def print_report(report: MolecularReport) -> None:
print("CLASSIFICATION DISTRIBUTION:")
total = report.proposition_count
print(
f" ✓ Molecular: {report.molecular_count:3d} ({report.molecular_count/total*100:5.1f}%)"
f" ✓ Molecular: {report.molecular_count:3d} ({report.molecular_count / total * 100:5.1f}%)"
)
print(
f" ⚠ Too Atomic: {report.too_atomic_count:3d} ({report.too_atomic_count/total*100:5.1f}%)"
f" ⚠ Too Atomic: {report.too_atomic_count:3d} ({report.too_atomic_count / total * 100:5.1f}%)"
)
print(
f" ⚠ Too Verbose: {report.too_verbose_count:3d} ({report.too_verbose_count/total*100:5.1f}%)"
f" ⚠ Too Verbose: {report.too_verbose_count:3d} ({report.too_verbose_count / total * 100:5.1f}%)"
)
print(
f" ~ Borderline: {report.borderline_count:3d} ({report.borderline_count/total*100:5.1f}%)"
f" ~ Borderline: {report.borderline_count:3d} ({report.borderline_count / total * 100:5.1f}%)"
)
# Show top issues
@ -1135,7 +1135,7 @@ async def main():
peer = extract_peer_name(trace)
conv_id = extract_conversation_id(trace, idx)
print(f"[{idx+1}/{len(all_traces)}] {conv_id} ({len(props)} props)...")
print(f"[{idx + 1}/{len(all_traces)}] {conv_id} ({len(props)} props)...")
try:
report = await judge.evaluate(props, msgs, peer, conv_id)
@ -1208,7 +1208,7 @@ async def main():
print("\nClassification:")
total_props = sum(r.proposition_count for r in results)
for k, v in agg["classification_totals"].items():
print(f" {k:<15} {v:4d} ({v/total_props*100:5.1f}%)")
print(f" {k:<15} {v:4d} ({v / total_props * 100:5.1f}%)")
else:
print(f"Duration: {format_duration(total_duration)}")
else:

20
tests/bench/oolong_common.py
View File

@ -652,20 +652,24 @@ def filter_dataset(
if max_context_len is not None:
dataset = dataset.filter(
lambda x: x.get(
"context_len",
calculate_context_length(str(x.get("context_window_text", ""))),
lambda x: (
x.get(
"context_len",
calculate_context_length(str(x.get("context_window_text", ""))),
)
<= max_context_len
)
<= max_context_len
)
if min_context_len is not None:
dataset = dataset.filter(
lambda x: x.get(
"context_len",
calculate_context_length(str(x.get("context_window_text", ""))),
lambda x: (
x.get(
"context_len",
calculate_context_length(str(x.get("context_window_text", ""))),
)
> min_context_len
)
> min_context_len
)
if max_examples is not None and max_examples > 0:

2
tests/bench/runner_common.py
View File

@ -99,7 +99,7 @@ def add_common_arguments(parser: argparse.ArgumentParser) -> None:
"--base-url",
type=str,
default=None,
help="Base URL for remote Honcho instance (e.g., https://groudon.fly.dev). Overrides --base-api-port.",
help="Base URL for remote Honcho instance (e.g., https://api.example.com). Overrides --base-api-port.",
)
parser.add_argument(

36
tests/crud/test_representation_manager.py
View File

@ -1,6 +1,5 @@
from contextlib import asynccontextmanager
from datetime import datetime, timezone
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
import pytest
@ -10,6 +9,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
from src import models
from src.crud.representation import RepresentationManager
from src.schemas.configuration import (
ResolvedConfiguration,
ResolvedDreamConfiguration,
ResolvedPeerCardConfiguration,
ResolvedReasoningConfiguration,
ResolvedSummaryConfiguration,
)
from src.utils.representation import (
DeductiveObservation,
ExplicitObservation,
@ -17,6 +23,20 @@ from src.utils.representation import (
)
def _resolved_config(*, dream_enabled: bool = False) -> ResolvedConfiguration:
"""Build a minimal ResolvedConfiguration for tests that only care about dream.enabled."""
return ResolvedConfiguration(
reasoning=ResolvedReasoningConfiguration(enabled=False),
peer_card=ResolvedPeerCardConfiguration(use=False, create=False),
summary=ResolvedSummaryConfiguration(
enabled=False,
messages_per_short_summary=20,
messages_per_long_summary=60,
),
dream=ResolvedDreamConfiguration(enabled=dream_enabled),
)
@asynccontextmanager
async def _fake_tracked_db(_name: str):
yield object()
@ -24,7 +44,7 @@ async def _fake_tracked_db(_name: str):
def _saved_observations(mock_save: AsyncMock):
call = mock_save.await_args
assert call is not None, "mock was not awaited"
assert call is not None, "mock_save was never awaited"
if "all_observations" in call.kwargs:
return call.kwargs["all_observations"]
if len(call.args) > 1:
@ -205,9 +225,7 @@ class TestRepresentationManagerSave:
message_ids=[1],
session_name="session",
message_created_at=datetime.now(timezone.utc),
message_level_configuration=SimpleNamespace( # pyright: ignore[reportArgumentType]
dream=SimpleNamespace(enabled=False)
),
message_level_configuration=_resolved_config(),
)
assert saved == 1
@ -261,9 +279,7 @@ class TestRepresentationManagerSave:
message_ids=[1],
session_name="session",
message_created_at=datetime.now(timezone.utc),
message_level_configuration=SimpleNamespace( # pyright: ignore[reportArgumentType]
dream=SimpleNamespace(enabled=False)
),
message_level_configuration=_resolved_config(),
)
assert saved == 1
@ -314,9 +330,7 @@ class TestRepresentationManagerSave:
message_ids=[1],
session_name="session",
message_created_at=datetime.now(timezone.utc),
message_level_configuration=SimpleNamespace( # pyright: ignore[reportArgumentType]
dream=SimpleNamespace(enabled=False)
),
message_level_configuration=_resolved_config(),
)
assert saved == 0

106
tests/deriver/test_deriver_processing.py
View File

@ -210,6 +210,112 @@ class TestDeriverProcessing:
# Verify the methods were called
assert mock_representation_manager.save_representation.called # type: ignore[attr-defined]
async def test_warns_when_response_input_tokens_less_than_messages_tokens(
self,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Data-quality invariant: provider should report at least as many
input tokens as we summed from messages. Drift surfaces as a WARNING
so analytics alerting can catch it."""
import logging
message = Mock(
id=1,
public_id="msg_drift",
session_name="session-1",
workspace_name="workspace-1",
peer_name="alice",
content="hello",
token_count=100,
created_at=datetime.now(timezone.utc),
)
configuration = Mock()
configuration.reasoning.enabled = True
mock_response = HonchoLLMCallResponse(
content=PromptRepresentation(explicit=[]),
input_tokens=10,
output_tokens=5,
finish_reasons=["STOP"],
)
with (
patch(
"src.deriver.deriver.honcho_llm_call",
new_callable=AsyncMock,
return_value=mock_response,
),
caplog.at_level(logging.WARNING, logger="src.deriver.deriver"),
):
await process_representation_tasks_batch(
messages=[message],
message_level_configuration=configuration,
observers=["bob"],
observed="alice",
queue_item_message_ids=[1],
)
assert any(
"token-breakdown invariant violated" in record.message
and record.levelno == logging.WARNING
for record in caplog.records
)
async def test_warns_when_prompt_scaffold_tokens_is_zero(
self,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Data-quality invariant: prompt scaffold estimator returning 0
signals a silent failure — log WARNING so the metric pipeline can
alert."""
import logging
message = Mock(
id=1,
public_id="msg_scaffold_zero",
session_name="session-1",
workspace_name="workspace-1",
peer_name="alice",
content="hello",
token_count=5,
created_at=datetime.now(timezone.utc),
)
configuration = Mock()
configuration.reasoning.enabled = True
mock_response = HonchoLLMCallResponse(
content=PromptRepresentation(explicit=[]),
input_tokens=100,
output_tokens=5,
finish_reasons=["STOP"],
)
with (
patch(
"src.deriver.deriver.honcho_llm_call",
new_callable=AsyncMock,
return_value=mock_response,
),
patch(
"src.deriver.deriver.estimate_deriver_prompt_tokens",
return_value=0,
),
caplog.at_level(logging.WARNING, logger="src.deriver.deriver"),
):
await process_representation_tasks_batch(
messages=[message],
message_level_configuration=configuration,
observers=["bob"],
observed="alice",
queue_item_message_ids=[1],
)
assert any(
"prompt_scaffold_tokens estimated as 0" in record.message
and record.levelno == logging.WARNING
for record in caplog.records
)
class TestBackwardsCompatibility:
"""Test backwards compatibility for queue items created before the deduplication change."""

235
tests/deriver/test_queue_processing.py
View File

@ -58,7 +58,7 @@ class TestQueueProcessing:
async def test_work_unit_claiming(
self,
db_session: AsyncSession,
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
sample_session_with_peers: tuple[models.Session, list[models.Peer]],
) -> None:
"""Test that work units can be claimed and are not available to other workers"""
@ -92,7 +92,7 @@ class TestQueueProcessing:
@pytest.mark.asyncio
async def test_get_and_claim_excludes_already_claimed(
self,
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
) -> None:
queue_manager = QueueManager()
first_batch = await queue_manager.get_and_claim_work_units()
@ -106,7 +106,7 @@ class TestQueueProcessing:
async def test_claim_work_unit_conflict_returns_false(
self,
db_session: AsyncSession,
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
) -> None:
# Pre-create an active session for a key
queue_manager = QueueManager()
@ -183,29 +183,31 @@ class TestQueueProcessing:
await db_session.commit()
await db_session.refresh(aqs)
_, items_to_process, _ = await qm.get_queue_item_batch(
batch = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=first.work_unit_key,
aqs_id=aqs.id,
)
items_to_process = batch.items_to_process
nxt = items_to_process[0] if items_to_process else None
assert nxt is not None and nxt.id == first.id
# Mark first processed, next should be the second
first.processed = True
await db_session.commit()
_, items_to_process2, _ = await qm.get_queue_item_batch(
batch2 = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=first.work_unit_key,
aqs_id=aqs.id,
)
items_to_process2 = batch2.items_to_process
nxt2 = items_to_process2[0] if items_to_process2 else None
assert nxt2 is not None and nxt2.id == second.id
@pytest.mark.asyncio
async def test_cleanup_work_unit_removes_row(
self,
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
sample_queue_items: list[models.QueueItem], # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
db_session: AsyncSession,
) -> None:
qm = QueueManager()
@ -358,6 +360,7 @@ class TestQueueProcessing:
observed: str | None = None, # pyright: ignore[reportUnusedParameter]
observers: list[str] | None = None, # pyright: ignore[reportUnusedParameter]
queue_item_message_ids: list[int] | None = None, # pyright: ignore[reportUnusedParameter]
**_extra: Any, # added hit_batch_token_cap / was_flush_enabled / batch_max_tokens
) -> None:
processed_batches.append(
{
@ -391,6 +394,203 @@ class TestQueueProcessing:
assert processed_batches[1]["payload_count"] == 1
assert all(b["task_type"] == "representation" for b in processed_batches)
@pytest.mark.asyncio
async def test_hit_batch_token_cap_reflects_post_filter_batch(
self,
db_session: AsyncSession,
sample_session_with_peers: tuple[models.Session, list[models.Peer]],
create_queue_payload: Callable[..., Any],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Regression: `hit_batch_token_cap` must reflect the actually-returned
batch (post config-filter), not the pre-filter superset. Previously
the flag used pre-filter `messages_context[-1].id`, which inflated
the range queried for cap detection and produced false positives
when the config-filter trimmed the trailing item from the batch.
"""
from src.deriver import queue_manager as qm_module
session, peers = sample_session_with_peers
peer = peers[0]
cap = settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS
# M1 + M2 sum to exactly the cap; M3 pushes over it. After SQL,
# messages_context = [M1, M2]; the cap is genuinely binding *on the
# pre-filter batch*. items_to_process = [QI(M1), QI(M2)]. We then
# simulate a config-filter trim that keeps only QI(M1). The
# actually-returned batch is [M1] alone — sum=400 < cap — so the
# cap-flag must report False.
token_counts = [400, cap - 400, 300]
messages: list[models.Message] = []
for i, tc in enumerate(token_counts):
m = models.Message(
session_name=session.name,
workspace_name=session.workspace_name,
peer_name=peer.name,
content=f"cap-test message {i}",
token_count=tc,
seq_in_session=i + 1,
)
db_session.add(m)
messages.append(m)
await db_session.commit()
for m in messages:
await db_session.refresh(m)
queue_items: list[models.QueueItem] = []
for m in messages:
payload = create_queue_payload( # type: ignore[reportUnknownArgumentType]
message=m,
task_type="representation",
observed=peer.name,
observer=peer.name,
)
work_unit_key = construct_work_unit_key(session.workspace_name, payload)
qi = models.QueueItem(
session_id=session.id,
task_type="representation",
work_unit_key=work_unit_key,
payload=payload,
processed=False,
workspace_name=session.workspace_name,
message_id=m.id,
)
db_session.add(qi)
queue_items.append(qi)
await db_session.commit()
for qi in queue_items:
await db_session.refresh(qi)
# Trim items_to_process down to the first queue item — mimics
# `_resolve_batch_configuration` cutting at a configuration boundary.
real_resolve = qm_module._resolve_batch_configuration # pyright: ignore[reportPrivateUsage]
def fake_resolve(
items: list[models.QueueItem],
) -> tuple[list[models.QueueItem], Any]:
_kept, cfg = real_resolve(items)
return (items[:1] if items else []), cfg
monkeypatch.setattr(qm_module, "_resolve_batch_configuration", fake_resolve)
qm = qm_module.QueueManager()
work_unit_key = queue_items[0].work_unit_key
claimed = await qm.claim_work_units(db_session, [work_unit_key])
aqs_id = claimed[work_unit_key]
await db_session.commit()
result = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=work_unit_key,
aqs_id=aqs_id,
)
# Returned batch is [M1] alone (config-filter trimmed M2). The cap
# wasn't binding on this batch — sum=400 < cap. Pre-fix code reported
# True (false positive) because it used pre-filter max_kept_id=M2.
assert len(result.messages_context) == 1
assert result.messages_context[0].id == messages[0].id
assert result.hit_batch_token_cap is False
@pytest.mark.asyncio
async def test_hit_batch_token_cap_fires_when_trailing_context_trimmed(
self,
db_session: AsyncSession,
sample_session_with_peers: tuple[models.Session, list[models.Peer]],
create_queue_payload: Callable[..., Any],
) -> None:
"""Regression: when SQL kept trailing NON-QUEUE context past the last
queued item, and the config filter trims that context away, the cap
check must still recognize that the SQL cap clamped queue work.
Pre-fix used `messages_context[-1].id == sql_max_kept_id` which goes
False whenever trailing context is dropped — producing a false
negative for the very case the cap-hit flag exists to report.
Post-fix keys on the queue-item boundary, which is unaffected by
trailing-context trimming.
"""
from src.deriver import queue_manager as qm_module
session, peers = sample_session_with_peers
peer_a = peers[0]
peer_b = peers[1] if len(peers) > 1 else peers[0]
cap = settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS
# Layout: 4 messages, ordered.
# M1 (peer_a, queue, 200)
# M2 (peer_a, queue, 200)
# M3 (peer_b, NON-queue context, cap - 300)
# M4 (peer_a, queue, 400)
# Cumulative tokens: M1=200, M2=400, M3=cap+100, M4=cap+500.
# SQL keeps M1+M2 (cumulative <= cap), excludes M3 onwards (over cap).
# Wait — we want SQL to keep through M3 (trailing context) but exclude
# M4 (queue). Adjust so M3 fits but M4 doesn't.
token_counts: list[tuple[models.Peer, int]] = [
(peer_a, 200), # M1 — queue
(peer_a, 200), # M2 — queue
(peer_b, cap - 700), # M3 — non-queue context; cumulative = cap-300
(peer_a, 400), # M4 — queue; cumulative cap+100 > cap → excluded
]
messages: list[models.Message] = []
for i, (msg_peer, tc) in enumerate(token_counts):
m = models.Message(
session_name=session.name,
workspace_name=session.workspace_name,
peer_name=msg_peer.name,
content=f"cap-test message {i}",
token_count=tc,
seq_in_session=i + 1,
)
db_session.add(m)
messages.append(m)
await db_session.commit()
for m in messages:
await db_session.refresh(m)
# Queue items for M1, M2, M4 only — M3 is non-queue context (peer_b).
# observed = peer_a, so the deriver's representation work unit covers
# peer_a messages; peer_b is treated as conversational context.
queue_items: list[models.QueueItem] = []
for m in [messages[0], messages[1], messages[3]]:
payload = create_queue_payload( # type: ignore[reportUnknownArgumentType]
message=m,
task_type="representation",
observed=peer_a.name,
observer=peer_a.name,
)
work_unit_key = construct_work_unit_key(session.workspace_name, payload)
qi = models.QueueItem(
session_id=session.id,
task_type="representation",
work_unit_key=work_unit_key,
payload=payload,
processed=False,
workspace_name=session.workspace_name,
message_id=m.id,
)
db_session.add(qi)
queue_items.append(qi)
await db_session.commit()
for qi in queue_items:
await db_session.refresh(qi)
qm = qm_module.QueueManager()
work_unit_key = queue_items[0].work_unit_key
claimed = await qm.claim_work_units(db_session, [work_unit_key])
aqs_id = claimed[work_unit_key]
await db_session.commit()
result = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=work_unit_key,
aqs_id=aqs_id,
)
# Cap fired: SQL stopped at M3 (cap budget exhausted), excluding the
# queue item M4. Config filter doesn't touch queue items here. The
# flag must report True.
assert result.hit_batch_token_cap is True
@pytest.mark.asyncio
async def test_token_batching_filters_by_work_unit(
self,
@ -485,11 +685,13 @@ class TestQueueProcessing:
await db_session.commit()
await db_session.refresh(alice_aqs)
alice_messages, alice_items, _ = await qm.get_queue_item_batch(
alice_batch = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=alice_work_unit_key,
aqs_id=alice_aqs.id,
)
alice_messages = alice_batch.messages_context
alice_items = alice_batch.items_to_process
assert len(alice_messages) == 6
alice_message_ids: set[int] = {m.id for m in alice_messages}
@ -513,11 +715,13 @@ class TestQueueProcessing:
await db_session.commit()
await db_session.refresh(bob_aqs)
bob_messages, bob_items, _ = await qm.get_queue_item_batch(
bob_batch = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=bob_work_unit_key,
aqs_id=bob_aqs.id,
)
bob_messages = bob_batch.messages_context
bob_items = bob_batch.items_to_process
# Bob should get 5 messages (1..5) - includes preceding alice message for context
assert len(bob_messages) == 5
@ -540,11 +744,13 @@ class TestQueueProcessing:
await db_session.commit()
await db_session.refresh(steve_aqs)
steve_messages, steve_items, _ = await qm.get_queue_item_batch(
steve_batch = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=steve_work_unit_key,
aqs_id=steve_aqs.id,
)
steve_messages = steve_batch.messages_context
steve_items = steve_batch.items_to_process
# Steve should get 6 messages (2..7) - includes preceding bob message for context
assert len(steve_messages) == 6
@ -658,11 +864,12 @@ class TestQueueProcessing:
await db_session.commit()
await db_session.refresh(alice_aqs)
alice_messages2, _, _ = await qm.get_queue_item_batch(
alice_batch2 = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=alice_work_unit_key,
aqs_id=alice_aqs.id,
)
alice_messages2 = alice_batch2.messages_context
# Includes preceding steve message for context -> [2,3,4]
assert len(alice_messages2) == 3
@ -682,11 +889,12 @@ class TestQueueProcessing:
await db_session.commit()
await db_session.refresh(bob_aqs)
bob_messages2, _, _ = await qm.get_queue_item_batch(
bob_batch2 = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=bob_work_unit_key,
aqs_id=bob_aqs.id,
)
bob_messages2 = bob_batch2.messages_context
assert len(bob_messages2) == 1
assert bob_messages2[0].id == messages[0].id # bob only
@ -702,11 +910,12 @@ class TestQueueProcessing:
await db_session.commit()
await db_session.refresh(steve_aqs)
steve_messages2, _, _ = await qm.get_queue_item_batch(
steve_batch2 = await qm.get_queue_item_batch(
task_type="representation",
work_unit_key=steve_work_unit_key,
aqs_id=steve_aqs.id,
)
steve_messages2 = steve_batch2.messages_context
# Includes preceding bob message for context -> [1,2]
assert len(steve_messages2) == 2
@ -926,6 +1135,7 @@ class TestQueueProcessing:
observed: str | None = None, # pyright: ignore[reportUnusedParameter]
observers: list[str] | None = None, # pyright: ignore[reportUnusedParameter]
queue_item_message_ids: list[int] | None = None, # pyright: ignore[reportUnusedParameter]
**_extra: Any, # added hit_batch_token_cap / was_flush_enabled / batch_max_tokens
) -> None:
processed_batches.append(
{
@ -1046,6 +1256,7 @@ class TestQueueProcessing:
observed: str | None = None, # pyright: ignore[reportUnusedParameter]
observers: list[str] | None = None, # pyright: ignore[reportUnusedParameter]
queue_item_message_ids: list[int] | None = None, # pyright: ignore[reportUnusedParameter]
**_extra: Any, # added hit_batch_token_cap / was_flush_enabled / batch_max_tokens
) -> None:
processed_batches.append(
{

183
tests/dreamer/test_dream_v2_rollups.py Normal file
View File

@ -0,0 +1,183 @@
# pyright: reportPrivateUsage=false
"""tests: DreamSpecialistEvent + DreamRunEvent v2 fields.
Targets:
- Schema bumps to v2 (DreamRunEvent + DreamSpecialistEvent).
- Specialist rollups come from `tool_result_metadata`, NOT tool-name counting.
- DreamRunEvent carries the scheduler reasons threaded through the dream
queue payload.
"""
from __future__ import annotations
from src.telemetry.events.dream import DreamRunEvent, DreamSpecialistEvent
from src.utils.queue_payload import DreamPayload
class TestSchemaVersionsBumpedToV2:
def test_dream_run_event_at_v2(self):
assert DreamRunEvent.schema_version() == 2
def test_dream_specialist_event_at_v2(self):
assert DreamSpecialistEvent.schema_version() == 2
class TestDreamRunEventV2Fields:
def test_scheduler_fields_default(self):
"""Existing callers that don't supply scheduling fields must
still construct a valid event — all new fields default to None or 0."""
event = DreamRunEvent(
run_id="abc",
workspace_name="ws",
session_name=None,
observer="o",
observed="user",
specialists_run=["deduction", "induction"],
deduction_success=True,
induction_success=True,
total_iterations=10,
total_input_tokens=100,
total_output_tokens=20,
total_duration_ms=1000.0,
)
assert event.dream_type is None
assert event.enabled_types_count == 0
assert event.trigger_reason is None
assert event.delay_reason is None
assert event.documents_since_last_dream_at_schedule is None
assert event.document_threshold is None
def test_scheduler_reasons_round_trip(self):
event = DreamRunEvent(
run_id="abc",
workspace_name="ws",
session_name=None,
observer="o",
observed="user",
specialists_run=["deduction"],
deduction_success=True,
induction_success=False,
total_iterations=5,
total_input_tokens=50,
total_output_tokens=10,
total_duration_ms=500.0,
dream_type="omni",
enabled_types_count=1,
trigger_reason="document_threshold",
delay_reason="idle_timeout",
documents_since_last_dream_at_schedule=60,
document_threshold=50,
)
assert event.dream_type == "omni"
assert event.trigger_reason == "document_threshold"
assert event.delay_reason == "idle_timeout"
assert event.documents_since_last_dream_at_schedule == 60
assert event.document_threshold == 50
def test_threshold_and_delay_are_separate(self):
"""The two scheduler gates are intentionally separate fields. The
snapshot semantics differ: trigger_reason describes WHY the dream
was scheduled (which gate tripped); delay_reason describes WHEN it
will fire (idle vs immediate)."""
event = DreamRunEvent(
run_id="abc",
workspace_name="ws",
session_name=None,
observer="o",
observed="user",
specialists_run=["induction"],
deduction_success=False,
induction_success=True,
total_iterations=3,
total_input_tokens=30,
total_output_tokens=5,
total_duration_ms=100.0,
trigger_reason="document_threshold",
delay_reason="immediate",
)
# trigger_reason captures the WHY; delay_reason captures the WHEN.
# They are separate dimensions — flattening into one field would lose
# the gate semantics that was specifically designed to expose.
assert event.trigger_reason != event.delay_reason
class TestDreamSpecialistEventV2Rollups:
def test_rollup_fields_default(self):
event = DreamSpecialistEvent(
run_id="abc",
specialist_type="deduction",
workspace_name="ws",
observer="o",
observed="user",
iterations=3,
tool_calls_count=5,
input_tokens=100,
output_tokens=20,
duration_ms=500.0,
success=True,
)
assert event.created_observation_count == 0
assert event.deleted_observation_count == 0
assert event.peer_card_updated is False
assert event.search_tool_calls_count == 0
def test_observation_counts_are_observation_truth_not_call_counts(self):
"""The whole point of sourcing rollups from ToolResult.metadata
instead of tool-name counts: a single `create_observations` call can
produce N observations (or zero on validation failure). must
report observation truth, not call truth."""
event = DreamSpecialistEvent(
run_id="abc",
specialist_type="deduction",
workspace_name="ws",
observer="o",
observed="user",
iterations=2,
# Two create_observations CALLS, but they produced 7 observations
# together (e.g. one batch of 5, one batch of 2). reports
# 7 (the metadata-sourced truth), not 2 (the call count).
tool_calls_count=2,
input_tokens=100,
output_tokens=20,
duration_ms=500.0,
success=True,
created_observation_count=7,
)
assert event.tool_calls_count == 2
assert event.created_observation_count == 7
class TestDreamPayloadSchedulerFields:
def test_payload_defaults(self):
from src.schemas import DreamType
payload = DreamPayload(
dream_type=DreamType.OMNI,
observer="o",
observed="user",
)
assert payload.trigger_reason is None
assert payload.delay_reason is None
assert payload.documents_since_last_dream_at_schedule is None
assert payload.document_threshold is None
def test_payload_threads_scheduler_reasons(self):
from src.schemas import DreamType
payload = DreamPayload(
dream_type=DreamType.OMNI,
observer="o",
observed="user",
trigger_reason="document_threshold",
delay_reason="idle_timeout",
documents_since_last_dream_at_schedule=55,
document_threshold=50,
)
# Round-trip through serialization → deserialization mimics what
# happens between scheduler enqueue and consumer dequeue.
data = payload.model_dump(mode="json")
restored = DreamPayload(**data)
assert restored.trigger_reason == "document_threshold"
assert restored.delay_reason == "idle_timeout"
assert restored.documents_since_last_dream_at_schedule == 55
assert restored.document_threshold == 50

1
tests/dreamer/test_dreamer_integration.py
View File

@ -368,6 +368,7 @@ class TestExecuteDreamSessionFilter:
observed: str,
dream_type: Any,
session_name: str,
**_scheduler_extra: Any, # trigger_reason / delay_reason / etc.
) -> None:
captured_kwargs.update(
{

57
tests/dreamer/test_model_config_usage.py
View File

@ -3,10 +3,65 @@ from unittest.mock import AsyncMock, patch
import pytest
from src.config import settings
from src.dreamer.specialists import DeductionSpecialist
from src.dreamer.specialists import DeductionSpecialist, InductionSpecialist
from src.llm import HonchoLLMCallResponse
def test_deduction_prompt_uses_identity_markers_framing() -> None:
"""Deduction prompt must frame the peer card as an identity store with the
entity-agnostic prefix taxonomy, not as a human bio sheet."""
prompt = DeductionSpecialist().build_system_prompt("alice", peer_card_enabled=True)
assert "identity store" in prompt
assert "stable identity markers" in prompt
for prefix in ("IDENTITY:", "ATTRIBUTE:", "RELATIONSHIP:", "INSTRUCTION:"):
assert prefix in prompt
# Cross-entity examples confirm the prompt is not biased toward humans.
assert "codebase" in prompt
assert "team" in prompt
# Behavioral content must be explicitly excluded.
assert "TRAIT:" in prompt
# The old human-shaped REQUIRED enumeration must be gone.
assert "Family members and relationships" not in prompt
assert "Core preferences and traits" not in prompt
def test_deduction_prompt_omits_peer_card_when_disabled() -> None:
prompt = DeductionSpecialist().build_system_prompt("alice", peer_card_enabled=False)
assert "PEER CARD" not in prompt
assert "IDENTITY:" not in prompt
def test_induction_prompt_has_no_peer_card_section() -> None:
"""Induction no longer writes to the peer card; its prompt must not reference it."""
prompt = InductionSpecialist().build_system_prompt("alice", peer_card_enabled=True)
assert "PEER CARD" not in prompt
assert "update_peer_card" not in prompt
def test_induction_specialist_cannot_update_peer_card() -> None:
"""Induction must have can_update_peer_card=False and no update_peer_card tool."""
specialist = InductionSpecialist()
assert specialist.can_update_peer_card is False
tool_names = {t["name"] for t in specialist.get_tools()}
assert "update_peer_card" not in tool_names
# Sanity: induction still has the discovery and create tools it actually needs.
assert "create_observations_inductive" in tool_names
assert "search_memory" in tool_names
def test_deduction_specialist_can_update_peer_card() -> None:
specialist = DeductionSpecialist()
assert specialist.can_update_peer_card is True
tool_names = {t["name"] for t in specialist.get_tools(peer_card_enabled=True)}
assert "update_peer_card" in tool_names
disabled_names = {t["name"] for t in specialist.get_tools(peer_card_enabled=False)}
assert "update_peer_card" not in disabled_names
@pytest.mark.asyncio
async def test_deduction_specialist_uses_nested_model_config(
monkeypatch: pytest.MonkeyPatch,

1
tests/integration/test_telemetry.py
View File

@ -142,6 +142,7 @@ def create_representation_event(
llm_call_ms=1200.0,
total_duration_ms=1300.0,
input_tokens=5000,
total_input_tokens=7500,
output_tokens=500,
)

37
tests/llm/test_backends/test_anthropic.py
View File

@ -123,13 +123,32 @@ async def test_anthropic_backend_skips_assistant_prefill_for_claude_4_models() -
@pytest.mark.asyncio
async def test_anthropic_backend_rejects_thinking_effort() -> None:
backend = AnthropicBackend(Mock())
with pytest.raises(ValueError, match="does not support thinking_effort"):
await backend.complete(
model="claude-haiku-4-5",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=100,
thinking_effort="high",
async def test_anthropic_backend_ignores_thinking_effort() -> None:
client = Mock()
client.messages.create = AsyncMock(
return_value=SimpleNamespace(
content=[TextBlock(type="text", text="ok")],
usage=SimpleNamespace(
input_tokens=10,
output_tokens=5,
cache_creation_input_tokens=0,
cache_read_input_tokens=0,
),
stop_reason="end_turn",
)
)
backend = AnthropicBackend(client)
await backend.complete(
model="claude-haiku-4-5",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=100,
thinking_effort="high",
)
await_args = client.messages.create.await_args
if await_args is None:
raise AssertionError("Expected Anthropic client call")
call = await_args.kwargs
assert "thinking" not in call
assert "reasoning_effort" not in call

84
tests/llm/test_backends/test_openai.py
View File

@ -3,7 +3,6 @@ from unittest.mock import AsyncMock, Mock
import pytest
from src.exceptions import ValidationException
from src.llm.backends.openai import OpenAIBackend
@ -153,18 +152,79 @@ async def test_openai_backend_does_not_treat_proxy_models_with_gpt5_substring_as
@pytest.mark.asyncio
async def test_openai_backend_rejects_thinking_budget_tokens() -> None:
backend = OpenAIBackend(Mock())
with pytest.raises(
ValidationException, match="does not support thinking_budget_tokens"
):
await backend.complete(
model="gpt-5-mini",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=100,
thinking_budget_tokens=256,
async def test_openai_backend_passes_thinking_budget_via_extra_body() -> None:
client = Mock()
client.chat.completions.create = AsyncMock(
return_value=SimpleNamespace(
choices=[
SimpleNamespace(
finish_reason="stop",
message=SimpleNamespace(
content="ok",
tool_calls=[],
reasoning_details=[],
),
)
],
usage=SimpleNamespace(
prompt_tokens=10,
completion_tokens=5,
prompt_tokens_details=None,
),
)
)
backend = OpenAIBackend(client)
await backend.complete(
model="x-ai/grok-4.1-fast",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=100,
thinking_budget_tokens=256,
)
await_args = client.chat.completions.create.await_args
if await_args is None:
raise AssertionError("Expected OpenAI create call")
call = await_args.kwargs
assert call["extra_body"] == {"reasoning": {"max_tokens": 256}}
@pytest.mark.asyncio
async def test_openai_backend_skips_extra_body_when_thinking_budget_zero() -> None:
client = Mock()
client.chat.completions.create = AsyncMock(
return_value=SimpleNamespace(
choices=[
SimpleNamespace(
finish_reason="stop",
message=SimpleNamespace(
content="ok",
tool_calls=[],
reasoning_details=[],
),
)
],
usage=SimpleNamespace(
prompt_tokens=10,
completion_tokens=5,
prompt_tokens_details=None,
),
)
)
backend = OpenAIBackend(client)
await backend.complete(
model="x-ai/grok-4.1-fast",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=100,
thinking_budget_tokens=0,
)
await_args = client.chat.completions.create.await_args
if await_args is None:
raise AssertionError("Expected OpenAI create call")
call = await_args.kwargs
assert "extra_body" not in call
@pytest.mark.asyncio

295
tests/llm/test_telemetry_agent_iteration.py Normal file
View File

@ -0,0 +1,295 @@
# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false
"""tests for AgentIterationEvent emission.
Targets:
- `src/llm/tool_loop.py::_emit_agent_iteration` fires one event per LLM
response, including the no-tool terminating iteration and the max-iteration
synthesis call.
- Emission is skipped when telemetry context is missing or under-specified
(no agent_type / parent_category / workspace / run_id).
- The caller-supplied `LLMTelemetryContext` is never mutated; per-iteration
copies set the iteration field on a fresh instance.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import patch
from src.llm.tool_loop import (
_emit_agent_iteration,
_telemetry_for_iteration,
)
from src.llm.types import HonchoLLMCallResponse, LLMTelemetryContext
from src.telemetry.events import AgentIterationEvent, BaseEvent
def _response(
*,
tool_calls: list[dict[str, Any]] | None = None,
input_tokens: int = 100,
output_tokens: int = 25,
cache_read: int = 0,
cache_creation: int = 0,
) -> HonchoLLMCallResponse[Any]:
return HonchoLLMCallResponse(
content="",
input_tokens=input_tokens,
output_tokens=output_tokens,
cache_read_input_tokens=cache_read,
cache_creation_input_tokens=cache_creation,
finish_reasons=["stop"],
tool_calls_made=tool_calls or [],
)
class TestTelemetryForIteration:
def test_returns_none_when_base_is_none(self):
assert _telemetry_for_iteration(None, 1) is None
def test_returns_fresh_copy_with_iteration_set(self):
base = LLMTelemetryContext(
workspace_name="ws",
call_purpose="dialectic.answer",
parent_category="dialectic",
agent_type="dialectic",
run_id="run-xyz",
iteration=None,
peer_name="user_peer",
)
copy_a = _telemetry_for_iteration(base, 3)
copy_b = _telemetry_for_iteration(base, 4)
assert copy_a is not None and copy_b is not None
assert copy_a is not base and copy_b is not base
# Original is never mutated.
assert base.iteration is None
assert copy_a.iteration == 3
assert copy_b.iteration == 4
# All other fields round-trip.
assert copy_a.run_id == "run-xyz"
assert copy_a.peer_name == "user_peer"
class TestEmitAgentIteration:
def test_emits_event_with_tool_calls(self):
emitted: list[BaseEvent] = []
telemetry = LLMTelemetryContext(
workspace_name="ws",
parent_category="dream",
agent_type="deduction",
run_id="run-1",
observer="obs",
observed="obj",
)
response = _response(
tool_calls=[
{"name": "search_memory", "id": "t1", "input": {}},
{"name": "create_observations", "id": "t2", "input": {}},
],
input_tokens=500,
output_tokens=80,
cache_read=12,
cache_creation=5,
)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_iteration(telemetry, iteration=2, response=response)
assert len(emitted) == 1
event = emitted[0]
assert isinstance(event, AgentIterationEvent)
assert event.run_id == "run-1"
assert event.parent_category == "dream"
assert event.agent_type == "deduction"
assert event.workspace_name == "ws"
assert event.observer == "obs"
assert event.observed == "obj"
assert event.iteration == 2
assert event.tool_calls == ["search_memory", "create_observations"]
assert event.input_tokens == 500
assert event.output_tokens == 80
assert event.cache_read_tokens == 12
assert event.cache_creation_tokens == 5
def test_emits_terminating_iteration_with_empty_tool_calls(self):
"""The no-tool terminating iteration still counts. Empty tool_calls
list must produce a valid AgentIterationEvent."""
emitted: list[BaseEvent] = []
telemetry = LLMTelemetryContext(
workspace_name="ws",
parent_category="dialectic",
agent_type="dialectic",
run_id="run-2",
peer_name="user_peer",
)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_iteration(telemetry, iteration=4, response=_response())
assert len(emitted) == 1
event = emitted[0]
assert isinstance(event, AgentIterationEvent)
assert event.tool_calls == []
assert event.iteration == 4
def test_skips_when_telemetry_is_none(self):
emitted: list[BaseEvent] = []
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_iteration(None, iteration=1, response=_response())
assert emitted == []
def test_skips_when_run_id_missing(self):
emitted: list[BaseEvent] = []
telemetry = LLMTelemetryContext(
workspace_name="ws",
parent_category="dialectic",
agent_type="dialectic",
run_id=None,
)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_iteration(telemetry, iteration=1, response=_response())
assert emitted == []
def test_skips_when_agent_type_or_parent_category_missing(self):
"""LLMCallCompletedEvent can fire without agent fields (system call),
but agent.iteration is by definition an agent-loop event. If agent
metadata is missing, skip emission rather than send a half-populated
event."""
emitted: list[BaseEvent] = []
telemetry = LLMTelemetryContext(
workspace_name="ws",
parent_category=None,
agent_type="dialectic",
run_id="run-3",
)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_iteration(telemetry, iteration=1, response=_response())
assert emitted == []
def test_skips_when_workspace_missing(self):
emitted: list[BaseEvent] = []
telemetry = LLMTelemetryContext(
workspace_name=None,
parent_category="dream",
agent_type="induction",
run_id="run-4",
)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_iteration(telemetry, iteration=1, response=_response())
assert emitted == []
def test_swallows_emit_failures(self):
"""Telemetry failures must not bleed into the LLM call path."""
def explode(*_args: Any, **_kwargs: Any) -> None:
raise RuntimeError("emitter wedged")
telemetry = LLMTelemetryContext(
workspace_name="ws",
parent_category="dream",
agent_type="deduction",
run_id="run-5",
)
with patch("src.telemetry.events.emit", side_effect=explode):
# Must not raise.
_emit_agent_iteration(telemetry, iteration=1, response=_response())
def test_volume_class_is_high_volume():
"""emission targets a high-volume event class so the sampler
can throttle iteration events independently of aggregates."""
assert AgentIterationEvent.volume_class() == "high_volume"
class TestIterationScope:
"""`iteration_scope()` in src/utils/types.py captures Tokens for the
per-loop ContextVars and resets them on exit. Defensive against a
subsequent tool loop in the same asyncio Task seeing stale state from
a previous loop (worker batches, tests using TestClient).
"""
def test_resets_iteration_and_tool_call_state_on_exit(self):
from src.utils.types import (
get_current_iteration,
get_current_provider_tool_call_id,
get_current_tool_call_seq,
get_last_tool_metadata,
iteration_scope,
set_current_iteration,
set_current_tool_call_seq,
set_last_tool_metadata,
)
# Pre-scope: defaults.
assert get_current_iteration() == 0
assert get_current_tool_call_seq() == 0
assert get_current_provider_tool_call_id() is None
assert get_last_tool_metadata() == {}
with iteration_scope():
set_current_iteration(7)
set_current_tool_call_seq(3, "toolu_abc")
set_last_tool_metadata({"k": "v"})
assert get_current_iteration() == 7
assert get_current_tool_call_seq() == 3
assert get_current_provider_tool_call_id() == "toolu_abc"
assert get_last_tool_metadata() == {"k": "v"}
# Post-scope: every ContextVar reset to pre-scope state.
assert get_current_iteration() == 0
assert get_current_tool_call_seq() == 0
assert get_current_provider_tool_call_id() is None
assert get_last_tool_metadata() == {}
def test_resets_on_exception(self):
"""Exception inside the block still triggers the reset path."""
import pytest
from src.utils.types import (
get_current_iteration,
iteration_scope,
set_current_iteration,
)
with pytest.raises(RuntimeError, match="boom"), iteration_scope():
set_current_iteration(5)
raise RuntimeError("boom")
assert get_current_iteration() == 0
def test_sequential_scopes_do_not_leak(self):
"""Two back-to-back scopes (mimicking sequential tool loops) — the
second sees a clean baseline, not stale values from the first."""
from src.utils.types import (
get_current_iteration,
iteration_scope,
set_current_iteration,
)
with iteration_scope():
set_current_iteration(9)
with iteration_scope():
# Inside scope #2: iteration starts at 0 (the scope reset it),
# not at 9 from the prior scope.
assert get_current_iteration() == 0

355
tests/llm/test_telemetry_agent_tool_call.py Normal file
View File

@ -0,0 +1,355 @@
# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false, reportUnusedFunction=false
"""tests for AgentToolCallCompletedEvent emission.
Targets:
- `_emit_agent_tool_call_completed` in `src/utils/agent_tools.py` builds a
well-formed event from ToolContext + per-call metadata.
- ToolResult dataclass behaves like a string for `in` / `str()` so tests of
the existing handler contract keep working.
- The `tool_call_seq` ContextVar disambiguates resource ids when the same
tool is called twice in one iteration.
- event opts into the high-volume sampler.
- Search handlers publish search-specific metadata
(top_k / used_embedding / query_tokens / results_count) so analytics can
filter by retrieval intent.
"""
from __future__ import annotations
from collections.abc import Generator
from typing import Any, final
from unittest.mock import patch
import pytest
from src.telemetry.events import AgentToolCallCompletedEvent, BaseEvent
from src.utils.agent_tools import _emit_agent_tool_call_completed
from src.utils.types import (
ToolResult,
get_current_provider_tool_call_id,
get_current_tool_call_seq,
set_current_tool_call_seq,
)
@pytest.fixture(autouse=True)
def _reset_tool_call_contextvars() -> Generator[None]:
"""Restore the tool-call ContextVars after each test.
`set_current_tool_call_seq` mutates module-level state; without this
fixture, tests that read `get_current_tool_call_seq()` expecting the
default 0 become order-dependent on whichever test ran before.
"""
prev_seq = get_current_tool_call_seq()
prev_provider_id = get_current_provider_tool_call_id()
try:
yield
finally:
set_current_tool_call_seq(prev_seq, prev_provider_id)
@final
class _StubToolContext:
"""Duck-typed ToolContext stand-in — emitter only reads identifiers."""
workspace_name: str
run_id: str | None
agent_type: str | None
parent_category: str | None
def __init__(
self,
*,
workspace_name: str = "ws",
run_id: str | None = "run-1",
agent_type: str | None = "dialectic",
parent_category: str | None = "dialectic",
):
self.workspace_name = workspace_name
self.run_id = run_id
self.agent_type = agent_type
self.parent_category = parent_category
class TestToolResult:
def test_str_returns_content(self):
result = ToolResult(content="hello", metadata={"x": 1})
assert str(result) == "hello"
def test_contains_delegates_to_content(self):
"""ToolResult must satisfy existing 'substring in result' assertions."""
result = ToolResult(content="Created 3 observations", metadata={})
assert "Created 3" in result
assert "missing" not in result
def test_metadata_defaults_to_empty(self):
result = ToolResult(content="x")
assert result.metadata == {}
class TestToolCallSeqContextVar:
def test_seq_and_provider_id_round_trip(self):
set_current_tool_call_seq(2, "toolu_abc")
assert get_current_tool_call_seq() == 2
assert get_current_provider_tool_call_id() == "toolu_abc"
def test_provider_id_can_be_none(self):
set_current_tool_call_seq(0, None)
assert get_current_tool_call_seq() == 0
assert get_current_provider_tool_call_id() is None
class TestEmitAgentToolCallCompleted:
def test_emits_event_with_full_context(self):
emitted: list[BaseEvent] = []
ctx = _StubToolContext(
run_id="run-7", agent_type="deduction", parent_category="dream"
)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_tool_call_completed(
ctx=ctx,
tool_name="search_memory",
duration_ms=42.5,
result_str="Found 5 observations",
metadata={
"top_k": 20,
"used_embedding": True,
"embedding_query_count": 1,
"query_tokens": 7,
"results_count": 5,
},
is_error=False,
iteration=3,
tool_call_seq=1,
provider_tool_call_id="toolu_xyz",
)
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, AgentToolCallCompletedEvent)
assert ev.run_id == "run-7"
assert ev.parent_category == "dream"
assert ev.agent_type == "deduction"
assert ev.workspace_name == "ws"
assert ev.iteration == 3
assert ev.tool_call_seq == 1
assert ev.provider_tool_call_id == "toolu_xyz"
assert ev.tool_name == "search_memory"
assert ev.duration_ms == 42.5
assert ev.is_error is False
assert ev.result_chars == len("Found 5 observations")
# Search-specific fields surface from metadata.
assert ev.top_k == 20
assert ev.used_embedding is True
assert ev.embedding_query_count == 1
assert ev.query_tokens == 7
assert ev.results_count == 5
def test_resource_id_disambiguates_same_tool_in_iteration(self):
"""Resource id = {run_id}:{iteration}:{tool_call_seq}. Two calls to
the same tool in one iteration must produce DIFFERENT ids — otherwise
deterministic id generation would collide and dedupe would drop one
of the events."""
ev_a = AgentToolCallCompletedEvent(
run_id="run-1",
iteration=2,
tool_call_seq=0,
parent_category="dialectic",
agent_type="dialectic",
workspace_name="ws",
tool_name="search_memory",
duration_ms=1.0,
result_chars=10,
result_tokens_estimate=3,
)
ev_b = AgentToolCallCompletedEvent(
run_id="run-1",
iteration=2,
tool_call_seq=1,
parent_category="dialectic",
agent_type="dialectic",
workspace_name="ws",
tool_name="search_memory",
duration_ms=1.0,
result_chars=10,
result_tokens_estimate=3,
)
assert ev_a.get_resource_id() != ev_b.get_resource_id()
# Same timestamp + different resource_id → different deterministic ids.
ev_b.timestamp = ev_a.timestamp
assert ev_a.generate_id() != ev_b.generate_id()
def test_skips_when_run_id_missing(self):
emitted: list[BaseEvent] = []
ctx = _StubToolContext(run_id=None)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_tool_call_completed(
ctx=ctx,
tool_name="search_memory",
duration_ms=0.0,
result_str="",
metadata={},
is_error=False,
iteration=1,
tool_call_seq=0,
provider_tool_call_id=None,
)
assert emitted == []
def test_skips_when_agent_metadata_incomplete(self):
emitted: list[BaseEvent] = []
ctx = _StubToolContext(agent_type=None)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_tool_call_completed(
ctx=ctx,
tool_name="search_memory",
duration_ms=0.0,
result_str="",
metadata={},
is_error=False,
iteration=1,
tool_call_seq=0,
provider_tool_call_id=None,
)
assert emitted == []
def test_swallows_emit_failures(self):
"""Telemetry must never bleed into the tool path."""
def explode(*_args: Any, **_kwargs: Any) -> None:
raise RuntimeError("emitter wedged")
ctx = _StubToolContext()
with patch("src.telemetry.events.emit", side_effect=explode):
# Must not raise.
_emit_agent_tool_call_completed(
ctx=ctx,
tool_name="search_memory",
duration_ms=0.0,
result_str="",
metadata={},
is_error=False,
iteration=1,
tool_call_seq=0,
provider_tool_call_id=None,
)
def test_truncation_metadata_round_trips(self):
emitted: list[BaseEvent] = []
ctx = _StubToolContext()
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_tool_call_completed(
ctx=ctx,
tool_name="get_recent_history",
duration_ms=1.0,
result_str="abc",
metadata={
"was_truncated": True,
"result_chars_before_truncation": 9000,
},
is_error=False,
iteration=1,
tool_call_seq=0,
provider_tool_call_id=None,
)
ev = emitted[0]
assert isinstance(ev, AgentToolCallCompletedEvent)
assert ev.was_truncated is True
assert ev.result_chars_before_truncation == 9000
# Truncation delta = before - after = 9000 - 3 = 8997. Calibration can
# compute that downstream; we just verify both fields land.
assert ev.result_chars == 3
def test_volume_class_is_high_volume():
"""event is high-volume — sampled alongside llm.call.completed."""
assert AgentToolCallCompletedEvent.volume_class() == "high_volume"
class TestMaybeTruncatedResult:
"""`_maybe_truncated_result` should wrap in ToolResult only when the
helper actually clamps the output, and the wrapping must surface
the fields the AgentToolCallCompletedEvent reads from metadata.
These tests guard against the regression where the truncation signal
used to be discarded by `_truncate_tool_output` returning bare str —
leaving `was_truncated` / `result_chars_before_truncation` as dead
fields on the event.
"""
def test_under_cap_returns_bare_string(self):
"""No-cap path keeps the bare-str contract — no metadata wrapping."""
from src.utils.agent_tools import _maybe_truncated_result
out = _maybe_truncated_result("hello")
assert out == "hello"
assert not isinstance(out, ToolResult)
def test_over_cap_returns_tool_result_with_metadata(self):
"""Truncated path wraps in ToolResult carrying the original size."""
from src.config import settings
from src.utils.agent_tools import _maybe_truncated_result
original = "x" * 5000
with patch.object(settings.LLM, "MAX_TOOL_OUTPUT_CHARS", 100):
out = _maybe_truncated_result(original)
assert isinstance(out, ToolResult)
assert out.metadata["was_truncated"] is True
assert out.metadata["result_chars_before_truncation"] == 5000
# Content carries the truncation marker so the LLM knows it was clamped.
assert "OUTPUT TRUNCATED" in out.content
def test_end_to_end_truncation_event(self):
"""Wired path: truncated handler output reaches the event with
was_truncated=True. This is the regression check for the
previously-dead `was_truncated` / `result_chars_before_truncation`
fields on AgentToolCallCompletedEvent.
"""
from src.config import settings
from src.utils.agent_tools import _maybe_truncated_result
emitted: list[BaseEvent] = []
ctx = _StubToolContext()
original = "y" * 10_000
with patch.object(settings.LLM, "MAX_TOOL_OUTPUT_CHARS", 200):
wrapped = _maybe_truncated_result(original)
assert isinstance(wrapped, ToolResult)
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_agent_tool_call_completed(
ctx=ctx,
tool_name="get_recent_history",
duration_ms=1.0,
result_str=wrapped.content,
metadata=wrapped.metadata,
is_error=False,
iteration=1,
tool_call_seq=0,
provider_tool_call_id=None,
)
ev = emitted[0]
assert isinstance(ev, AgentToolCallCompletedEvent)
assert ev.was_truncated is True
assert ev.result_chars_before_truncation == 10_000

686
tests/llm/test_telemetry_llm_call.py Normal file
View File

@ -0,0 +1,686 @@
# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false
"""tests for LLMCallCompletedEvent emission and the high-volume sampler.
Targets:
- `src/llm/executor.py::honcho_llm_call_inner` emits one event per call,
on both success and failure (try/finally).
- `LLMTelemetryContext` round-trips workspace/run_id/iteration/call_purpose
onto the event without mutating the caller-supplied context.
- The sampler at `src/telemetry/emitter.py::_should_sample` keeps every event
of a run together (same run_id → same decision) and lets ground_truth
events through unconditionally.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import AsyncMock, patch
import pytest
from src.llm.backend import CompletionResult as BackendCompletionResult
from src.llm.executor import _emit_llm_call_completed
from src.llm.runtime import AttemptPlan
from src.llm.types import LLMTelemetryContext
from src.telemetry.events import (
BaseEvent,
CallPurpose,
DialecticCompletedEvent,
LLMCallCompletedEvent,
)
def _make_plan(
*, attempt: int = 1, retry_attempts: int = 3, is_fallback: bool = False
) -> AttemptPlan:
"""Minimal AttemptPlan; client/selected_config are unused by the emitter helper."""
return AttemptPlan(
provider="anthropic",
model="claude-sonnet-4-5",
client=object(),
thinking_budget_tokens=None,
reasoning_effort=None,
selected_config=object(),
attempt=attempt,
retry_attempts=retry_attempts,
is_fallback=is_fallback,
)
class TestEmitLLMCallCompleted:
def test_emits_success_event(self):
emitted: list[BaseEvent] = []
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_llm_call_completed(
plan=_make_plan(),
telemetry=LLMTelemetryContext(
workspace_name="ws1",
call_purpose=CallPurpose.DIALECTIC_ANSWER.value,
parent_category="dialectic",
run_id="run-xyz",
iteration=2,
),
provider="anthropic",
model="claude-sonnet-4-5",
max_tokens=2048,
duration_ms=300.0,
has_tools=True,
was_stream=False,
outcome="success",
result=BackendCompletionResult(
content="hi",
input_tokens=10,
output_tokens=5,
cache_read_input_tokens=2,
cache_creation_input_tokens=1,
finish_reason="stop",
),
error=None,
)
assert len(emitted) == 1
event = emitted[0]
assert isinstance(event, LLMCallCompletedEvent)
assert event.outcome == "success"
assert event.is_final_attempt is False
assert event.workspace_name == "ws1"
assert event.run_id == "run-xyz"
assert event.iteration == 2
assert event.call_purpose == CallPurpose.DIALECTIC_ANSWER
assert event.provider_input_tokens == 10
assert event.provider_output_tokens == 5
assert event.cache_read_tokens == 2
assert event.cache_creation_tokens == 1
assert event.finish_reason == "stop"
assert event.has_tools is True
def test_emits_error_event_with_class_name(self):
emitted: list[BaseEvent] = []
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_llm_call_completed(
plan=_make_plan(attempt=3, retry_attempts=3, is_fallback=True),
telemetry=None,
provider="openai",
model="gpt-4",
max_tokens=512,
duration_ms=15.0,
has_tools=False,
was_stream=False,
outcome="error",
result=None,
error=RuntimeError("nope"),
)
assert len(emitted) == 1
event = emitted[0]
assert isinstance(event, LLMCallCompletedEvent)
assert event.outcome == "error"
# On the last attempt, is_final_attempt must be True (replaces the
# synthetic "retry_exhausted" outcome from earlier drafts).
assert event.is_final_attempt is True
assert event.error_class == "RuntimeError"
assert event.was_fallback is True
# No result → token fields are 0.
assert event.provider_input_tokens == 0
assert event.provider_output_tokens == 0
def test_unknown_call_purpose_silently_dropped(self):
"""Unknown call_purpose strings should not raise; event still emits."""
emitted: list[BaseEvent] = []
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_llm_call_completed(
plan=_make_plan(),
telemetry=LLMTelemetryContext(call_purpose="not.a.real.purpose"),
provider="anthropic",
model="claude",
max_tokens=1,
duration_ms=0.0,
has_tools=False,
was_stream=False,
outcome="success",
result=BackendCompletionResult(),
error=None,
)
assert len(emitted) == 1
event = emitted[0]
assert isinstance(event, LLMCallCompletedEvent)
# Unknown purpose drops to None rather than raising.
assert event.call_purpose is None
def test_provider_label_inferred_from_openrouter_model_prefix(self):
emitted: list[BaseEvent] = []
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
_emit_llm_call_completed(
plan=_make_plan(),
telemetry=None,
provider="openai",
model="anthropic/claude-3-5-sonnet",
max_tokens=1,
duration_ms=0.0,
has_tools=False,
was_stream=False,
outcome="success",
result=BackendCompletionResult(),
error=None,
)
event = emitted[0]
assert isinstance(event, LLMCallCompletedEvent)
assert event.provider_label == "anthropic"
def test_telemetry_failures_swallowed(self):
"""A broken emitter must NOT propagate exceptions out of the LLM path."""
def explode(*_args: Any, **_kwargs: Any) -> None:
raise RuntimeError("emitter wedged")
with patch("src.telemetry.events.emit", side_effect=explode):
# Must not raise.
_emit_llm_call_completed(
plan=_make_plan(),
telemetry=None,
provider="anthropic",
model="claude",
max_tokens=1,
duration_ms=0.0,
has_tools=False,
was_stream=False,
outcome="success",
result=BackendCompletionResult(),
error=None,
)
class TestSampler:
"""Tests for the deterministic high-volume sampler."""
def test_rate_one_passes_everything(self):
from src.telemetry.emitter import _should_sample
event = LLMCallCompletedEvent(
transport="anthropic",
model="m",
effective_max_output_tokens=1,
outcome="success",
is_final_attempt=True,
attempt=1,
retry_attempts=1,
was_fallback=False,
duration_ms=0.0,
run_id="anything",
)
assert _should_sample(event, 1.0) is True
def test_rate_zero_drops_everything(self):
from src.telemetry.emitter import _should_sample
event = LLMCallCompletedEvent(
transport="anthropic",
model="m",
effective_max_output_tokens=1,
outcome="success",
is_final_attempt=True,
attempt=1,
retry_attempts=1,
was_fallback=False,
duration_ms=0.0,
run_id="anything",
)
assert _should_sample(event, 0.0) is False
def test_same_run_id_gets_same_decision(self):
"""Two events with the same run_id must hash to the same bucket so an
entire trace is either kept or dropped — never half-sampled."""
from src.telemetry.emitter import _should_sample
def make(iteration: int) -> LLMCallCompletedEvent:
return LLMCallCompletedEvent(
transport="anthropic",
model="m",
effective_max_output_tokens=1,
outcome="success",
is_final_attempt=False,
attempt=1,
retry_attempts=1,
was_fallback=False,
duration_ms=0.0,
run_id="stable-run-id",
iteration=iteration,
)
rate = 0.5
a = _should_sample(make(1), rate)
b = _should_sample(make(2), rate)
c = _should_sample(make(3), rate)
assert a == b == c
class TestExecutorEndToEnd:
"""Exercise honcho_llm_call_inner's try/finally on both paths."""
@pytest.mark.asyncio
async def test_success_path_emits_one_event(self):
from src.llm import executor
emitted: list[BaseEvent] = []
result = BackendCompletionResult(
content="ok", input_tokens=3, output_tokens=2, finish_reason="stop"
)
with (
patch.object(executor, "CLIENTS", {"anthropic": object()}),
patch.object(
executor,
"backend_for_provider",
return_value=object(),
),
patch.object(
executor,
"execute_completion",
new=AsyncMock(return_value=result),
),
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
):
await executor.honcho_llm_call_inner(
"anthropic",
"claude-sonnet-4-5",
"hello",
max_tokens=128,
plan=_make_plan(),
telemetry=LLMTelemetryContext(
workspace_name="ws",
call_purpose=CallPurpose.DERIVER_REPRESENTATION.value,
parent_category="representation",
),
)
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, LLMCallCompletedEvent)
assert ev.outcome == "success"
assert ev.provider_output_tokens == 2
@pytest.mark.asyncio
async def test_cancelled_path_emits_cancelled_outcome(self):
"""asyncio.CancelledError mid-call surfaces as outcome='cancelled', not
'error' — client disconnects / shutdowns must not pollute error rates."""
import asyncio
from src.llm import executor
emitted: list[BaseEvent] = []
async def _cancel(*_args: Any, **_kwargs: Any) -> Any:
raise asyncio.CancelledError()
with (
patch.object(executor, "CLIENTS", {"anthropic": object()}),
patch.object(executor, "backend_for_provider", return_value=object()),
patch.object(executor, "execute_completion", new=_cancel),
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
pytest.raises(asyncio.CancelledError),
):
await executor.honcho_llm_call_inner(
"anthropic",
"claude-sonnet-4-5",
"hello",
max_tokens=128,
plan=_make_plan(),
telemetry=None,
)
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, LLMCallCompletedEvent)
assert ev.outcome == "cancelled"
assert ev.error_class == "CancelledError"
@pytest.mark.asyncio
async def test_stream_cancelled_emits_cancelled_outcome(self):
"""Stream path: mid-iteration CancelledError surfaces as 'cancelled'."""
import asyncio
from collections.abc import AsyncIterator
from src.llm import executor
emitted: list[BaseEvent] = []
async def _cancelling_stream() -> AsyncIterator[Any]:
# one chunk then cancel — simulates a client disconnect mid-stream.
yield object() # caller's `async for` consumes this
raise asyncio.CancelledError()
async def _setup_stream(*_args: Any, **_kwargs: Any) -> AsyncIterator[Any]:
return _cancelling_stream()
with (
patch.object(executor, "CLIENTS", {"anthropic": object()}),
patch.object(executor, "backend_for_provider", return_value=object()),
patch.object(executor, "execute_stream", new=_setup_stream),
patch.object(
executor,
"stream_chunk_to_response_chunk",
side_effect=lambda chunk: chunk,
),
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
):
stream = await executor.honcho_llm_call_inner(
"anthropic",
"claude-sonnet-4-5",
"hello",
max_tokens=128,
plan=_make_plan(),
telemetry=None,
stream=True,
)
with pytest.raises(asyncio.CancelledError):
async for _ in stream:
pass
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, LLMCallCompletedEvent)
assert ev.outcome == "cancelled"
assert ev.was_stream is True
@pytest.mark.asyncio
async def test_stream_setup_failure_emits_and_propagates(self):
"""Stream-setup errors must propagate out of the AWAITED
`honcho_llm_call_inner` call (not deferred until first iteration),
so the outer retry wrapper in tool_loop.stream_final_response sees
them. Regression check for the bug where `_stream()` returned a
generator without awaiting `execute_stream`, hiding setup failures
from tenacity.
"""
from src.llm import executor
emitted: list[BaseEvent] = []
async def _setup_explodes(*_args: Any, **_kwargs: Any) -> Any:
raise RuntimeError("rate limited")
with (
patch.object(executor, "CLIENTS", {"anthropic": object()}),
patch.object(executor, "backend_for_provider", return_value=object()),
patch.object(executor, "execute_stream", new=_setup_explodes),
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
pytest.raises(RuntimeError, match="rate limited"),
):
# The await itself must raise — that's how tenacity sees it.
await executor.honcho_llm_call_inner(
"anthropic",
"claude-sonnet-4-5",
"hello",
max_tokens=128,
plan=_make_plan(),
telemetry=None,
stream=True,
)
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, LLMCallCompletedEvent)
assert ev.outcome == "error"
assert ev.was_stream is True
assert ev.error_class == "RuntimeError"
@pytest.mark.asyncio
async def test_error_path_still_emits_via_finally(self):
from src.llm import executor
emitted: list[BaseEvent] = []
async def _boom(*_args: Any, **_kwargs: Any) -> Any:
raise RuntimeError("backend exploded")
with (
patch.object(executor, "CLIENTS", {"anthropic": object()}),
patch.object(
executor,
"backend_for_provider",
return_value=object(),
),
patch.object(executor, "execute_completion", new=_boom),
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
pytest.raises(RuntimeError),
):
await executor.honcho_llm_call_inner(
"anthropic",
"claude-sonnet-4-5",
"hello",
max_tokens=128,
plan=_make_plan(attempt=3, retry_attempts=3),
telemetry=None,
)
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, LLMCallCompletedEvent)
assert ev.outcome == "error"
assert ev.is_final_attempt is True
assert ev.error_class == "RuntimeError"
def test_ground_truth_event_skips_sampler():
"""DialecticCompletedEvent declares _volume_class='ground_truth' (default
on BaseEvent) so the sampler should never gate it, even at rate 0.0.
Drives the real emitter under a zero-rate config and asserts the
ground-truth event still lands in the buffer — the actual bypass
behavior in emit() — not just the helper-function semantics in
_should_sample. Regression guard for a future refactor that pushes
ground_truth events through the sampling decision.
"""
from src.telemetry.emitter import TelemetryEmitter, _should_sample
# The volume_class declaration is the gate emit() consults.
assert DialecticCompletedEvent.volume_class() == "ground_truth"
event = DialecticCompletedEvent(
run_id="r",
workspace_name="ws",
peer_name="p",
reasoning_level="medium",
total_duration_ms=10.0,
input_tokens=1,
output_tokens=1,
)
# Sampler is deterministic 1.0 → True for any event (sanity).
assert _should_sample(event, 1.0) is True
# Drive the emitter directly under rate=0.0. enabled=True requires a
# non-None endpoint; we use a placeholder URL — the emitter buffers but
# we never flush, so no HTTP traffic is generated. The buffer growing
# proves the ground_truth event bypassed the sampler.
emitter = TelemetryEmitter(endpoint="http://test/events", enabled=True)
with patch("src.config.settings") as mock_settings:
mock_settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE = 0.0
mock_settings.TELEMETRY.NAMESPACE = "test"
emitter.emit(event)
assert emitter.buffer_size == 1 # ground_truth survived the sampler
# Sanity check: a high-volume event under rate=0.0 DOES get dropped,
# proving the test setup actually exercises the sampling code path.
from src.telemetry.events import LLMCallCompletedEvent
sampled_event = LLMCallCompletedEvent(
transport="anthropic",
model="m",
effective_max_output_tokens=1,
finish_reason="stop",
outcome="success",
is_final_attempt=True,
attempt=1,
retry_attempts=1,
was_fallback=False,
duration_ms=1.0,
has_tools=False,
was_stream=False,
)
assert sampled_event.volume_class() == "high_volume"
with patch("src.config.settings") as mock_settings:
mock_settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE = 0.0
mock_settings.TELEMETRY.NAMESPACE = "test"
emitter.emit(sampled_event)
# Buffer still 1 — high_volume event was sampled out.
assert emitter.buffer_size == 1
class TestStreamFinalResponseRetryAttempt:
"""`stream_final_response` (src/llm/tool_loop.py) must bump the
per-attempt index on the plan it passes to `honcho_llm_call_inner`.
Previously every retried stream-setup emit reported the same `attempt`
value because the pinned `winning_plan` was reused unchanged. Fix 13
plumbs a per-retry plan via `dataclasses.replace`.
"""
@pytest.mark.asyncio
async def test_attempt_index_bumps_across_retries(self):
from collections.abc import AsyncIterator
from src.llm import executor, tool_loop
emitted: list[BaseEvent] = []
# execute_stream raises on the first two calls, succeeds on the third.
call_count = 0
async def _flaky_setup(*_args: Any, **_kwargs: Any) -> AsyncIterator[Any]:
nonlocal call_count
call_count += 1
if call_count < 3:
raise RuntimeError("transient")
async def _ok_stream() -> AsyncIterator[Any]:
# Empty async generator — the unreachable yield is required
# to keep this function an async generator (no `async def
# ... -> AsyncIterator: return` shortcut exists in Python).
return
yield # pyright: ignore[reportUnreachable]
return _ok_stream()
# selected_config=None lets effective_config_for_call synthesize a
# minimal ModelConfig — avoids needing a real ModelConfig in this test.
winning_plan = AttemptPlan(
provider="anthropic",
model="claude-sonnet-4-5",
client=object(),
thinking_budget_tokens=None,
reasoning_effort=None,
selected_config=None,
attempt=1,
retry_attempts=3,
is_fallback=False,
)
with (
patch.object(executor, "CLIENTS", {"anthropic": object()}),
patch.object(executor, "backend_for_provider", return_value=object()),
patch.object(executor, "execute_stream", new=_flaky_setup),
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
):
stream = tool_loop.stream_final_response(
winning_plan=winning_plan,
prompt="hi",
max_tokens=64,
conversation_messages=[{"role": "user", "content": "x"}],
response_model=None,
json_mode=False,
temperature=None,
stop_seqs=None,
verbosity=None,
enable_retry=True,
retry_attempts=3,
before_retry_callback=lambda _r: None,
telemetry=None,
)
# Drain (empty) so the wrapper's finally fires for the success attempt.
async for _chunk in stream:
pass
# 3 emissions: attempts 1 & 2 errored, attempt 3 succeeded.
llm_events = [e for e in emitted if isinstance(e, LLMCallCompletedEvent)]
assert [e.attempt for e in llm_events] == [1, 2, 3]
# Final attempt flag: only True on the last retry (attempt 3 of 3).
assert [e.is_final_attempt for e in llm_events] == [False, False, True]
# First two errored, last succeeded.
assert [e.outcome for e in llm_events] == ["error", "error", "success"]
class TestStreamingResponseTokenWriteBack:
"""`StreamingResponseWithMetadata` must accumulate the final-stream's
output_tokens (reported in usage chunks by OpenAI/Anthropic) into its
`output_tokens` attribute as the stream drains, so DialecticCompletedEvent
sees tool-loop totals + final-stream totals — not tool-loop totals alone.
"""
@pytest.mark.asyncio
async def test_output_tokens_folds_in_final_stream_usage(self):
from src.llm.types import (
HonchoLLMCallStreamChunk,
StreamingResponseWithMetadata,
)
async def _fake_stream() -> Any:
# Content-only chunks, then a final usage chunk with cumulative
# output_tokens=137 — matches OpenAI's include_usage pattern.
yield HonchoLLMCallStreamChunk(content="hel", output_tokens=None)
yield HonchoLLMCallStreamChunk(content="lo", output_tokens=None)
yield HonchoLLMCallStreamChunk(content="", is_done=True, output_tokens=137)
wrapper = StreamingResponseWithMetadata(
stream=_fake_stream(),
tool_calls_made=[],
input_tokens=200,
output_tokens=50, # tool-loop running output total
cache_creation_input_tokens=0,
cache_read_input_tokens=0,
)
# Before drain, the wrapper holds only the tool-loop total.
assert wrapper.output_tokens == 50
chunks: list[HonchoLLMCallStreamChunk] = []
async for chunk in wrapper:
chunks.append(chunk)
# After drain, the final-stream's 137 output tokens fold in.
assert wrapper.output_tokens == 50 + 137
# And we yielded every chunk to the caller — the wrapper is a
# passthrough, not a sink.
assert len(chunks) == 3

197
tests/llm/test_tool_loop_truncation.py Normal file
View File

@ -0,0 +1,197 @@
# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false
"""Regression tests for `hit_input_token_cap` propagation through
`execute_tool_loop`.
The toolless path (`src/llm/api.py:325-340`) detects the cap hit up-front
by comparing input tokens against `max_input_tokens`. Before this fix,
the tool-loop path called `truncate_messages_to_fit` per iteration but
never propagated the flag — Dialectic (the main tool-loop consumer)
under-reported `hit_input_token_cap` on dialectic/representation events.
The rule is intentionally token-based, not message-count-based, so the
deriver's single-prompt path (where `truncate_messages_to_fit` keeps the
last unit even when oversized) still surfaces a real cap hit.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import patch
import pytest
from src.llm import tool_loop
from src.llm.runtime import AttemptPlan
from src.llm.tool_loop import execute_tool_loop
from src.llm.types import HonchoLLMCallResponse
def _make_plan() -> AttemptPlan:
# `selected_config=None` works for these tests since `_call_with_messages`
# passes it straight through to the mocked `honcho_llm_call_inner`.
return AttemptPlan(
provider="anthropic",
model="claude-sonnet-4-5",
client=object(),
thinking_budget_tokens=None,
reasoning_effort=None,
selected_config=None,
attempt=1,
retry_attempts=1,
is_fallback=False,
)
async def _terminating_call(*_args: Any, **_kwargs: Any) -> HonchoLLMCallResponse[Any]:
# No tool calls — execute_tool_loop terminates after iteration 1.
return HonchoLLMCallResponse(
content="done",
input_tokens=10,
output_tokens=5,
cache_creation_input_tokens=0,
cache_read_input_tokens=0,
finish_reasons=["stop"],
tool_calls_made=[],
)
@pytest.mark.asyncio
async def test_hit_input_token_cap_fires_when_input_exceeds_cap():
"""When the input message list exceeds `max_input_tokens` by token
count, the response carries `hit_input_token_cap=True`. Regression
check for the rule switch from message-count to token-based.
"""
# Pretend the conversation totals 200 tokens; cap is 100.
with (
patch.object(tool_loop, "honcho_llm_call_inner", new=_terminating_call),
patch("src.llm.conversation.count_message_tokens", return_value=200),
patch(
"src.llm.conversation.truncate_messages_to_fit",
side_effect=lambda msgs, _cap: msgs,
),
):
result = await execute_tool_loop(
prompt="hi",
max_tokens=64,
messages=[{"role": "user", "content": "huge"}],
tools=[
{
"name": "noop",
"description": "no-op",
"input_schema": {"type": "object"},
}
],
tool_choice="auto",
tool_executor=lambda _name, _input: "",
max_tool_iterations=5,
response_model=None,
json_mode=False,
temperature=None,
stop_seqs=None,
verbosity=None,
enable_retry=False,
retry_attempts=1,
max_input_tokens=100,
get_attempt_plan=_make_plan,
before_retry_callback=lambda _r: None,
stream_final=False,
telemetry=None,
)
assert isinstance(result, HonchoLLMCallResponse)
assert result.hit_input_token_cap is True
@pytest.mark.asyncio
async def test_hit_input_token_cap_false_when_under_cap():
"""Input tokens under cap → flag stays False, no false positive."""
with (
patch.object(tool_loop, "honcho_llm_call_inner", new=_terminating_call),
patch("src.llm.conversation.count_message_tokens", return_value=50),
patch(
"src.llm.conversation.truncate_messages_to_fit",
side_effect=lambda msgs, _cap: msgs,
),
):
result = await execute_tool_loop(
prompt="hi",
max_tokens=64,
messages=[{"role": "user", "content": "small"}],
tools=[
{
"name": "noop",
"description": "no-op",
"input_schema": {"type": "object"},
}
],
tool_choice="auto",
tool_executor=lambda _name, _input: "",
max_tool_iterations=5,
response_model=None,
json_mode=False,
temperature=None,
stop_seqs=None,
verbosity=None,
enable_retry=False,
retry_attempts=1,
max_input_tokens=100,
get_attempt_plan=_make_plan,
before_retry_callback=lambda _r: None,
stream_final=False,
telemetry=None,
)
assert isinstance(result, HonchoLLMCallResponse)
assert result.hit_input_token_cap is False
@pytest.mark.asyncio
async def test_hit_input_token_cap_fires_even_when_truncate_cant_shrink():
"""Critical regression: the single-message over-cap case (deriver's
prompt-only call) used to silently return hit=False because
`truncate_messages_to_fit` keeps the last unit even when oversized,
making the old message-count-based check stay at False. The new
token-based rule correctly catches this case.
"""
with (
patch.object(tool_loop, "honcho_llm_call_inner", new=_terminating_call),
patch("src.llm.conversation.count_message_tokens", return_value=99_999),
# Truncate is a no-op (matches real behavior for single-message inputs).
patch(
"src.llm.conversation.truncate_messages_to_fit",
side_effect=lambda msgs, _cap: msgs,
),
):
result = await execute_tool_loop(
prompt="hi",
max_tokens=64,
messages=[{"role": "user", "content": "x" * 1_000_000}],
tools=[
{
"name": "noop",
"description": "no-op",
"input_schema": {"type": "object"},
}
],
tool_choice="auto",
tool_executor=lambda _name, _input: "",
max_tool_iterations=5,
response_model=None,
json_mode=False,
temperature=None,
stop_seqs=None,
verbosity=None,
enable_retry=False,
retry_attempts=1,
max_input_tokens=1_000,
get_attempt_plan=_make_plan,
before_retry_callback=lambda _r: None,
stream_final=False,
telemetry=None,
)
assert isinstance(result, HonchoLLMCallResponse)
assert result.hit_input_token_cap is True

298
tests/routes/test_peers.py
View File

@ -1,3 +1,4 @@
import datetime
from typing import Any
import pytest
@ -5,7 +6,7 @@ from fastapi.testclient import TestClient
from nanoid import generate as generate_nanoid
from sqlalchemy.ext.asyncio import AsyncSession
from src import crud
from src import crud, models
from src.models import Peer, Workspace
@ -169,6 +170,148 @@ def test_get_peers_with_null_filter(
assert isinstance(data["items"], list)
def test_get_peers_with_reverse(
client: TestClient, sample_data: tuple[Workspace, Peer]
):
"""Test peer listing with reverse creation-time ordering."""
test_workspace, _ = sample_data
reverse_group = f"reverse-peers-{generate_nanoid()}"
first_name = f"reverse-peer-a-{generate_nanoid()}"
second_name = f"reverse-peer-b-{generate_nanoid()}"
first_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers",
json={"name": first_name, "metadata": {"reverse_group": reverse_group}},
)
assert first_response.status_code in [200, 201]
second_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers",
json={"name": second_name, "metadata": {"reverse_group": reverse_group}},
)
assert second_response.status_code in [200, 201]
normal_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/list",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert reverse_response.status_code == 200
assert [item["id"] for item in normal_response.json()["items"]] == [
first_name,
second_name,
]
assert [item["id"] for item in reverse_response.json()["items"]] == [
second_name,
first_name,
]
@pytest.mark.asyncio
async def test_get_peers_reverse_uses_id_tiebreaker(
client: TestClient,
db_session: AsyncSession,
sample_data: tuple[Workspace, Peer],
):
"""Peers with identical created_at fall back to ordering by id (nanoid PK)."""
test_workspace, _ = sample_data
reverse_group = f"tiebreaker-peers-{generate_nanoid()}"
shared_created_at = datetime.datetime(
2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc
)
low_id = "A" * 21
high_id = "z" * 21
low_name = f"tie-low-peer-{generate_nanoid()}"
high_name = f"tie-high-peer-{generate_nanoid()}"
db_session.add(
models.Peer(
id=low_id,
name=low_name,
workspace_name=test_workspace.name,
created_at=shared_created_at,
h_metadata={"reverse_group": reverse_group},
)
)
db_session.add(
models.Peer(
id=high_id,
name=high_name,
workspace_name=test_workspace.name,
created_at=shared_created_at,
h_metadata={"reverse_group": reverse_group},
)
)
await db_session.commit()
normal_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/list",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert reverse_response.status_code == 200
# When created_at ties, ordering falls back to the nanoid id: low_id < high_id
# lexicographically, so low sorts first ascending and last descending.
assert [item["id"] for item in normal_response.json()["items"]] == [
low_name,
high_name,
]
assert [item["id"] for item in reverse_response.json()["items"]] == [
high_name,
low_name,
]
def test_get_peers_reverse_with_pagination(
client: TestClient, sample_data: tuple[Workspace, Peer]
):
"""Paged reverse listing returns newest-first across consecutive pages."""
test_workspace, _ = sample_data
reverse_group = f"paged-reverse-peers-{generate_nanoid()}"
peer_names = [f"paged-reverse-peer-{i}-{generate_nanoid()}" for i in range(3)]
for peer_name in peer_names:
response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers",
json={"name": peer_name, "metadata": {"reverse_group": reverse_group}},
)
assert response.status_code in [200, 201]
page_one = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true&page=1&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_one.status_code == 200
page_two = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true&page=2&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_two.status_code == 200
page_three = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/list?reverse=true&page=3&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_three.status_code == 200
assert page_one.json()["total"] == 3
assert [item["id"] for item in page_one.json()["items"]] == [peer_names[2]]
assert [item["id"] for item in page_two.json()["items"]] == [peer_names[1]]
assert [item["id"] for item in page_three.json()["items"]] == [peer_names[0]]
def test_update_peer(client: TestClient, sample_data: tuple[Workspace, Peer]):
test_workspace, test_peer = sample_data
response = client.put(
@ -308,6 +451,159 @@ def test_get_sessions_for_peer_with_empty_filter(
assert isinstance(data["items"], list)
def test_get_sessions_for_peer_with_reverse(
client: TestClient, sample_data: tuple[Workspace, Peer]
):
"""Test peer session listing with reverse creation-time ordering."""
test_workspace, test_peer = sample_data
reverse_group = f"reverse-peer-sessions-{generate_nanoid()}"
first_session = f"reverse-peer-session-a-{generate_nanoid()}"
second_session = f"reverse-peer-session-b-{generate_nanoid()}"
first_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions",
json={
"id": first_session,
"peer_names": {test_peer.name: {}},
"metadata": {"reverse_group": reverse_group},
},
)
assert first_response.status_code in [200, 201]
second_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions",
json={
"id": second_session,
"peer_names": {test_peer.name: {}},
"metadata": {"reverse_group": reverse_group},
},
)
assert second_response.status_code in [200, 201]
normal_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert reverse_response.status_code == 200
assert [item["id"] for item in normal_response.json()["items"]] == [
first_session,
second_session,
]
assert [item["id"] for item in reverse_response.json()["items"]] == [
second_session,
first_session,
]
@pytest.mark.asyncio
async def test_get_sessions_for_peer_reverse_uses_id_tiebreaker(
client: TestClient,
db_session: AsyncSession,
sample_data: tuple[Workspace, Peer],
):
"""Peer-scoped sessions with identical created_at fall back to ordering by id."""
test_workspace, test_peer = sample_data
reverse_group = f"tiebreaker-peer-sessions-{generate_nanoid()}"
shared_created_at = datetime.datetime(
2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc
)
low_id = "A" * 21
high_id = "z" * 21
low_name = f"tie-low-peer-session-{generate_nanoid()}"
high_name = f"tie-high-peer-session-{generate_nanoid()}"
for session_id, session_name in ((low_id, low_name), (high_id, high_name)):
db_session.add(
models.Session(
id=session_id,
name=session_name,
workspace_name=test_workspace.name,
created_at=shared_created_at,
h_metadata={"reverse_group": reverse_group},
)
)
db_session.add(
models.SessionPeer(
workspace_name=test_workspace.name,
session_name=session_name,
peer_name=test_peer.name,
)
)
await db_session.commit()
normal_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert reverse_response.status_code == 200
assert [item["id"] for item in normal_response.json()["items"]] == [
low_name,
high_name,
]
assert [item["id"] for item in reverse_response.json()["items"]] == [
high_name,
low_name,
]
def test_get_sessions_for_peer_reverse_with_pagination(
client: TestClient, sample_data: tuple[Workspace, Peer]
):
"""Paged reverse listing of a peer's sessions returns newest-first across pages."""
test_workspace, test_peer = sample_data
reverse_group = f"paged-reverse-peer-sessions-{generate_nanoid()}"
session_names = [
f"paged-reverse-peer-session-{i}-{generate_nanoid()}" for i in range(3)
]
for session_name in session_names:
response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions",
json={
"id": session_name,
"peer_names": {test_peer.name: {}},
"metadata": {"reverse_group": reverse_group},
},
)
assert response.status_code in [200, 201]
page_one = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true&page=1&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_one.status_code == 200
page_two = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true&page=2&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_two.status_code == 200
page_three = client.post(
f"/v3/workspaces/{test_workspace.name}/peers/{test_peer.name}/sessions?reverse=true&page=3&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_three.status_code == 200
assert page_one.json()["total"] == 3
assert [item["id"] for item in page_one.json()["items"]] == [session_names[2]]
assert [item["id"] for item in page_two.json()["items"]] == [session_names[1]]
assert [item["id"] for item in page_three.json()["items"]] == [session_names[0]]
def test_chat(
client: TestClient,
sample_data: tuple[Workspace, Peer],

158
tests/routes/test_sessions.py
View File

@ -1,8 +1,12 @@
import datetime
from typing import Any
import pytest
from fastapi.testclient import TestClient
from nanoid import generate as generate_nanoid
from sqlalchemy.ext.asyncio import AsyncSession
from src import models
from src.models import Peer, Workspace
@ -223,6 +227,160 @@ def test_get_sessions_with_empty_filter(
assert isinstance(data["items"], list)
def test_get_sessions_with_reverse(
client: TestClient, sample_data: tuple[Workspace, Peer]
):
"""Test session listing with reverse creation-time ordering."""
test_workspace, test_peer = sample_data
reverse_group = f"reverse-sessions-{generate_nanoid()}"
first_session = f"reverse-session-a-{generate_nanoid()}"
second_session = f"reverse-session-b-{generate_nanoid()}"
first_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions",
json={
"id": first_session,
"peer_names": {test_peer.name: {}},
"metadata": {"reverse_group": reverse_group},
},
)
assert first_response.status_code in [200, 201]
second_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions",
json={
"id": second_session,
"peer_names": {test_peer.name: {}},
"metadata": {"reverse_group": reverse_group},
},
)
assert second_response.status_code in [200, 201]
normal_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions/list",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert reverse_response.status_code == 200
assert [item["id"] for item in normal_response.json()["items"]] == [
first_session,
second_session,
]
assert [item["id"] for item in reverse_response.json()["items"]] == [
second_session,
first_session,
]
@pytest.mark.asyncio
async def test_get_sessions_reverse_uses_id_tiebreaker(
client: TestClient,
db_session: AsyncSession,
sample_data: tuple[Workspace, Peer],
):
"""Sessions with identical created_at fall back to ordering by id (nanoid PK)."""
test_workspace, _ = sample_data
reverse_group = f"tiebreaker-sessions-{generate_nanoid()}"
shared_created_at = datetime.datetime(
2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc
)
low_id = "A" * 21
high_id = "z" * 21
low_name = f"tie-low-{generate_nanoid()}"
high_name = f"tie-high-{generate_nanoid()}"
db_session.add(
models.Session(
id=low_id,
name=low_name,
workspace_name=test_workspace.name,
created_at=shared_created_at,
h_metadata={"reverse_group": reverse_group},
)
)
db_session.add(
models.Session(
id=high_id,
name=high_name,
workspace_name=test_workspace.name,
created_at=shared_created_at,
h_metadata={"reverse_group": reverse_group},
)
)
await db_session.commit()
normal_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions/list",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert reverse_response.status_code == 200
# When created_at ties, ordering falls back to the nanoid id: low_id < high_id
# lexicographically, so low sorts first ascending and last descending.
assert [item["id"] for item in normal_response.json()["items"]] == [
low_name,
high_name,
]
assert [item["id"] for item in reverse_response.json()["items"]] == [
high_name,
low_name,
]
def test_get_sessions_reverse_with_pagination(
client: TestClient, sample_data: tuple[Workspace, Peer]
):
"""Paged reverse listing returns newest-first across consecutive pages."""
test_workspace, test_peer = sample_data
reverse_group = f"paged-reverse-sessions-{generate_nanoid()}"
session_names = [f"paged-reverse-session-{i}-{generate_nanoid()}" for i in range(3)]
for session_name in session_names:
response = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions",
json={
"id": session_name,
"peer_names": {test_peer.name: {}},
"metadata": {"reverse_group": reverse_group},
},
)
assert response.status_code in [200, 201]
page_one = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true&page=1&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_one.status_code == 200
page_two = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true&page=2&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_two.status_code == 200
page_three = client.post(
f"/v3/workspaces/{test_workspace.name}/sessions/list?reverse=true&page=3&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_three.status_code == 200
assert page_one.json()["total"] == 3
assert [item["id"] for item in page_one.json()["items"]] == [session_names[2]]
assert [item["id"] for item in page_two.json()["items"]] == [session_names[1]]
assert [item["id"] for item in page_three.json()["items"]] == [session_names[0]]
def test_update_delete_metadata(
client: TestClient, sample_data: tuple[Workspace, Peer]
):

130
tests/routes/test_workspaces.py
View File

@ -1,3 +1,4 @@
import datetime
from typing import Any
from unittest.mock import AsyncMock, patch
@ -125,6 +126,135 @@ async def test_get_all_workspaces_with_null_filter(client: TestClient):
assert isinstance(data["items"], list)
@pytest.mark.asyncio
async def test_get_all_workspaces_with_reverse(client: TestClient):
"""Test workspace listing with reverse creation-time ordering."""
first_name = f"reverse-workspace-{generate_nanoid()}"
second_name = f"reverse-workspace-{generate_nanoid()}"
first_response = client.post(
"/v3/workspaces",
json={"name": first_name, "metadata": {"reverse_group": first_name}},
)
assert first_response.status_code in [200, 201]
second_response = client.post(
"/v3/workspaces",
json={"name": second_name, "metadata": {"reverse_group": first_name}},
)
assert second_response.status_code in [200, 201]
normal_response = client.post(
"/v3/workspaces/list",
json={"filters": {"metadata": {"reverse_group": first_name}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
"/v3/workspaces/list?reverse=true",
json={"filters": {"metadata": {"reverse_group": first_name}}},
)
assert reverse_response.status_code == 200
assert [item["id"] for item in normal_response.json()["items"]] == [
first_name,
second_name,
]
assert [item["id"] for item in reverse_response.json()["items"]] == [
second_name,
first_name,
]
@pytest.mark.asyncio
async def test_get_all_workspaces_reverse_uses_id_tiebreaker(
client: TestClient, db_session: AsyncSession
):
"""Workspaces with identical created_at fall back to ordering by id (nanoid PK)."""
reverse_group = f"tiebreaker-{generate_nanoid()}"
shared_created_at = datetime.datetime(
2026, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc
)
low_id = "A" * 21
high_id = "z" * 21
low_name = f"tie-low-{generate_nanoid()}"
high_name = f"tie-high-{generate_nanoid()}"
db_session.add(
models.Workspace(
id=low_id,
name=low_name,
created_at=shared_created_at,
h_metadata={"reverse_group": reverse_group},
)
)
db_session.add(
models.Workspace(
id=high_id,
name=high_name,
created_at=shared_created_at,
h_metadata={"reverse_group": reverse_group},
)
)
await db_session.commit()
normal_response = client.post(
"/v3/workspaces/list",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert normal_response.status_code == 200
reverse_response = client.post(
"/v3/workspaces/list?reverse=true",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert reverse_response.status_code == 200
normal_items = [item["id"] for item in normal_response.json()["items"]]
reverse_items = [item["id"] for item in reverse_response.json()["items"]]
# When created_at ties, ordering falls back to the nanoid id: low_id < high_id
# lexicographically, so the workspace with id="AAA..." sorts first ascending.
assert normal_items == [low_name, high_name]
assert reverse_items == [high_name, low_name]
@pytest.mark.asyncio
async def test_get_all_workspaces_reverse_with_pagination(client: TestClient):
"""Paged reverse listing returns newest-first across consecutive pages."""
reverse_group = f"paged-reverse-{generate_nanoid()}"
names = [f"paged-reverse-{i}-{generate_nanoid()}" for i in range(3)]
for name in names:
response = client.post(
"/v3/workspaces",
json={"name": name, "metadata": {"reverse_group": reverse_group}},
)
assert response.status_code in [200, 201]
page_one = client.post(
"/v3/workspaces/list?reverse=true&page=1&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_one.status_code == 200
page_two = client.post(
"/v3/workspaces/list?reverse=true&page=2&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_two.status_code == 200
page_three = client.post(
"/v3/workspaces/list?reverse=true&page=3&size=1",
json={"filters": {"metadata": {"reverse_group": reverse_group}}},
)
assert page_three.status_code == 200
assert page_one.json()["total"] == 3
assert [item["id"] for item in page_one.json()["items"]] == [names[2]]
assert [item["id"] for item in page_two.json()["items"]] == [names[1]]
assert [item["id"] for item in page_three.json()["items"]] == [names[0]]
def test_update_workspace(client: TestClient, sample_data: tuple[Workspace, Peer]):
test_workspace, _ = sample_data
_new_name = str(generate_nanoid())

2
tests/scripts/test_configure_embeddings.py
View File

@ -1,4 +1,4 @@
"""Phase 3: configure_embeddings script tests."""
"""Tests for the configure_embeddings script."""
from __future__ import annotations

49
tests/sdk/test_session.py
View File

@ -200,6 +200,55 @@ async def test_session_peer_config(client_fixture: tuple[Honcho, str]):
assert retrieved_config.observe_others
@pytest.mark.asyncio
async def test_session_create_with_peers(client_fixture: tuple[Honcho, str]):
"""
Tests creating a session with peers attached in a single call.
"""
honcho_client, client_type = client_fixture
if client_type == "async":
session = await honcho_client.aio.session(
id="test-session-create-peers-async",
peers=["create-peer-async-a", "create-peer-async-b"],
)
assert isinstance(session, Session)
peers = await session.aio.peers()
peer_ids = {p.id for p in peers}
assert "create-peer-async-a" in peer_ids
assert "create-peer-async-b" in peer_ids
config = SessionPeerConfig(observe_me=True, observe_others=False)
peer = await honcho_client.aio.peer(id="create-peer-async-config")
session_with_config = await honcho_client.aio.session(
id="test-session-create-peers-config-async",
peers=[(peer, config)],
)
retrieved = await session_with_config.aio.get_peer_configuration(peer)
assert retrieved.observe_me is True
assert retrieved.observe_others is False
else:
session = honcho_client.session(
id="test-session-create-peers",
peers=["create-peer-a", "create-peer-b"],
)
assert isinstance(session, Session)
peers = session.peers()
peer_ids = {p.id for p in peers}
assert "create-peer-a" in peer_ids
assert "create-peer-b" in peer_ids
config = SessionPeerConfig(observe_me=True, observe_others=False)
peer = honcho_client.peer(id="create-peer-config")
session_with_config = honcho_client.session(
id="test-session-create-peers-config",
peers=[(peer, config)],
)
retrieved = session_with_config.get_peer_configuration(peer)
assert retrieved.observe_me is True
assert retrieved.observe_others is False
@pytest.mark.asyncio
async def test_session_messages(client_fixture: tuple[Honcho, str]):
"""

6
tests/startup/test_embedding_validator.py
View File

@ -1,4 +1,4 @@
"""Phase 2: startup embedding-schema validator + VECTOR_STORE_DIMENSIONS deprecation."""
"""Startup embedding-schema validator + VECTOR_STORE_DIMENSIONS deprecation."""
from __future__ import annotations
@ -178,9 +178,9 @@ def test_vector_store_dimensions_explicit_set_warns(
def test_non_1536_pgvector_without_migrated_no_longer_raises_at_config_time() -> None:
"""Phase 2 removed the dim-vs-MIGRATED guard. Constructing AppSettings
"""The dim-vs-MIGRATED guard has been removed. Constructing AppSettings
with non-1536 + default pgvector + MIGRATED=false should now succeed
(the runtime schema validator at startup is the new safety net)."""
(the runtime schema validator at startup is the safety net)."""
env = {
**os.environ,
"PYTHON_DOTENV_DISABLED": "1",

126
tests/telemetry/conftest.py
View File

@ -2,7 +2,7 @@
"""Fixtures for telemetry unit tests.
This module provides:
- Sample event fixtures for all 12 event types
- Sample event fixtures for all telemetry event types
- Mock settings fixtures for controlling telemetry configuration
- Mock HTTP client fixtures for testing the emitter without network calls
"""
@ -19,10 +19,16 @@ from src.telemetry.events.agent import (
AgentToolPeerCardUpdatedEvent,
AgentToolSummaryCreatedEvent,
)
from src.telemetry.events.api import (
FileUploadedEvent,
GetContextEvent,
MessageCreatedEvent,
)
from src.telemetry.events.base import BaseEvent
from src.telemetry.events.deletion import DeletionCompletedEvent
from src.telemetry.events.dialectic import DialecticCompletedEvent
from src.telemetry.events.dream import DreamRunEvent, DreamSpecialistEvent
from src.telemetry.events.llm import CallPurpose, LLMCallCompletedEvent
from src.telemetry.events.reconciliation import (
CleanupStaleItemsCompletedEvent,
SyncVectorsCompletedEvent,
@ -64,7 +70,101 @@ def sample_representation_event(
llm_call_ms=1200.0,
total_duration_ms=1300.0,
input_tokens=5000,
total_input_tokens=7500,
output_tokens=500,
# additive fields
queued_message_count=3,
prompt_message_count=10,
prompt_message_tokens=7000,
extra_context_message_count=7,
extra_context_tokens=2000,
prompt_scaffold_tokens=500,
batch_max_tokens=20_000,
max_input_tokens=23_000,
was_flush_enabled=False,
hit_batch_token_cap=False,
hit_input_token_cap=False,
observer_count=1,
)
@pytest.fixture
def sample_message_created_event(fixed_timestamp: datetime) -> MessageCreatedEvent:
"""Create a sample MessageCreatedEvent for testing."""
return MessageCreatedEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
session_name="test_session",
message_count=2,
total_tokens=250,
last_message_id="msg_abc123_fixture_____",
)
@pytest.fixture
def sample_file_uploaded_event(fixed_timestamp: datetime) -> FileUploadedEvent:
"""Create a sample FileUploadedEvent for testing."""
return FileUploadedEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
session_name="test_session",
peer_name="user_peer",
file_id="file_123",
filename="notes.txt",
content_type="text/plain",
file_size_bytes=1024,
message_count=2,
total_tokens=250,
)
@pytest.fixture
def sample_llm_call_event(fixed_timestamp: datetime) -> LLMCallCompletedEvent:
"""Sample LLMCallCompletedEvent ()."""
return LLMCallCompletedEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
call_purpose=CallPurpose.DIALECTIC_ANSWER,
parent_category="dialectic",
transport="anthropic",
provider_label=None,
model="claude-sonnet-4-5",
effective_max_output_tokens=4096,
provider_input_tokens=1234,
provider_output_tokens=567,
cache_read_tokens=100,
cache_creation_tokens=50,
finish_reason="stop",
outcome="success",
is_final_attempt=False,
attempt=1,
retry_attempts=3,
was_fallback=False,
duration_ms=1100.5,
has_tools=True,
tool_call_count=2,
run_id="abc12345",
iteration=1,
)
@pytest.fixture
def sample_get_context_event(fixed_timestamp: datetime) -> GetContextEvent:
"""Create a sample GetContextEvent for testing."""
return GetContextEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
context_scope="session",
session_name="test_session",
tokens_requested=4000,
message_count=10,
has_summary=True,
has_representation=False,
has_peer_card=False,
search_query_provided=False,
include_summary=True,
peer_perspective_provided=False,
total_duration_ms=25.0,
)
@ -87,6 +187,13 @@ def sample_dream_run_event(fixed_timestamp: datetime) -> DreamRunEvent:
total_input_tokens=25000,
total_output_tokens=3000,
total_duration_ms=45000.0,
# additive fields
dream_type="omni",
enabled_types_count=1,
trigger_reason="document_threshold",
delay_reason="idle_timeout",
documents_since_last_dream_at_schedule=55,
document_threshold=50,
)
@ -106,6 +213,11 @@ def sample_dream_specialist_event(fixed_timestamp: datetime) -> DreamSpecialistE
output_tokens=2000,
duration_ms=25000.0,
success=True,
# additive rollups
created_observation_count=7,
deleted_observation_count=2,
peer_card_updated=True,
search_tool_calls_count=4,
)
@ -225,6 +337,10 @@ def sample_summary_created_event(
summary_type="short",
input_tokens=4000,
output_tokens=300,
# additive token breakdown
previous_summary_tokens=200,
message_tokens=3500,
prompt_scaffold_tokens=300,
)
@ -276,6 +392,9 @@ def sample_cleanup_event(fixed_timestamp: datetime) -> CleanupStaleItemsComplete
@pytest.fixture
def all_sample_events(
sample_representation_event: RepresentationCompletedEvent,
sample_message_created_event: MessageCreatedEvent,
sample_file_uploaded_event: FileUploadedEvent,
sample_get_context_event: GetContextEvent,
sample_dream_run_event: DreamRunEvent,
sample_dream_specialist_event: DreamSpecialistEvent,
sample_dialectic_event: DialecticCompletedEvent,
@ -287,10 +406,14 @@ def all_sample_events(
sample_deletion_event: DeletionCompletedEvent,
sample_sync_vectors_event: SyncVectorsCompletedEvent,
sample_cleanup_event: CleanupStaleItemsCompletedEvent,
sample_llm_call_event: LLMCallCompletedEvent,
) -> list[BaseEvent]:
"""Return all sample events as a list for parametrized tests."""
return [
sample_representation_event,
sample_message_created_event,
sample_file_uploaded_event,
sample_get_context_event,
sample_dream_run_event,
sample_dream_specialist_event,
sample_dialectic_event,
@ -302,6 +425,7 @@ def all_sample_events(
sample_deletion_event,
sample_sync_vectors_event,
sample_cleanup_event,
sample_llm_call_event,
]

333
tests/telemetry/test_embedding_call_event.py Normal file
View File

@ -0,0 +1,333 @@
# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false
"""tests for EmbeddingCallCompletedEvent emission.
Targets:
- New `EmbeddingCallCompletedEvent` (embedding.call.completed) at schema v1
with high_volume sampling.
- `EmbeddingCallPurpose` closed enum.
- The `embedding_call_purpose` context manager round-trips the slug onto
the event via the ContextVar, without changing call signatures.
- `_emit_embedding_call` wrapper emits on success AND on exception, and
propagates the underlying error unchanged.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import patch
import pytest
from src.embedding_client import _emit_embedding_call
from src.telemetry.events import (
BaseEvent,
EmbeddingCallCompletedEvent,
EmbeddingCallPurpose,
)
from src.utils.types import (
embedding_call_purpose,
get_embedding_call_purpose,
)
class TestEventShape:
def test_event_type_and_version(self):
assert EmbeddingCallCompletedEvent.event_type() == "embedding.call.completed"
assert EmbeddingCallCompletedEvent.schema_version() == 1
assert EmbeddingCallCompletedEvent.category() == "llm"
def test_volume_class_is_high_volume(self):
"""event participates in HIGH_VOLUME_SAMPLE_RATE alongside
llm.call.completed — search-heavy paths can flood the buffer."""
assert EmbeddingCallCompletedEvent.volume_class() == "high_volume"
def test_resource_id_disambiguates(self):
ev_a = EmbeddingCallCompletedEvent(
provider="openai",
model="text-embedding-3-small",
input_count=5,
duration_ms=10.0,
outcome="success",
call_purpose=EmbeddingCallPurpose.SEARCH_MEMORY,
)
ev_b = EmbeddingCallCompletedEvent(
provider="openai",
model="text-embedding-3-small",
input_count=5,
duration_ms=10.0,
outcome="success",
call_purpose=EmbeddingCallPurpose.SEARCH_MESSAGES,
)
# Different purpose → different resource id.
assert ev_a.get_resource_id() != ev_b.get_resource_id()
class TestEmbeddingCallPurposeEnum:
def test_known_values(self):
# The closed taxonomy. Adding a value here requires a coordinated
# update with downstream analytics that filter on call_purpose.
assert EmbeddingCallPurpose.SEARCH_MEMORY.value == "search_memory"
assert EmbeddingCallPurpose.SEARCH_MESSAGES.value == "search_messages"
assert EmbeddingCallPurpose.CREATE_OBSERVATIONS.value == "create_observations"
assert EmbeddingCallPurpose.VECTOR_SYNC.value == "vector_sync"
assert EmbeddingCallPurpose.SUMMARY.value == "summary"
assert EmbeddingCallPurpose.MESSAGE_CREATE.value == "message_create"
class TestContextManager:
def test_sets_and_clears(self):
assert get_embedding_call_purpose() is None
with embedding_call_purpose("search_memory"):
assert get_embedding_call_purpose() == "search_memory"
# ContextVar must reset on exit.
assert get_embedding_call_purpose() is None
def test_nested_context_managers_restore_outer(self):
"""Nested usage shouldn't lose the outer purpose on inner-exit."""
with embedding_call_purpose("search_memory"):
assert get_embedding_call_purpose() == "search_memory"
with embedding_call_purpose("create_observations"):
assert get_embedding_call_purpose() == "create_observations"
# After inner exits, outer purpose must be restored — not None.
assert get_embedding_call_purpose() == "search_memory"
def test_exception_in_block_still_resets(self):
try:
with embedding_call_purpose("search_memory"):
raise RuntimeError("boom")
except RuntimeError:
pass
assert get_embedding_call_purpose() is None
class TestEmitEmbeddingCallWrapper:
@pytest.mark.asyncio
async def test_success_emits_event_with_purpose(self):
emitted: list[BaseEvent] = []
async def _fake_call() -> list[float]:
return [0.1, 0.2, 0.3]
with (
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
embedding_call_purpose("search_memory"),
):
result = await _emit_embedding_call(
provider="openai",
model="text-embedding-3-small",
texts=["query"],
input_tokens_estimate=3,
fn=_fake_call,
)
assert result == [0.1, 0.2, 0.3]
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, EmbeddingCallCompletedEvent)
assert ev.outcome == "success"
assert ev.provider == "openai"
assert ev.model == "text-embedding-3-small"
assert ev.input_count == 1
assert ev.input_tokens_estimate == 3
assert ev.call_purpose == EmbeddingCallPurpose.SEARCH_MEMORY
assert ev.error_class is None
@pytest.mark.asyncio
async def test_exception_emits_error_event_and_propagates(self):
emitted: list[BaseEvent] = []
async def _boom() -> list[float]:
raise RuntimeError("provider down")
with (
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
pytest.raises(RuntimeError, match="provider down"),
):
await _emit_embedding_call(
provider="gemini",
model="text-embedding-005",
texts=["a", "b"],
input_tokens_estimate=10,
fn=_boom,
)
# Event emitted on the error path too (try/finally).
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, EmbeddingCallCompletedEvent)
assert ev.outcome == "error"
assert ev.error_class == "RuntimeError"
assert ev.input_count == 2
@pytest.mark.asyncio
async def test_cancellation_emits_cancelled_outcome(self):
"""asyncio.CancelledError surfaces as outcome='cancelled', not 'error'
— client disconnects / shutdowns must not pollute error rates."""
import asyncio
emitted: list[BaseEvent] = []
async def _cancel() -> list[float]:
raise asyncio.CancelledError()
with (
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
pytest.raises(asyncio.CancelledError),
):
await _emit_embedding_call(
provider="openai",
model="text-embedding-3-small",
texts=["a"],
input_tokens_estimate=4,
fn=_cancel,
)
assert len(emitted) == 1
ev = emitted[0]
assert isinstance(ev, EmbeddingCallCompletedEvent)
assert ev.outcome == "cancelled"
assert ev.error_class == "CancelledError"
@pytest.mark.asyncio
async def test_unknown_purpose_drops_to_none(self):
"""Unknown call_purpose ContextVar strings shouldn't crash the
emitter — they fall through to call_purpose=None on the event."""
emitted: list[BaseEvent] = []
async def _fake_call() -> list[float]:
return [0.1]
with (
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
embedding_call_purpose("not.a.real.purpose"),
):
await _emit_embedding_call(
provider="openai",
model="x",
texts=["q"],
input_tokens_estimate=1,
fn=_fake_call,
)
ev = emitted[0]
assert isinstance(ev, EmbeddingCallCompletedEvent)
assert ev.call_purpose is None
@pytest.mark.asyncio
async def test_telemetry_failure_swallowed(self):
"""Telemetry path must not propagate exceptions into the caller."""
def explode(*_args: Any, **_kwargs: Any) -> None:
raise RuntimeError("emitter wedged")
async def _fake_call() -> str:
return "ok"
with patch("src.telemetry.events.emit", side_effect=explode):
# Must NOT raise from the wrapper even though emit() throws.
result = await _emit_embedding_call(
provider="openai",
model="x",
texts=["q"],
input_tokens_estimate=1,
fn=_fake_call,
)
assert result == "ok"
class TestIsFinalAttempt:
"""`is_final_attempt` must reflect real retry state — one-shot callers
report True (no further attempt), retry-loop callers thread the real
index. Previously hardcoded to False; that conflated one-shot success,
mid-retry failure, and exhausted retry on dashboards.
"""
@pytest.mark.asyncio
async def test_oneshot_default_is_true(self):
emitted: list[BaseEvent] = []
async def _fake_call() -> list[float]:
return [0.1]
with patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
):
await _emit_embedding_call(
provider="openai",
model="x",
texts=["q"],
input_tokens_estimate=1,
fn=_fake_call,
)
ev = emitted[0]
assert isinstance(ev, EmbeddingCallCompletedEvent)
assert ev.is_final_attempt is True
@pytest.mark.asyncio
async def test_mid_retry_is_false(self):
emitted: list[BaseEvent] = []
async def _boom() -> list[float]:
raise RuntimeError("transient")
with (
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
pytest.raises(RuntimeError),
):
await _emit_embedding_call(
provider="openai",
model="x",
texts=["q"],
input_tokens_estimate=1,
fn=_boom,
is_final_attempt=False,
)
ev = emitted[0]
assert isinstance(ev, EmbeddingCallCompletedEvent)
assert ev.is_final_attempt is False
@pytest.mark.asyncio
async def test_exhausted_retry_is_true(self):
emitted: list[BaseEvent] = []
async def _boom() -> list[float]:
raise RuntimeError("permanent")
with (
patch(
"src.telemetry.events.emit",
side_effect=lambda event: emitted.append(event),
),
pytest.raises(RuntimeError),
):
await _emit_embedding_call(
provider="openai",
model="x",
texts=["q"],
input_tokens_estimate=1,
fn=_boom,
is_final_attempt=True,
)
ev = emitted[0]
assert isinstance(ev, EmbeddingCallCompletedEvent)
assert ev.is_final_attempt is True
assert ev.outcome == "error"

1
tests/telemetry/test_emit_function.py
View File

@ -32,6 +32,7 @@ def create_test_event() -> RepresentationCompletedEvent:
llm_call_ms=100.0,
total_duration_ms=110.0,
input_tokens=100,
total_input_tokens=150,
output_tokens=50,
)

57
tests/telemetry/test_emitter.py
View File

@ -44,6 +44,7 @@ def create_test_event(message_id: str = "msg_001") -> RepresentationCompletedEve
llm_call_ms=100.0,
total_duration_ms=110.0,
input_tokens=100,
total_input_tokens=150,
output_tokens=50,
)
@ -980,3 +981,59 @@ class TestCloudEventFormat:
cloud_event = json.loads(captured_content)
# Without namespace, source should be /honcho/{category}
assert cloud_event["source"] == "/honcho/representation"
class TestHonchoVersionInjection:
"""Tests for honcho_version body injection."""
@pytest.mark.asyncio
async def test_honcho_version_present_in_body(self):
"""honcho_version is unconditionally injected into event.data from the
HONCHO_VERSION constant (sourced from pyproject.toml)."""
from src._version import HONCHO_VERSION
emitter = TelemetryEmitter(endpoint="http://test:8001/events")
captured_content = None
async def capture_post(url, content=None, headers=None):
nonlocal captured_content
captured_content = content
response = MagicMock()
response.status_code = 200
response.raise_for_status = MagicMock()
return response
mock_client = AsyncMock()
mock_client.post = capture_post
mock_client.aclose = AsyncMock()
emitter._client = mock_client
emitter._running = True
with patch("src.config.settings") as mock_settings:
mock_settings.TELEMETRY.NAMESPACE = "test"
event = create_test_event()
emitter.emit(event)
await emitter.flush()
assert captured_content is not None
cloud_event = json.loads(captured_content)
assert cloud_event["data"]["honcho_version"] == HONCHO_VERSION
def test_emit_does_not_mutate_event_instance(self):
"""contract: emit() injects into the serialized body, never the
event instance. Tests asserting on the event object stay deterministic."""
emitter = TelemetryEmitter(endpoint="http://test:8001/events")
emitter._running = True
with patch("src.config.settings") as mock_settings:
mock_settings.TELEMETRY.NAMESPACE = "test"
event = create_test_event()
before = event.model_dump()
emitter.emit(event)
after = event.model_dump()
# The event instance must be unchanged by emit().
assert before == after
assert "honcho_version" not in after

371
tests/telemetry/test_events.py
View File

@ -1,7 +1,7 @@
# pyright: reportUnknownParameterType=false, reportMissingParameterType=false, reportUnusedParameter=false
"""Unit tests for telemetry event classes.
Tests all 12 event types for:
Tests all telemetry event types for:
- Correct instantiation with required fields
- event_type(), schema_version(), category() class methods
- get_resource_id() returns expected format
@ -21,10 +21,16 @@ from src.telemetry.events.agent import (
AgentToolPeerCardUpdatedEvent,
AgentToolSummaryCreatedEvent,
)
from src.telemetry.events.api import (
FileUploadedEvent,
GetContextEvent,
MessageCreatedEvent,
)
from src.telemetry.events.base import BaseEvent, generate_event_id
from src.telemetry.events.deletion import DeletionCompletedEvent
from src.telemetry.events.dialectic import DialecticCompletedEvent
from src.telemetry.events.dream import DreamRunEvent, DreamSpecialistEvent
from src.telemetry.events.llm import CallPurpose, LLMCallCompletedEvent
from src.telemetry.events.reconciliation import (
CleanupStaleItemsCompletedEvent,
SyncVectorsCompletedEvent,
@ -91,6 +97,27 @@ class TestGenerateEventId:
# Base64url encoded 16 bytes = 22 chars (without padding)
assert len(event_id) == 4 + 22 # "evt_" + 22 chars
def test_honcho_version_changes_id(self, fixed_timestamp: datetime):
"""Same event from different deploys must produce distinct IDs so
downstream dedupe doesn't silently merge events whose payload shape
may have shifted between versions."""
id_a = generate_event_id(
"test.event", fixed_timestamp, "resource_1", honcho_version="2.0.0"
)
id_b = generate_event_id(
"test.event", fixed_timestamp, "resource_1", honcho_version="2.0.1"
)
assert id_a != id_b
def test_honcho_version_none_matches_empty(self, fixed_timestamp: datetime):
"""None and unset version segments are equivalent — backwards-
compatible with callers that don't pass the new kwarg yet."""
id_default = generate_event_id("test.event", fixed_timestamp, "resource_1")
id_explicit_none = generate_event_id(
"test.event", fixed_timestamp, "resource_1", honcho_version=None
)
assert id_default == id_explicit_none
# =============================================================================
# Tests for BaseEvent class
@ -116,6 +143,7 @@ class TestBaseEvent:
llm_call_ms=100.0,
total_duration_ms=110.0,
input_tokens=100,
total_input_tokens=150,
output_tokens=50,
)
@ -142,10 +170,6 @@ class TestRepresentationCompletedEvent:
"""event_type() returns correct value."""
assert RepresentationCompletedEvent.event_type() == "representation.completed"
def test_schema_version(self):
"""schema_version() returns correct value."""
assert RepresentationCompletedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert RepresentationCompletedEvent.category() == "representation"
@ -181,6 +205,280 @@ class TestRepresentationCompletedEvent:
assert data["explicit_conclusion_count"] == 5
# =============================================================================
# Tests for LLMCallCompletedEvent ()
# =============================================================================
class TestLLMCallCompletedEvent:
"""Tests for the LLMCallCompletedEvent."""
def test_event_type(self):
assert LLMCallCompletedEvent.event_type() == "llm.call.completed"
def test_category(self):
assert LLMCallCompletedEvent.category() == "llm"
def test_volume_class(self):
# event must be high_volume so the sampler picks it up.
assert LLMCallCompletedEvent.volume_class() == "high_volume"
def test_get_resource_id_includes_attempt(
self, sample_llm_call_event: LLMCallCompletedEvent
):
# Resource id must include attempt so multiple retry attempts in one
# iteration get distinct deterministic ids.
assert (
sample_llm_call_event.get_resource_id()
== "abc12345:1:1:anthropic:claude-sonnet-4-5"
)
def test_call_purpose_enum_values(self):
# The closed taxonomy used by callers.
assert CallPurpose.DERIVER_REPRESENTATION.value == "deriver.representation"
assert CallPurpose.DIALECTIC_ANSWER.value == "dialectic.answer"
assert CallPurpose.DREAM_DEDUCTION.value == "dream.deduction"
assert CallPurpose.DREAM_INDUCTION.value == "dream.induction"
assert CallPurpose.SUMMARY_SHORT.value == "summary.short"
assert CallPurpose.SUMMARY_LONG.value == "summary.long"
def test_error_outcome_with_error_class(self, fixed_timestamp: datetime):
event = LLMCallCompletedEvent(
timestamp=fixed_timestamp,
transport="openai",
model="gpt-4",
effective_max_output_tokens=512,
outcome="error",
is_final_attempt=True,
error_class="RateLimitError",
attempt=3,
retry_attempts=3,
was_fallback=True,
duration_ms=200.0,
)
assert event.outcome == "error"
assert event.error_class == "RateLimitError"
assert event.is_final_attempt is True
# Token fields default to 0 when no result was produced.
assert event.provider_input_tokens == 0
assert event.provider_output_tokens == 0
def test_stream_placeholder_has_zero_tokens(self, fixed_timestamp: datetime):
event = LLMCallCompletedEvent(
timestamp=fixed_timestamp,
transport="anthropic",
model="claude-sonnet-4-5",
effective_max_output_tokens=2048,
outcome="success",
is_final_attempt=False,
attempt=1,
retry_attempts=3,
was_fallback=False,
duration_ms=0.0,
was_stream=True,
)
assert event.was_stream is True
assert event.provider_input_tokens == 0
assert event.provider_output_tokens == 0
# =============================================================================
# Tests for MessageCreatedEvent
# =============================================================================
class TestMessageCreatedEvent:
"""Tests for MessageCreatedEvent."""
def test_event_type(self):
"""event_type() returns correct value."""
assert MessageCreatedEvent.event_type() == "message.created"
def test_category(self):
"""category() returns correct value."""
assert MessageCreatedEvent.category() == "api"
def test_get_resource_id(self, sample_message_created_event: MessageCreatedEvent):
"""get_resource_id() keys on workspace:session:source:last_message_id."""
assert (
sample_message_created_event.get_resource_id()
== "test_workspace:test_session:api:msg_abc123_fixture_____"
)
def test_source_defaults_to_api(self, fixed_timestamp: datetime):
"""source defaults to api."""
event = MessageCreatedEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
session_name="test_session",
message_count=1,
total_tokens=100,
last_message_id="msg_default_source_____",
)
assert event.source == "api"
def test_distinct_batches_get_distinct_ids(self, fixed_timestamp: datetime):
"""Two batches of the same size in the same session+source must produce
different event ids — the previous (v1) key collided here."""
e1 = MessageCreatedEvent(
timestamp=fixed_timestamp,
workspace_name="ws",
session_name="sess",
message_count=5,
total_tokens=500,
source="api",
last_message_id="msg_first_batch________",
)
e2 = MessageCreatedEvent(
timestamp=fixed_timestamp,
workspace_name="ws",
session_name="sess",
message_count=5,
total_tokens=500,
source="api",
last_message_id="msg_second_batch_______",
)
assert e1.generate_id() != e2.generate_id()
assert e1.get_resource_id() != e2.get_resource_id()
# =============================================================================
# Tests for FileUploadedEvent
# =============================================================================
class TestFileUploadedEvent:
"""Tests for FileUploadedEvent."""
def test_event_type(self):
"""event_type() returns correct value."""
assert FileUploadedEvent.event_type() == "file.uploaded"
def test_category(self):
"""category() returns correct value."""
assert FileUploadedEvent.category() == "api"
def test_get_resource_id(self, sample_file_uploaded_event: FileUploadedEvent):
"""get_resource_id() returns workspace:session:file format."""
assert (
sample_file_uploaded_event.get_resource_id()
== "test_workspace:test_session:file_123"
)
def test_optional_file_fields(self, fixed_timestamp: datetime):
"""filename, content_type, and file_size_bytes are optional."""
event = FileUploadedEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
session_name="test_session",
peer_name="user_peer",
file_id="file_123",
message_count=1,
total_tokens=100,
)
assert event.filename is None
assert event.content_type is None
assert event.file_size_bytes is None
# =============================================================================
# Tests for GetContextEvent
# =============================================================================
class TestGetContextEvent:
"""Tests for GetContextEvent."""
def test_event_type(self):
"""event_type() returns correct value."""
assert GetContextEvent.event_type() == "context.retrieved"
def test_category(self):
"""category() returns correct value."""
assert GetContextEvent.category() == "api"
def test_get_resource_id_session(self, sample_get_context_event: GetContextEvent):
"""session context resource ID includes workspace and session.
Uses empty-string sentinel for unset peer/target so that a peer
literally named "none" can't collide with the absent-peer case.
"""
assert (
sample_get_context_event.get_resource_id()
== "test_workspace:session:test_session::"
)
def test_get_resource_id_disambiguates_peer_named_none(
self, fixed_timestamp: datetime
):
"""Regression: a peer literally named "none" must NOT collide with
absent-peer resource ids. Empty-string sentinel guards this.
"""
absent = GetContextEvent(
timestamp=fixed_timestamp,
workspace_name="ws",
context_scope="peer",
total_duration_ms=1.0,
)
literal_none = GetContextEvent(
timestamp=fixed_timestamp,
workspace_name="ws",
context_scope="peer",
peer_name="none",
target_name="none",
total_duration_ms=1.0,
)
assert absent.get_resource_id() != literal_none.get_resource_id()
def test_get_resource_id_peer(self, fixed_timestamp: datetime):
"""peer context resource ID includes observer and observed peers."""
event = GetContextEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
context_scope="peer",
peer_name="observer",
target_name="observed",
total_duration_ms=10.0,
)
assert event.get_resource_id() == "test_workspace:peer:observer:observed"
def test_context_defaults(self, fixed_timestamp: datetime):
"""Context booleans and counts have conservative defaults."""
event = GetContextEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
context_scope="session",
session_name="test_session",
total_duration_ms=10.0,
)
assert event.message_count == 0
assert event.has_summary is False
assert event.has_representation is False
assert event.include_summary is None
assert event.tokens_requested is None
assert event.peer_perspective_provided is False
def test_session_context_can_record_raw_request_options(
self, fixed_timestamp: datetime
):
"""Raw request options can be recorded independently of resolved values."""
event = GetContextEvent(
timestamp=fixed_timestamp,
workspace_name="test_workspace",
context_scope="session",
session_name="test_session",
peer_name="observer",
target_name="observed",
tokens_requested=8000,
include_summary=False,
peer_perspective_provided=True,
total_duration_ms=10.0,
)
assert event.tokens_requested == 8000
assert event.include_summary is False
assert event.peer_perspective_provided is True
# =============================================================================
# Tests for DreamRunEvent
# =============================================================================
@ -193,10 +491,6 @@ class TestDreamRunEvent:
"""event_type() returns correct value."""
assert DreamRunEvent.event_type() == "dream.run"
def test_schema_version(self):
"""schema_version() returns correct value."""
assert DreamRunEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert DreamRunEvent.category() == "dream"
@ -229,10 +523,6 @@ class TestDreamSpecialistEvent:
"""event_type() returns correct value."""
assert DreamSpecialistEvent.event_type() == "dream.specialist"
def test_schema_version(self):
"""schema_version() returns correct value."""
assert DreamSpecialistEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert DreamSpecialistEvent.category() == "dream"
@ -262,10 +552,6 @@ class TestDialecticCompletedEvent:
"""event_type() returns correct value."""
assert DialecticCompletedEvent.event_type() == "dialectic.completed"
def test_schema_version(self):
"""schema_version() returns correct value."""
assert DialecticCompletedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert DialecticCompletedEvent.category() == "dialectic"
@ -324,10 +610,6 @@ class TestAgentIterationEvent:
"""event_type() returns correct value."""
assert AgentIterationEvent.event_type() == "agent.iteration"
def test_schema_version(self):
"""schema_version() returns correct value."""
assert AgentIterationEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert AgentIterationEvent.category() == "agent"
@ -380,10 +662,6 @@ class TestAgentToolConclusionsCreatedEvent:
== "agent.tool.conclusions.created"
)
def test_schema_version(self):
"""schema_version() returns correct value."""
assert AgentToolConclusionsCreatedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert AgentToolConclusionsCreatedEvent.category() == "agent"
@ -420,10 +698,6 @@ class TestAgentToolConclusionsDeletedEvent:
== "agent.tool.conclusions.deleted"
)
def test_schema_version(self):
"""schema_version() returns correct value."""
assert AgentToolConclusionsDeletedEvent.schema_version() == 2
def test_category(self):
"""category() returns correct value."""
assert AgentToolConclusionsDeletedEvent.category() == "agent"
@ -452,10 +726,6 @@ class TestAgentToolPeerCardUpdatedEvent:
AgentToolPeerCardUpdatedEvent.event_type() == "agent.tool.peer_card.updated"
)
def test_schema_version(self):
"""schema_version() returns correct value."""
assert AgentToolPeerCardUpdatedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert AgentToolPeerCardUpdatedEvent.category() == "agent"
@ -482,10 +752,6 @@ class TestAgentToolSummaryCreatedEvent:
"""event_type() returns correct value."""
assert AgentToolSummaryCreatedEvent.event_type() == "agent.tool.summary.created"
def test_schema_version(self):
"""schema_version() returns correct value."""
assert AgentToolSummaryCreatedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert AgentToolSummaryCreatedEvent.category() == "agent"
@ -532,10 +798,6 @@ class TestDeletionCompletedEvent:
"""event_type() returns correct value."""
assert DeletionCompletedEvent.event_type() == "deletion.completed"
def test_schema_version(self):
"""schema_version() returns correct value."""
assert DeletionCompletedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert DeletionCompletedEvent.category() == "deletion"
@ -601,10 +863,6 @@ class TestSyncVectorsCompletedEvent:
== "reconciliation.sync_vectors.completed"
)
def test_schema_version(self):
"""schema_version() returns correct value."""
assert SyncVectorsCompletedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert SyncVectorsCompletedEvent.category() == "reconciliation"
@ -642,10 +900,6 @@ class TestCleanupStaleItemsCompletedEvent:
== "reconciliation.cleanup_stale_items.completed"
)
def test_schema_version(self):
"""schema_version() returns correct value."""
assert CleanupStaleItemsCompletedEvent.schema_version() == 1
def test_category(self):
"""category() returns correct value."""
assert CleanupStaleItemsCompletedEvent.category() == "reconciliation"
@ -665,6 +919,25 @@ class TestCleanupStaleItemsCompletedEvent:
assert event.documents_cleaned == 0
assert event.queue_items_cleaned == 0
def test_queue_items_cleaned_round_trips_through_pydantic(
self, fixed_timestamp: datetime
):
"""Regression: `queue_items_cleaned` is a real field, not just
plumbing. Previously the consumer emit site dropped the captured
`deleted_count` and the field always defaulted to 0 on the wire.
"""
event = CleanupStaleItemsCompletedEvent(
timestamp=fixed_timestamp,
total_duration_ms=500.0,
queue_items_cleaned=42,
)
assert event.queue_items_cleaned == 42
# Serialize → deserialize to ensure the field crosses the wire.
data = event.model_dump(mode="json")
assert data["queue_items_cleaned"] == 42
round_tripped = CleanupStaleItemsCompletedEvent.model_validate(data)
assert round_tripped.queue_items_cleaned == 42
# =============================================================================
# Parametrized tests across all event types

194
tests/telemetry/test_representation_v2_fields.py Normal file
View File

@ -0,0 +1,194 @@
# pyright: reportPrivateUsage=false, reportUnknownLambdaType=false, reportUnknownArgumentType=false, reportArgumentType=false
"""tests for RepresentationCompletedEvent additive fields + truncation.
Targets:
- Schema stays at v2 (additive, no bump). Existing `input_tokens` semantics
unchanged.
- fields are defaultable (no breakage for callers that ignore them)
and round-trip through Pydantic serialization.
- `HonchoLLMCallResponse.hit_input_token_cap` defaults to False but can be
flipped by the tool-less cap-detection path in `src/llm/api.py`.
"""
from __future__ import annotations
from src.llm.types import HonchoLLMCallResponse
from src.telemetry.events.representation import RepresentationCompletedEvent
class TestRepresentationV2AdditiveFields:
def test_schema_stays_at_v2(self):
"""is additive — schema_version must NOT bump to 3."""
assert RepresentationCompletedEvent.schema_version() == 2
def test_new_fields_are_optional(self):
"""Existing callers must keep working without supplying any new
fields. All new fields default."""
event = RepresentationCompletedEvent(
workspace_name="ws",
session_name="s",
observed="user",
queue_items_processed=1,
earliest_message_id="m1",
latest_message_id="m1",
message_count=1,
explicit_conclusion_count=0,
context_preparation_ms=10.0,
llm_call_ms=100.0,
total_duration_ms=110.0,
input_tokens=100,
total_input_tokens=200,
output_tokens=50,
)
# All fields land with defaults.
assert event.queued_message_count == 0
assert event.prompt_message_count == 0
assert event.prompt_message_tokens == 0
assert event.extra_context_message_count == 0
assert event.extra_context_tokens == 0
assert event.prompt_scaffold_tokens == 0
assert event.batch_max_tokens == 0
assert event.max_input_tokens == 0
assert event.was_flush_enabled is False
assert event.hit_batch_token_cap is False
assert event.hit_input_token_cap is False
assert event.observer_count == 0
def test_input_tokens_semantics_preserved(self):
"""The downstream metering key must remain 'queued-message tokens'.
Added many fields, but `input_tokens` MUST stay as the downstream
metering key for representation.completed. Don't rename or
repurpose without coordinating with consumers.
"""
event = RepresentationCompletedEvent(
workspace_name="ws",
session_name="s",
observed="user",
queue_items_processed=2,
earliest_message_id="m1",
latest_message_id="m5",
message_count=5,
explicit_conclusion_count=3,
context_preparation_ms=10.0,
llm_call_ms=100.0,
total_duration_ms=110.0,
input_tokens=300, # ← queued message tokens; the billing key
total_input_tokens=900, # provider-side total
output_tokens=50,
queued_message_count=2,
prompt_message_count=5,
prompt_message_tokens=800,
extra_context_message_count=3,
extra_context_tokens=500,
prompt_scaffold_tokens=100,
)
# input_tokens and queued_message_count should describe the same set
# (queue items being reasoned about) — assert the conceptual link.
assert event.input_tokens == 300
assert event.queued_message_count == 2
# And the breakdown adds up: extra + scaffold + queued ≈ total provider input
# (it's an approximation — provider includes formatting overhead).
assert (
event.extra_context_tokens
+ event.input_tokens
+ event.prompt_scaffold_tokens
== 900
)
assert event.total_input_tokens == 900
def test_cap_hit_flags(self):
event = RepresentationCompletedEvent(
workspace_name="ws",
session_name="s",
observed="user",
queue_items_processed=10,
earliest_message_id="m1",
latest_message_id="m10",
message_count=10,
explicit_conclusion_count=5,
context_preparation_ms=10.0,
llm_call_ms=100.0,
total_duration_ms=110.0,
input_tokens=20_000,
total_input_tokens=23_000,
output_tokens=500,
batch_max_tokens=20_000,
max_input_tokens=23_000,
was_flush_enabled=True,
hit_batch_token_cap=True,
hit_input_token_cap=True,
observer_count=2,
)
assert event.was_flush_enabled is True
assert event.hit_batch_token_cap is True
assert event.hit_input_token_cap is True
assert event.batch_max_tokens == 20_000
assert event.max_input_tokens == 23_000
assert event.observer_count == 2
def test_model_dump_includes_new_fields(self):
event = RepresentationCompletedEvent(
workspace_name="ws",
session_name="s",
observed="user",
queue_items_processed=1,
earliest_message_id="m1",
latest_message_id="m1",
message_count=1,
explicit_conclusion_count=0,
context_preparation_ms=10.0,
llm_call_ms=100.0,
total_duration_ms=110.0,
input_tokens=100,
total_input_tokens=150,
output_tokens=50,
hit_batch_token_cap=True,
)
data = event.model_dump(mode="json")
for field in (
"queued_message_count",
"prompt_message_count",
"prompt_message_tokens",
"extra_context_message_count",
"extra_context_tokens",
"prompt_scaffold_tokens",
"batch_max_tokens",
"max_input_tokens",
"was_flush_enabled",
"hit_batch_token_cap",
"hit_input_token_cap",
"observer_count",
):
assert field in data, f"missing field: {field}"
assert data["hit_batch_token_cap"] is True
class TestHitInputTokenCapFlag:
"""`HonchoLLMCallResponse.hit_input_token_cap` is the bridge between the
tool-less cap-detection path in src/llm/api.py and the deriver's
`hit_input_token_cap` field on RepresentationCompletedEvent.
The flag is token-based — it fires whenever the original input exceeded
`max_input_tokens`, whether or not message truncation could actually
shrink the input below cap (the deriver's single-prompt case can't).
"""
def test_defaults_to_false(self):
response = HonchoLLMCallResponse(
content="hi",
input_tokens=10,
output_tokens=5,
finish_reasons=["stop"],
)
assert response.hit_input_token_cap is False
def test_can_be_flipped(self):
response = HonchoLLMCallResponse(
content="hi",
input_tokens=10,
output_tokens=5,
finish_reasons=["stop"],
)
response.hit_input_token_cap = True
assert response.hit_input_token_cap is True

122
tests/telemetry/test_summary_v2_fields.py Normal file
View File

@ -0,0 +1,122 @@
# pyright: reportPrivateUsage=false
"""tests for AgentToolSummaryCreatedEvent additive token breakdown.
Targets:
- Schema bumps to v2.
- `input_tokens` semantic preserved (provider-side input).
- New breakdown fields (`previous_summary_tokens`, `message_tokens`,
`prompt_scaffold_tokens`) default cleanly and round-trip.
- The conceptual relationship: message_tokens + previous_summary_tokens
describes the *user-data* portion of the input; prompt_scaffold_tokens is
the static instruction portion. Together they approximate the provider
input_tokens (modulo formatting overhead).
"""
from __future__ import annotations
from src.telemetry.events.agent import AgentToolSummaryCreatedEvent
class TestAdditiveFields:
def test_new_fields_default_to_zero(self):
"""Callers that omit the breakdown fields must construct valid events."""
event = AgentToolSummaryCreatedEvent(
run_id="r",
iteration=0,
parent_category="deriver",
agent_type="summarizer",
workspace_name="ws",
session_name="s",
message_id="m1",
message_count=10,
message_seq_in_session=10,
summary_type="short",
input_tokens=1000,
output_tokens=100,
)
assert event.previous_summary_tokens == 0
assert event.message_tokens == 0
assert event.prompt_scaffold_tokens == 0
def test_input_tokens_semantic_preserved(self):
"""`input_tokens` continues to be the provider-side LLM input count.
We deliberately do NOT add a redundant `provider_input_tokens` —
the existing field already serves that purpose, and a duplicate
would silently fork downstream queries."""
event = AgentToolSummaryCreatedEvent(
run_id="r",
iteration=0,
parent_category="deriver",
agent_type="summarizer",
workspace_name="ws",
session_name="s",
message_id="m1",
message_count=10,
message_seq_in_session=10,
summary_type="long",
input_tokens=5000, # provider-reported total
output_tokens=400,
previous_summary_tokens=500,
message_tokens=4000,
prompt_scaffold_tokens=400,
)
# message + prev_summary + scaffold ≈ input_tokens (small drift from
# provider-side formatting overhead is expected).
breakdown_sum = (
event.message_tokens
+ event.previous_summary_tokens
+ event.prompt_scaffold_tokens
)
assert breakdown_sum <= event.input_tokens + 200 # allow small overhead
assert event.input_tokens == 5000
def test_first_summary_has_zero_previous_summary_tokens(self):
"""When there's no prior summary for the session, the breakdown
carries `previous_summary_tokens=0` so analytics can distinguish
first-summary calls from rollup calls."""
event = AgentToolSummaryCreatedEvent(
run_id="r",
iteration=0,
parent_category="deriver",
agent_type="summarizer",
workspace_name="ws",
session_name="s",
message_id="m1",
message_count=5,
message_seq_in_session=5,
summary_type="short",
input_tokens=1500,
output_tokens=150,
previous_summary_tokens=0, # first summary for this session
message_tokens=1200,
prompt_scaffold_tokens=300,
)
assert event.previous_summary_tokens == 0
assert event.message_tokens > 0
def test_model_dump_includes_breakdown_fields(self):
event = AgentToolSummaryCreatedEvent(
run_id="r",
iteration=0,
parent_category="deriver",
agent_type="summarizer",
workspace_name="ws",
session_name="s",
message_id="m1",
message_count=10,
message_seq_in_session=10,
summary_type="short",
input_tokens=1000,
output_tokens=100,
previous_summary_tokens=100,
message_tokens=700,
prompt_scaffold_tokens=200,
)
data = event.model_dump(mode="json")
for field in (
"previous_summary_tokens",
"message_tokens",
"prompt_scaffold_tokens",
):
assert field in data, f"missing field: {field}"
assert data["message_tokens"] == 700

2
tests/test_models_vector_dim.py
View File

@ -1,4 +1,4 @@
"""Phase 1: verify src/models.py honors EMBEDDING_VECTOR_DIMENSIONS at import time."""
"""Verify src/models.py honors EMBEDDING_VECTOR_DIMENSIONS at import time."""
from __future__ import annotations

204
tests/utils/test_agent_tools.py
View File

@ -14,7 +14,9 @@ from sqlalchemy.ext.asyncio import AsyncSession
from src import crud, models, schemas
from src.config import settings
from src.utils.agent_tools import (
MAX_PEER_CARD_ENTRY_LENGTH,
MAX_PEER_CARD_FACTS,
PEER_CARD_ALLOWED_PREFIXES,
ObservationsCreatedResult,
ToolContext,
_handle_create_observations, # pyright: ignore[reportPrivateUsage]
@ -32,6 +34,7 @@ from src.utils.agent_tools import (
_handle_search_messages, # pyright: ignore[reportPrivateUsage]
_handle_search_messages_temporal, # pyright: ignore[reportPrivateUsage]
_handle_update_peer_card, # pyright: ignore[reportPrivateUsage]
_validate_peer_card_entry, # pyright: ignore[reportPrivateUsage]
create_observations,
create_tool_executor,
extract_preferences,
@ -253,7 +256,8 @@ class TestCreateObservations:
result = await _handle_create_observations(ctx, {"observations": []})
assert "ERROR" in result
assert "empty" in result.lower()
# Handlers may return ToolResult (); str() returns .content.
assert "empty" in str(result).lower()
async def test_batch_embedding_failure_falls_back_to_individual_embeds(
self,
@ -415,7 +419,9 @@ class TestCreateObservations:
result = await create_observations(
observations=[
schemas.ObservationInput(content=" ", level="explicit"),
schemas.ObservationInput(content=" trimmed observation ", level="explicit"),
schemas.ObservationInput(
content=" trimmed observation ", level="explicit"
),
],
observer=peer1.name,
observed=peer2.name,
@ -753,8 +759,12 @@ class TestSearchMessages:
result = await _handle_search_messages(ctx, {"query": "test message"})
# Should return some result (may be empty if semantic search doesn't match)
assert isinstance(result, str)
# handler may return ToolResult (with search metadata) or
# a plain str. Both carry the result text; just check it's
# introspectable as string content.
from src.utils.types import ToolResult
assert isinstance(result, str | ToolResult)
@pytest.mark.asyncio
@ -895,7 +905,7 @@ class TestGetRecentHistory:
result = await _handle_get_recent_history(ctx, {})
assert "Conversation history" in result
assert "messages" in result.lower()
assert "messages" in str(result).lower()
async def test_without_session_uses_observed(
self,
@ -1016,7 +1026,14 @@ class TestUpdatePeerCard:
ctx = make_tool_context()
result = await _handle_update_peer_card(
ctx, {"content": ["Name: John", "Location: NYC", "Occupation: Engineer"]}
ctx,
{
"content": [
"IDENTITY: Name: John",
"ATTRIBUTE: Location: NYC",
"ATTRIBUTE: Occupation: Engineer",
]
},
)
assert "Updated peer card" in result
@ -1031,7 +1048,7 @@ class TestUpdatePeerCard:
observed=peer2.name,
)
assert peer_card is not None
assert "Name: John" in peer_card
assert "IDENTITY: Name: John" in peer_card
async def test_deduplicates_and_caps_peer_card(
self,
@ -1043,8 +1060,15 @@ class TestUpdatePeerCard:
workspace, peer1, peer2, _, _, _ = tool_test_data
ctx = make_tool_context()
oversized = ["Name: John", " Name: John ", "", " "]
oversized.extend([f"Fact {i}" for i in range(MAX_PEER_CARD_FACTS + 5)])
oversized = [
"IDENTITY: Name: John",
" IDENTITY: Name: John ",
"",
" ",
]
oversized.extend(
[f"IDENTITY: Aliases: alias-{i}" for i in range(MAX_PEER_CARD_FACTS + 5)]
)
await _handle_update_peer_card(ctx, {"content": oversized})
@ -1059,7 +1083,7 @@ class TestUpdatePeerCard:
assert peer_card is not None
assert len(peer_card) == MAX_PEER_CARD_FACTS
assert all(line.strip() for line in peer_card)
assert peer_card.count("Name: John") == 1
assert peer_card.count("IDENTITY: Name: John") == 1
async def test_none_content_preserves_existing_card(
self,
@ -1073,12 +1097,13 @@ class TestUpdatePeerCard:
# First, create a valid peer card
await _handle_update_peer_card(
ctx, {"content": ["Name: Alice", "Location: NYC"]}
ctx,
{"content": ["IDENTITY: Name: Alice", "ATTRIBUTE: Location: NYC"]},
)
# Now attempt to update with None — should be a no-op
result = await _handle_update_peer_card(ctx, {"content": None})
assert "empty" in result.lower()
assert "empty" in str(result).lower()
# Refresh the observer so the identity map picks up the committed update
await db_session.refresh(peer1)
@ -1090,7 +1115,7 @@ class TestUpdatePeerCard:
observed=peer2.name,
)
assert peer_card is not None
assert "Name: Alice" in peer_card
assert "IDENTITY: Name: Alice" in peer_card
async def test_empty_list_preserves_existing_card(
self,
@ -1103,11 +1128,14 @@ class TestUpdatePeerCard:
ctx = make_tool_context()
# First, create a valid peer card
await _handle_update_peer_card(ctx, {"content": ["Name: Bob", "Age: 30"]})
await _handle_update_peer_card(
ctx,
{"content": ["IDENTITY: Name: Bob", "ATTRIBUTE: Age: 30"]},
)
# Now attempt to update with empty list — should be a no-op
result = await _handle_update_peer_card(ctx, {"content": []})
assert "empty" in result.lower()
assert "empty" in str(result).lower()
# Refresh the observer so the identity map picks up the committed update
await db_session.refresh(peer1)
@ -1119,7 +1147,151 @@ class TestUpdatePeerCard:
observed=peer2.name,
)
assert peer_card is not None
assert "Name: Bob" in peer_card
assert "IDENTITY: Name: Bob" in peer_card
async def test_rejects_entries_without_allowed_prefix(
self,
db_session: AsyncSession,
tool_test_data: Any,
make_tool_context: Callable[..., ToolContext],
):
"""Entries without an allowed prefix are dropped; valid entries pass through."""
from src.utils.types import ToolResult
workspace, peer1, peer2, _, _, _ = tool_test_data
ctx = make_tool_context()
result = await _handle_update_peer_card(
ctx,
{
"content": [
"IDENTITY: Name: Carol",
"Age: 39+", # rejected: no prefix
"Daughter: Keyan", # rejected: no prefix
"TRAIT: Methodical", # rejected: TRAIT not allowed
"PREFERENCE: Tea", # rejected: bare PREFERENCE not allowed
"ATTRIBUTE: Location: Germantown, TN",
]
},
)
# Partial-reject success path must surface the rejection in the tool
# response so the model can re-emit the dropped entries (with correct
# prefixes) on a retry instead of silently losing them.
assert isinstance(result, ToolResult)
content_lower = str(result).lower()
assert "updated peer card" in content_lower
assert "rejected 4 of 6" in content_lower
# At least one rejected sample should appear so the model knows what
# to fix.
assert "age: 39+" in content_lower or "trait: methodical" in content_lower
assert result.metadata is not None
assert result.metadata["peer_card_updated"] is True
assert result.metadata["facts_count"] == 2
assert result.metadata["rejected_count"] == 4
await db_session.refresh(peer1)
peer_card = await crud.get_peer_card(
db_session,
workspace_name=workspace.name,
observer=peer1.name,
observed=peer2.name,
)
assert peer_card is not None
assert peer_card == [
"IDENTITY: Name: Carol",
"ATTRIBUTE: Location: Germantown, TN",
]
async def test_all_entries_rejected_preserves_existing_card(
self,
db_session: AsyncSession,
tool_test_data: Any,
make_tool_context: Callable[..., ToolContext],
):
"""When every entry fails validation, the existing card is preserved."""
workspace, peer1, peer2, _, _, _ = tool_test_data
ctx = make_tool_context()
await _handle_update_peer_card(ctx, {"content": ["IDENTITY: Name: Dana"]})
result = await _handle_update_peer_card(
ctx,
{
"content": [
"TRAIT: Detail-oriented",
"PREFERENCE: Coffee",
"Random unprefixed line",
]
},
)
assert "rejected" in str(result).lower()
await db_session.refresh(peer1)
peer_card = await crud.get_peer_card(
db_session,
workspace_name=workspace.name,
observer=peer1.name,
observed=peer2.name,
)
assert peer_card == ["IDENTITY: Name: Dana"]
class TestPeerCardEntryValidator:
"""Unit tests for the pure structural validator."""
@pytest.mark.parametrize(
"entry",
[
"IDENTITY: Name: Alice",
"ATTRIBUTE: Location: NYC",
"ATTRIBUTE: Prefers tea",
"RELATIONSHIP: Spouse: Bob",
"RELATIONSHIP: Maintainer: vineeth",
"INSTRUCTION: Call me Vee",
"INSTRUCTION: Never push to main without review",
],
)
def test_accepts_well_formed_entries(self, entry: str):
assert _validate_peer_card_entry(entry) is True
@pytest.mark.parametrize(
"entry",
[
"",
" ",
"Name: Alice", # missing prefix
"Age: 39+", # missing prefix
"Daughter: Keyan", # missing prefix
"TRAIT: Methodical", # disallowed kind
"PREFERENCE: Tea", # disallowed kind
"identity: name: alice", # wrong case
"IDENTITY:Name: Alice", # missing space after colon
"IDENTITY: ", # empty body
"IDENTITY: ", # whitespace-only body
],
)
def test_rejects_malformed_entries(self, entry: str):
assert _validate_peer_card_entry(entry) is False
def test_rejects_over_length_cap(self):
long_value = "x" * (MAX_PEER_CARD_ENTRY_LENGTH + 1)
assert _validate_peer_card_entry(f"IDENTITY: Name: {long_value}") is False
def test_accepts_at_length_cap(self):
# Build an entry exactly at the cap.
prefix = "IDENTITY: "
body = "x" * (MAX_PEER_CARD_ENTRY_LENGTH - len(prefix))
assert _validate_peer_card_entry(prefix + body) is True
def test_allowed_prefixes_constant_is_complete(self):
# Guard against silent drift between the prompt and the validator.
assert PEER_CARD_ALLOWED_PREFIXES == (
"IDENTITY:",
"ATTRIBUTE:",
"RELATIONSHIP:",
"INSTRUCTION:",
)
@pytest.mark.asyncio

Some files were not shown because too many files have changed in this diff Show More