honcho/tests/deriver
Aakash Kattelu ddbb90e36f
fix(embedding): truncate in batch embed and return results breakdown (#1019)
* fix(deriver): truncate oversize observations so one cannot drop the batch

simple_batch_embed raised ValueError when any input exceeded the per-input
token cap, which failed the entire deriver save when a single observation
was over-length. Add on_oversize="truncate": oversize inputs are embedded
from a token-capped prefix (re-encoded until it fits, with a warning),
preserving one vector per input. Default stays "raise" so existing callers
are unchanged. RepresentationManager opts into truncate.

Also add a live embedding test that fails on main (raise / missing kwarg)
and passes once a mixed short+oversize batch survives.

Refs #569

* fix(deriver): surface failure when all observer saves fail

When every observer's save_representation failed (e.g. embedding retries
exhausted under a sustained 429), the deriver logged the error and returned
normally, so the queue marked the work unit processed with zero documents
saved. Collect per-observer errors and, after telemetry is emitted, raise
RepresentationSaveError when no observer succeeded. Partial failures stay
processed (saved observers must not be discarded) and are recorded via an
additive failed_observer_count on RepresentationCompletedEvent.

Refs #728

* fix(embedding): guarantee truncation progress and truncate on re-embed

The retry slice in _truncate_to_token_limit always recomputed the same
keep count, so a slice whose re-encode grew past the cap could oscillate.
Decrement keep after each unsuccessful retry.

Document re-embed in the reconciler used the default on_oversize="raise",
so one oversize document failed every other document in the batch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore: drop ticket ids and shrink comments to one sentence

Comments and docstrings describe current behavior, not the PR that
introduced them. Ticket numbers stay in the commit/PR.

* chore: annotate RepresentationSaveError and assert truncate on re-embed

* fix(embedding): truncate on conclusion create paths and document BPE loop

Storage callers in create_observations (API + agent tools) now pass
on_oversize="truncate" so a single oversize item cannot drop the batch.
Docstring on _truncate_to_token_limit notes why decode/re-encode is load-bearing.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 11:42:38 -04:00
..
README.md create Representation class and use it to unify all formatting (#214) 2025-10-07 15:28:44 -04:00
__init__.py Vineeth/dev 1027 (#177) 2025-08-06 16:20:22 -04:00
conftest.py feat: honcho 3.0, sdks 2.0, excise stainless, update v3 docs, changelogs (#331) 2026-01-22 15:16:28 -05:00
test_deriver_processing.py fix(embedding): truncate in batch embed and return results breakdown (#1019) 2026-08-20 11:42:38 -04:00
test_embed_now.py add prometheus metric for in_flight 2026-07-09 12:01:09 -04:00
test_enqueue_dream.py fix(dreamer): threshold and time-guard semantics (#573) 2026-04-30 11:40:51 -04:00
test_prompts.py feat: deriver custom instructions (#609) 2026-05-11 18:05:42 -04:00
test_queue_operations.py feat: webhooks (#168) 2025-08-06 17:52:35 -04:00
test_queue_processing.py rename to REPRESENTATION_BATCH_TARGET_INPUT_TOKENS 2026-07-09 10:42:52 -04:00
test_representation_crud.py feat: agentic dreamer and agentic dialectic (#309) 2026-01-12 15:12:17 -05:00
test_scope_backfill.py feat: scope backfill-by-copy and removal reconciliation jobs (#904) 2026-08-14 15:08:43 -04:00
test_vector_reconciliation.py fix(embedding): truncate in batch embed and return results breakdown (#1019) 2026-08-20 11:42:38 -04:00

README.md

Deriver Testing

This directory contains tests for the deriver system, which handles background processing of messages to extract insights and update working representations.

Structure

  • conftest.py - Shared fixtures for deriver testing
  • test_queue_operations.py - Tests for basic queue operations
  • test_deriver_processing.py - Tests for deriver processing logic
  • test_queue_processing.py - Tests for queue manager and work unit processing

Key Fixtures

Database Fixtures

  • sample_session_with_peers - Creates a session with multiple peers having different observation configurations
  • sample_messages - Creates sample messages for testing
  • sample_queue_items - Creates queue items with various payload types (representation, summary)

Queue Fixtures

  • create_queue_payload - Helper to create queue payloads for testing
  • add_queue_items - Helper to add queue items to the database
  • create_active_queue_session - Helper to create active queue sessions for work unit tracking

Mocking Fixtures

  • mock_critical_analysis_call - Mocks the critical analysis LLM call
  • mock_queue_manager - Mocks the queue manager for testing
  • mock_representation_manager - Mocks the representation manager operations

Testing Patterns

Creating Queue Items

# Create representation payloads
payload = create_queue_payload(
    message=message,
    task_type="representation",
    observer=observer_peer.name,
    observed=message.peer_name
)

# Add to queue
queue_items = await add_queue_items([payload], session.id)

Testing Work Units

# Create a work unit
work_unit = WorkUnit(
    session_id=session.id,
    task_type="representation",
    observer=observer,
    observed=observed
)

# Test string representation
assert str(work_unit) == f"({session.id}, {observed.name}, {observer.name}, representation)"