* fix(deriver): truncate oversize observations so one cannot drop the batch simple_batch_embed raised ValueError when any input exceeded the per-input token cap, which failed the entire deriver save when a single observation was over-length. Add on_oversize="truncate": oversize inputs are embedded from a token-capped prefix (re-encoded until it fits, with a warning), preserving one vector per input. Default stays "raise" so existing callers are unchanged. RepresentationManager opts into truncate. Also add a live embedding test that fails on main (raise / missing kwarg) and passes once a mixed short+oversize batch survives. Refs #569 * fix(deriver): surface failure when all observer saves fail When every observer's save_representation failed (e.g. embedding retries exhausted under a sustained 429), the deriver logged the error and returned normally, so the queue marked the work unit processed with zero documents saved. Collect per-observer errors and, after telemetry is emitted, raise RepresentationSaveError when no observer succeeded. Partial failures stay processed (saved observers must not be discarded) and are recorded via an additive failed_observer_count on RepresentationCompletedEvent. Refs #728 * fix(embedding): guarantee truncation progress and truncate on re-embed The retry slice in _truncate_to_token_limit always recomputed the same keep count, so a slice whose re-encode grew past the cap could oscillate. Decrement keep after each unsuccessful retry. Document re-embed in the reconciler used the default on_oversize="raise", so one oversize document failed every other document in the batch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: drop ticket ids and shrink comments to one sentence Comments and docstrings describe current behavior, not the PR that introduced them. Ticket numbers stay in the commit/PR. * chore: annotate RepresentationSaveError and assert truncate on re-embed * fix(embedding): truncate on conclusion create paths and document BPE loop Storage callers in create_observations (API + agent tools) now pass on_oversize="truncate" so a single oversize item cannot drop the batch. Docstring on _truncate_to_token_limit notes why decode/re-encode is load-bearing. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| README.md | ||
| __init__.py | ||
| conftest.py | ||
| test_deriver_processing.py | ||
| test_embed_now.py | ||
| test_enqueue_dream.py | ||
| test_prompts.py | ||
| test_queue_operations.py | ||
| test_queue_processing.py | ||
| test_representation_crud.py | ||
| test_scope_backfill.py | ||
| test_vector_reconciliation.py | ||
README.md
Deriver Testing
This directory contains tests for the deriver system, which handles background processing of messages to extract insights and update working representations.
Structure
conftest.py- Shared fixtures for deriver testingtest_queue_operations.py- Tests for basic queue operationstest_deriver_processing.py- Tests for deriver processing logictest_queue_processing.py- Tests for queue manager and work unit processing
Key Fixtures
Database Fixtures
sample_session_with_peers- Creates a session with multiple peers having different observation configurationssample_messages- Creates sample messages for testingsample_queue_items- Creates queue items with various payload types (representation, summary)
Queue Fixtures
create_queue_payload- Helper to create queue payloads for testingadd_queue_items- Helper to add queue items to the databasecreate_active_queue_session- Helper to create active queue sessions for work unit tracking
Mocking Fixtures
mock_critical_analysis_call- Mocks the critical analysis LLM callmock_queue_manager- Mocks the queue manager for testingmock_representation_manager- Mocks the representation manager operations
Testing Patterns
Creating Queue Items
# Create representation payloads
payload = create_queue_payload(
message=message,
task_type="representation",
observer=observer_peer.name,
observed=message.peer_name
)
# Add to queue
queue_items = await add_queue_items([payload], session.id)
Testing Work Units
# Create a work unit
work_unit = WorkUnit(
session_id=session.id,
task_type="representation",
observer=observer,
observed=observed
)
# Test string representation
assert str(work_unit) == f"({session.id}, {observed.name}, {observer.name}, representation)"