* feat: defer embedding messages * fix: rm gauges * feat: embed messages immediately on create with reconciler fallback (#766) Adds embed_messages_now background task so newly created messages are searchable within seconds instead of waiting up to the reconciler interval. Three-phase claim/lease → embed → persist never holds a DB session across the embedding call; the reconciler remains the fallback for failures and stragglers. * fix: harden immediate-embed fast path and cover its error branches Wrap embed_messages_now in a top-level try/except so a failure in the claim or persist phase degrades to "reconciler will retry" instead of escaping into the background-task runner; the rows stay pending+leased and the reconciler heals them. Add tests for the previously-uncovered branches: external-store-unavailable persist path, the file-upload endpoint's embed scheduling, and direct unit tests for the shared compute_chunk_positions / build_message_vector_record helpers. Document the semantic-search eventual-consistency window in search.mdx (keyword matches are immediate; vector matches lag creation by seconds). * fix: don't hold DB session across vector-store upserts * fix: align semantic-search function to filter null rows --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| README.md | ||
| __init__.py | ||
| conftest.py | ||
| test_deriver_processing.py | ||
| test_embed_now.py | ||
| test_enqueue_dream.py | ||
| test_prompts.py | ||
| test_queue_operations.py | ||
| test_queue_processing.py | ||
| test_representation_crud.py | ||
| test_vector_reconciliation.py | ||
README.md
Deriver Testing
This directory contains tests for the deriver system, which handles background processing of messages to extract insights and update working representations.
Structure
conftest.py- Shared fixtures for deriver testingtest_queue_operations.py- Tests for basic queue operationstest_deriver_processing.py- Tests for deriver processing logictest_queue_processing.py- Tests for queue manager and work unit processing
Key Fixtures
Database Fixtures
sample_session_with_peers- Creates a session with multiple peers having different observation configurationssample_messages- Creates sample messages for testingsample_queue_items- Creates queue items with various payload types (representation, summary)
Queue Fixtures
create_queue_payload- Helper to create queue payloads for testingadd_queue_items- Helper to add queue items to the databasecreate_active_queue_session- Helper to create active queue sessions for work unit tracking
Mocking Fixtures
mock_critical_analysis_call- Mocks the critical analysis LLM callmock_queue_manager- Mocks the queue manager for testingmock_representation_manager- Mocks the representation manager operations
Testing Patterns
Creating Queue Items
# Create representation payloads
payload = create_queue_payload(
message=message,
task_type="representation",
observer=observer_peer.name,
observed=message.peer_name
)
# Add to queue
queue_items = await add_queue_items([payload], session.id)
Testing Work Units
# Create a work unit
work_unit = WorkUnit(
session_id=session.id,
task_type="representation",
observer=observer,
observed=observed
)
# Test string representation
assert str(work_unit) == f"({session.id}, {observed.name}, {observer.name}, representation)"