* perf(reconciler): only trace Sentry transactions when work is found The reconciler enqueues sync_vectors every ~5 min per deriver instance. process_item wrapped every dequeued reconciler task in a single process_reconciler_task transaction, so idle cycles (the common case, where the cycle finds no rows and exits immediately) still created and sampled a transaction + profile, draining Sentry tracing/profiling quota. Remove the top-level transaction and push tracing into the sync batch helpers, starting a per-batch transaction only after rows are confirmed. Idle cycles now emit zero transactions; busy sweeps emit one smaller transaction per batch operation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(telemetry): drop infra/scrape transactions via a Sentry traces sampler Sentry was sampling every transaction at a flat traces_sample_rate with no sampler. The Prometheus /metrics scrape endpoint alone accounted for ~92% of all traced transactions (and their profiles), with /openapi.json and the deriver metrics server adding more pure noise. Add a traces_sampler that returns 0.0 for infra/scrape endpoints (/metrics, /health, /openapi.json, /docs, /redoc, and metrics/openapi transaction names) and the configured rate for real traffic. Sampling here (vs before_send_transaction) means dropped transactions are never recorded or profiled and the decision propagates to child spans. Shared init covers both the API server and the deriver worker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| README.md | ||
| __init__.py | ||
| conftest.py | ||
| test_deriver_processing.py | ||
| test_embed_now.py | ||
| test_enqueue_dream.py | ||
| test_prompts.py | ||
| test_queue_operations.py | ||
| test_queue_processing.py | ||
| test_representation_crud.py | ||
| test_vector_reconciliation.py | ||
README.md
Deriver Testing
This directory contains tests for the deriver system, which handles background processing of messages to extract insights and update working representations.
Structure
conftest.py- Shared fixtures for deriver testingtest_queue_operations.py- Tests for basic queue operationstest_deriver_processing.py- Tests for deriver processing logictest_queue_processing.py- Tests for queue manager and work unit processing
Key Fixtures
Database Fixtures
sample_session_with_peers- Creates a session with multiple peers having different observation configurationssample_messages- Creates sample messages for testingsample_queue_items- Creates queue items with various payload types (representation, summary)
Queue Fixtures
create_queue_payload- Helper to create queue payloads for testingadd_queue_items- Helper to add queue items to the databasecreate_active_queue_session- Helper to create active queue sessions for work unit tracking
Mocking Fixtures
mock_critical_analysis_call- Mocks the critical analysis LLM callmock_queue_manager- Mocks the queue manager for testingmock_representation_manager- Mocks the representation manager operations
Testing Patterns
Creating Queue Items
# Create representation payloads
payload = create_queue_payload(
message=message,
task_type="representation",
observer=observer_peer.name,
observed=message.peer_name
)
# Add to queue
queue_items = await add_queue_items([payload], session.id)
Testing Work Units
# Create a work unit
work_unit = WorkUnit(
session_id=session.id,
task_type="representation",
observer=observer,
observed=observed
)
# Test string representation
assert str(work_unit) == f"({session.id}, {observed.name}, {observer.name}, representation)"