39 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Vineeth Voruganti
|
07c79e20e3 |
fix(db): lazy retrying session + review fixes for connection backoff
Address Codex/CodeRabbit review on PR #758. - Replace eager checkout with HonchoAsyncSession: a lazy AsyncSession that checks out its connection (with retry) on the first DB-touching call, not at construction. Request handlers doing non-DB work (embedding/file/LLM) before their first query no longer pin a connection across it, while the API path still gets checkout retry. Only the checkout is retried — the statement runs once via super(), so writes are never duplicated. Tracing's set_config moves into the same lazy acquire hook. - Roll the session back on a retryable checkout failure before retrying, so a failed autobegin can't leave it pending-rollback. - Lower default POOL_TIMEOUT to 5s and validate it stays under the retry budget for pooled (non-null) POOL_CLASS; update config.toml.example and v2/v3 docs. - Clamp pool overflow gauge to >= 0 (was negative before the pool fills). - Remove double-sleep in the deriver idle poll (true backoff cap, not 2x); make in-flight instrumentation registration idempotent. - Tests: HonchoAsyncSession lazy/idempotent acquire, statement-runs-once, tracing, commit/rollback flag reset, get_db no-acquire-at-entry, polling-loop single-sleep, and the POOL_TIMEOUT/retry-budget validator. |
|
|
Vineeth Voruganti
|
6cd96b065f |
feat(db): add connection retry, adaptive deriver polling, and pool metrics
Add resilience and visibility for DB connection handling under transaction- pooler (Supavisor) saturation, where client-connection limits get exhausted across many tenants. - get_db/tracked_db now force an eager pool checkout with bounded exponential backoff (tenacity), retrying SQLAlchemy TimeoutError + OperationalError so transient pooler rejections degrade gracefully instead of 500ing. Toggle via DB_CONNECTION_RETRY_ENABLED (+ delay/backoff knobs); ~10s default budget. - Deriver polling backs off when idle or erroring (base -> max, x2 each cycle) and snaps back to base on claimed work, cutting steady-state query load. Toggle via DERIVER_POLLING_BACKOFF_ENABLED (+ max/multiplier). - Add scrape-time db_pool_connections Prometheus gauge (checked_out/checked_in/ size/overflow, labeled api|deriver), registered in both the API lifespan and the deriver metrics server. - Make SqlalchemyIntegration explicit in both Sentry inits; wrap connection acquisition in a db.pool.acquire span and capture live pool stats on retry-exhaustion. |
|
Rajat Ahuja
|
b5f24a6ac5
|
feat: add new cloudevents for api routes (#637)
* feat: add new cloudevents for api routes
* fix: add total input tokens to RepresentationCompletedEvent
* feat(telemetry): inject honcho_version + emitter health metrics
* feat(telemetry): per-LLM-call event with try/finally emission + sampler
Adds LLMCallCompletedEvent (llm.call.completed) — fires once per provider hit
with full cost-attribution context: transport/provider_label, model, token
counts with cache breakdown, finish_reason, outcome (success or error),
is_final_attempt flag, retry/fallback state, duration, tool-call shape,
streaming flag, and agent correlation (run_id + iteration).
- src/telemetry/events/llm.py: new event class + CallPurpose closed enum
(deriver.representation, dialectic.answer, dream.deduction|induction,
summary.short|long). Resource id includes attempt so multi-attempt retries
in one iteration get distinct deterministic ids.
- src/telemetry/events/base.py: BaseEvent._volume_class ClassVar (default
"ground_truth"); the new event opts into "high_volume".
- src/config.py: TelemetrySettings.HIGH_VOLUME_SAMPLE_RATE (default 1.0).
- src/telemetry/emitter.py: deterministic sampler keyed on run_id (so an
entire agent trace is kept or dropped together). Aggregate envelopes
bypass the sampler. Sampled-out events increment the dedicated counter
separate from buffer_full/send_failed drops.
- src/llm/runtime.py: AttemptPlan gains attempt/retry_attempts/is_fallback
so the executor reads retry state without re-deriving it.
- src/llm/types.py: LLMTelemetryContext dataclass carrying workspace,
call_purpose, run_id, iteration, peer fields. Iteration is mutable so
the tool loop can set it per inner call.
- src/llm/executor.py: honcho_llm_call_inner wraps the backend call in
try/finally — emits on success AND on exception, with is_final_attempt
computed from AttemptPlan. Stream path emits a was_stream=True placeholder
(token totals deferred until streaming completion is wired through).
Telemetry failures swallowed.
- src/llm/api.py: threads telemetry kwarg through all 4 signatures into
both honcho_llm_call_inner and execute_tool_loop.
- src/llm/tool_loop.py: _telemetry_for_iteration helper copies the caller
context with iteration set per call — covers both the normal iteration
loop AND the max-iteration synthesis call (iteration N+1).
Tests cover success/error emission, sampler trace-coherence (same run_id →
same decision), volume_class enforcement, unknown call_purpose tolerance,
provider_label inference, and telemetry failure isolation. 378/378 pass.
* feat(telemetry): emit agent.iteration on every LLM response + synthesis
AgentIterationEvent was defined but never emitted on this branch. Phase 2
wires it up in execute_tool_loop so every LLM call inside an agentic loop
produces one event — including the no-tool terminating iteration and the
max-iteration synthesis call — and threads LLMTelemetryContext from dialectic
and dreamer specialists down through honcho_llm_call.
- src/telemetry/events/agent.py: AgentIterationEvent opts into
_volume_class="high_volume" so the Phase 1 sampler throttles it.
- src/llm/tool_loop.py: _emit_agent_iteration() helper fires once per
honcho_llm_call_inner response, BEFORE the no-tool early return so the
terminating iteration is counted. A second emission fires for the
max-iteration synthesis call BEFORE final_response is mutated with
cumulative totals (otherwise the per-iteration counts would double-count).
Emission is defensively skipped when telemetry context lacks run_id /
agent_type / parent_category / workspace_name; emit failures are swallowed.
- src/dreamer/specialists.py: BaseSpecialist.run passes LLMTelemetryContext
with parent_category="dream", agent_type=self.name, observer/observed,
call_purpose=f"dream.{self.name}".
- src/dialectic/core.py: _telemetry_context() builds a shared context for
both answer() and answer_stream(), using self._run_id (always set) +
workspace + observed peer.
Tests cover fresh-copy semantics, per-iteration vs terminating emission,
defensive skip cases, telemetry-failure isolation, and volume_class. 408/408
pass across telemetry + llm + utils + dreamer + dialectic.
* feat(telemetry): agent.tool.call.completed event + ToolResult metadata
Adds the missing generic per-tool-call event so read-only tools (search_*,
get_recent_history, get_observation_context, etc.) and the four existing
state-change tools all produce a telemetry record. Built on a new internal
ToolResult(content, metadata) contract so handlers can surface
search-specific fields (top_k/used_embedding/query_tokens/results_count)
to Phase 3 and create/delete counts to Phase 5's specialist rollups.
- src/telemetry/events/agent.py: AgentToolCallCompletedEvent at v1 with
_volume_class="high_volume". Resource id = {run_id}:{iteration}:{tool_call_seq}
so two calls to the same tool in one iteration don't collide
deterministic ids and get dedup-dropped downstream.
- src/utils/types.py: ToolResult dataclass; two new ContextVars
(_current_tool_call_seq + _last_tool_metadata) so tool_loop and the
execute_tool closure can communicate per-call telemetry without changing
the public Callable[[str, dict], Any] signature.
- src/utils/agent_tools.py: execute_tool times handlers, unwraps ToolResult,
publishes metadata, emits the event. Handlers updated to ToolResult
where useful: create/delete observations, update_peer_card, search_memory,
search_messages. Other handlers continue to return str.
- src/llm/tool_loop.py: set_current_tool_call_seq before each executor call;
read get_last_tool_metadata after and stash on all_tool_calls[i] for
Phase 5 rollups.
Tests cover ToolResult str-likeness, ContextVar round-trip, full-context
emission with search metadata, resource-id disambiguation, defensive skip
cases, telemetry isolation, truncation metadata, volume_class. 420/420 pass.
* feat(telemetry): RepresentationCompletedEvent v2 token breakdown + tool-less truncation
Bulks out the deriver's per-batch telemetry without bumping the event schema
version. New additive fields capture the full token breakdown (queued vs.
extra-context vs. scaffold), the cap configuration (batch_max_tokens,
max_input_tokens, was_flush_enabled), real cap-hit flags, and observer
fanout. `input_tokens` stays unchanged as the queued-message-tokens billing
key Xatu's Stripe meter reads.
The big enabler: src/llm/api.py now actually enforces max_input_tokens on
the tool-less LLM path. Before this, the deriver passed the kwarg but the
path silently dropped it — so the configured cap was advisory and
hit_input_token_cap couldn't be measured. Phase 4 wires truncation through
the same truncate_messages_to_fit helper the tool loop uses and surfaces
input_was_truncated on HonchoLLMCallResponse.
- src/telemetry/events/representation.py: 12 additive fields, schema_version
stays at 2.
- src/llm/types.py: input_was_truncated on HonchoLLMCallResponse.
- src/llm/api.py: tool-less path truncates messages before dispatch, flips
input_was_truncated on the response when clamping occurs. Split into
Literal[True]/Literal[False] branches for typecheck.
- src/deriver/queue_manager.py: QueueBatchResult dataclass replaces the
3-tuple return from get_queue_item_batch; carries hit_batch_token_cap
(computed from cumulative token sum vs cap), was_flush_enabled snapshot,
and batch_max_tokens. Worker loop unpacks + forwards.
- src/deriver/consumer.py: process_representation_batch gains the three
flag kwargs and forwards.
- src/deriver/deriver.py: derives the breakdown fields locally, populates
the new fields on emit, sources hit_input_token_cap from
response.input_was_truncated.
Tests cover schema stability, defaultable fields, input_tokens semantic
preservation, cap-hit flag round-trip, model_dump completeness, and
HonchoLLMCallResponse.input_was_truncated mutability. Existing
test_queue_processing.py tests updated for QueueBatchResult and mock
process_representation_batch signature. 479/479 pass.
* feat(telemetry): DreamRunEvent v2 scheduler reasons + DreamSpecialistEvent v2 rollups
Bumps both dream events to v2 with additive fields. DreamRunEvent gains
scheduler context (threshold_reason / delay_reason / documents_since_last_dream_at_schedule /
document_threshold / dream_type / enabled_types_count) threaded through the
dream queue payload — the two scheduler gates stay as separate fields rather
than collapsing into one trigger_reason, preserving the WHY-vs-WHEN
semantics. DreamSpecialistEvent gains denormalized rollups
(created_observation_count / deleted_observation_count / peer_card_updated /
search_tool_calls_count) sourced from Phase 3's ToolResult.metadata so the
counts reflect observation truth, not call truth.
- src/telemetry/events/dream.py: schema_version → 2 for both events; new
fields all defaultable so older producers still construct valid events.
- src/utils/queue_payload.py: DreamPayload + create_dream_payload accept
threshold_reason / delay_reason / documents_since_last_dream_at_schedule /
document_threshold.
- src/dreamer/dream_scheduler.py: check_and_schedule_dream computes the two
reasons at decision time and threads them through schedule_dream →
_delayed_dream → execute_dream → enqueue_dream.
- src/deriver/enqueue.py: create_dream_record / enqueue_dream gain the
kwargs and persist on the queue payload.
- src/dreamer/orchestrator.py: process_dream unpacks the payload; run_dream
accepts the kwargs and stamps them on DreamRunEvent.
- src/dreamer/specialists.py: BaseSpecialist.run walks response.tool_calls_made
and sums ToolResult.metadata.created_count / .deleted_count, sets
peer_card_updated, counts search-tool calls by name.
Tests cover schema_version bumps, defaultable Phase 5 fields,
threshold-vs-delay semantics, observation-vs-call-count rollup distinction,
and DreamPayload round-trip. Existing tests updated for the schema bump
and the new enqueue_dream kwargs. 488/488 pass.
* feat(telemetry): AgentToolSummaryCreatedEvent v2 token breakdown
Bumps schema_version to 2 and adds three additive breakdown fields so
analytics can answer "how much of a summary call's cost was the previous-
summary rollup vs. the new messages vs. the scaffold instructions".
- src/telemetry/events/agent.py: previous_summary_tokens, message_tokens,
prompt_scaffold_tokens added with sensible 0 defaults. input_tokens
retains its current semantic (provider-side LLM tokens) — the plan's
proposed `provider_input_tokens` was omitted because input_tokens
already serves that purpose and a duplicate would fork queries.
- src/utils/summarizer.py: emit now populates the three new fields from
values already in scope (messages_tokens, previous_summary_tokens,
prompt_tokens). Hoisted prompt_tokens calculation out of the
is_fallback conditional so both the save-summary path and the emit
share one binding — basedpyright couldn't prove the sibling-scope
binding was safe, and the compute is cheap + idempotent.
Tests cover schema bump, defaultable fields, input_tokens semantic
preservation, first-summary edge case, and breakdown round-trip.
493/493 pass.
* feat(telemetry): embedding.call.completed event + call-purpose ContextVar
Adds the final piece of cost-attribution telemetry: per-embedding-call
events covering every provider hit (single + batch + retry attempts).
Embedding calls are real provider spend that was invisible before this
phase; search-heavy paths (dialectic agentic) can produce more embedding
calls than LLM calls, so the new event participates in the shared
HIGH_VOLUME_SAMPLE_RATE.
- src/telemetry/events/llm.py: EmbeddingCallCompletedEvent at v1 with
_volume_class="high_volume". EmbeddingCallPurpose closed enum
(search_memory / search_messages / create_observations / vector_sync /
summary / message_create). Resource id = run:purpose:provider:model:input_count
so per-iteration calls in one agentic run don't collide.
- src/utils/types.py: _embedding_call_purpose ContextVar plus
@contextmanager wrapper. Nesting-safe via ContextVar.reset(token).
Callers wrap embedding-driving operations in
`with embedding_call_purpose("search_memory"): ...` — no changes to
the embedding client signature.
- src/embedding_client.py: _emit_embedding_call wraps each provider hit
with try/finally so success AND error paths emit. Errors propagate
unchanged. Each retry attempt of _process_batch emits its own event.
Unknown call_purpose slugs drop to None (validation against the enum
happens at emit time, not at context-manager-set time).
- src/utils/agent_tools.py: search_memory / search_messages /
search_messages_temporal / create_observations (batch + fallback) all
tag their embedding calls.
- src/crud/representation.py: save_representation tags with
CREATE_OBSERVATIONS; get_working_representation precompute tags with
SEARCH_MEMORY.
- src/crud/message.py: create_messages batch embed tags with
MESSAGE_CREATE; search_messages/temporal fallback tags with
SEARCH_MESSAGES.
Tests cover event shape, enum closure, ContextVar nesting/exception
cleanup, wrapper success+error emission, unknown-purpose graceful
fallback, telemetry-failure isolation. 550/550 pass across the full
telemetry+llm+utils+dreamer+dialectic+deriver+crud test set.
* chore: fix tests
* fix(telemetry): address review findings on stream events, context propagation, and cap detection
Five findings from a post-Phase-7 review (one resolved by the merge from
main, four addressed here):
- src/llm/executor.py: stream-path LLMCallCompletedEvent now fires AFTER
the stream is set up and drained (or on exception), with real duration
and accurate outcome. Previously the event was emitted before
execute_stream() ran and was always recorded as outcome="success" with
duration_ms=0, which silently masked stream-setup and stream-drain
failures. Wrapping the async generator in try/finally surfaces the real
outcome; token counts stay 0 because we still don't have them at stream
end (aggregate envelopes carry totals).
- src/deriver/deriver.py + src/utils/summarizer.py: deriver and summarizer
LLM calls now thread LLMTelemetryContext into honcho_llm_call. Before
this, the closed CallPurpose enum had DERIVER_REPRESENTATION /
SUMMARY_SHORT / SUMMARY_LONG slugs but those production call sites
didn't actually pass `telemetry=`, so their LLMCallCompletedEvents lost
workspace_name, parent_category, and call_purpose. summarizer threads
workspace_name through _create_and_save_summary → _create_summary →
create_short_summary / create_long_summary.
- src/utils/types.py + src/embedding_client.py: embedding_call_purpose
ctx manager now accepts workspace_name and run_id kwargs, backed by
two new ContextVars. EmbeddingCallCompletedEvent's publisher reads
both via get_embedding_workspace_name / get_embedding_run_id so
embedding events carry workspace and run correlation. All call sites
updated: search_memory / search_messages / search_messages_temporal /
_handle_create_observations_impl pass ctx.workspace_name +
ctx.run_id; create_observations standalone and create_messages pass
workspace_name; RepresentationManager.save_representation and
get_working_representation pass self.workspace_name.
- src/deriver/queue_manager.py: hit_batch_token_cap detection rewritten.
Previously summed kept-rows' token_count and checked against
batch_max_tokens, but the SQL filter `cumulative_token_count <= cap`
guarantees kept rows stay under the cap, so the flag almost never
fired. Now uses two follow-up queries: total token_count across the
included id range + EXISTS check for any session message past the
last-kept id. Both true → cap was actually binding.
(The fifth finding — deriver scaffold-token computation needing
estimate_deriver_prompt_tokens(custom_instructions) — was resolved by
the merge from main; the Phase 4 emit at src/deriver/deriver.py:283
already sources prompt_scaffold_tokens from the wrapped helper.)
567/567 telemetry+llm+utils+dreamer+dialectic+deriver+crud tests pass.
ruff + basedpyright clean.
* chore: ruff linting
* chore: clean AI generated comments references specs
* fix: address coderabbit changes
* fix(telemetry): address remaining PR review findings
Six findings from the PR 637 telemetry review batched into one commit.
- src/llm/executor.py + src/embedding_client.py: asyncio.CancelledError
now surfaces as outcome="cancelled" on both stream and sync paths,
distinct from "error". Client disconnects mid-stream and server
shutdowns are normal control flow and should not feed error-rate
alerting. LLMCallCompletedEvent and EmbeddingCallCompletedEvent
outcome Literal extended; docstrings + tests cover the new state.
- src/utils/types.py + src/llm/tool_loop.py: new iteration_scope()
context manager captures and resets the four per-tool-loop
ContextVars (_current_iteration, _current_tool_call_seq,
_current_provider_tool_call_id, _last_tool_metadata). Applied as a
typed decorator to execute_tool_loop so back-to-back loops in the
same asyncio Task (worker batches, tests) don't observe stale state.
- src/telemetry/events/api.py + src/routers/messages.py:
MessageCreatedEvent schema v1 → v2. Added required last_message_id
(nanoid public_id of the trailing message); get_resource_id now keys
on it instead of message_count, eliminating the collision case where
two same-size batches in the same session+source produced identical
event ids. message_count stays on the body for analytics.
- src/deriver/queue_manager.py: hit_batch_token_cap now computed from
the FINAL post-config-filter batch. Previously the flag used the
pre-filter messages_context[-1].id, which produced false positives
when _resolve_batch_configuration trimmed the trailing queue item —
telemetry reported a cap-hit when the actual returned batch was
short for unrelated reasons. Cap-detection block moved inside the
async with after the filter; no extra DB connection.
- src/config.py + src/telemetry/emitter.py: documented the
HIGH_VOLUME_SAMPLE_RATE orphan trade-off (rate<1.0 keeps aggregates
but drops children, so JOIN ON run_id queries see partial traces).
Behavior unchanged — rate defaults to 1.0.
- src/deriver/deriver.py: WARNING-level invariant logs when
response.input_tokens < messages_tokens (provider tokenization
drift) or prompt_scaffold_tokens <= 0 (estimator silent failure).
Best-effort — telemetry never bleeds into the deriver path
* fix(telemetry): stream retry, embed attempts, truncation, dedup
Address remaining audit findings on the cloudevents PR:
- Stream setup now runs inside the awaited honcho_llm_call_inner so
tenacity's retry wrapper in stream_final_response catches transient
setup failures (rate-limit, auth, network). Previously the returned
generator deferred execute_stream until first iteration — outside
the retry wrapper — crashing the request and bypassing telemetry.
- Embedding _emit_embedding_call gains an is_final_attempt parameter;
_process_batch threads the real retry index so dashboards stop
conflating one-shot, mid-retry, and exhausted-retry calls.
- _truncate_tool_output returns (text, original_chars, was_truncated)
and a new _maybe_truncated_result helper wraps in ToolResult when
truncation happens. Five handlers migrated. AgentToolCallCompletedEvent
fields was_truncated and result_chars_before_truncation are now
populated instead of always None/False.
- execute_tool_loop tracks any_iteration_truncated and stamps
input_was_truncated on the final response (both HonchoLLMCallResponse
and StreamingResponseWithMetadata). Dialectic now reports
hit_input_token_cap correctly.
- GetContextEvent.get_resource_id uses empty-string sentinel instead
of literal "none" so a peer named "none" can't collide with absent.
- generate_event_id folds honcho_version into the deterministic id so
same logical event from different deploys produces distinct ids.
* fix(telemetry): address audit findings across LLM/embed/event paths
Three rounds of telemetry audit findings, grouped by area:
Retry correctness
- Stream LLM setup now runs inside the awaited honcho_llm_call_inner so
tenacity's outer retry catches setup failures (Fix 1). Previously the
inner generator deferred execute_stream past the retry wrapper.
- stream_final_response bumps the per-retry attempt index via
dataclasses.replace so emitted events show [1, 2, 3] instead of
[1, 1, 1] (Fix 13).
- Embedding _emit_embedding_call takes is_final_attempt; _process_batch
threads the real retry index (Fix 2).
Token + cost reporting
- HonchoLLMCallResponse.hit_input_token_cap (renamed from
input_was_truncated) uses a token-based rule so single-message
over-cap inputs are correctly flagged — the deriver's prompt-only
path used to silently fly through. Propagated through tool_loop's
per-iteration check (Fix 4) and into RepresentationCompletedEvent.
- DialecticCompletedEvent gains hit_input_token_cap; output_tokens now
folds in the final-stream's cumulative usage via
StreamingResponseWithMetadata.__aiter__ (Fix 7).
Event emission completeness
- AgentToolCallCompletedEvent's was_truncated /
result_chars_before_truncation populated by _truncate_tool_output via
a new _maybe_truncated_result wrapper; 5 handlers migrated (Fix 3).
- DreamSpecialistEvent emits on failure with success=False + new
error_class field, via try/finally (Fix 11).
- DeletionCompletedEvent emits on failure paths via try/finally
(Fix 12).
- CleanupStaleItemsCompletedEvent.queue_items_cleaned populated from
deleted_count (Fix 8).
Embedding call attribution (Fix 9)
- embedding_call_purpose context manager accepts parent_category.
- 4 new EmbeddingCallPurpose enum values: DIALECTIC_PREFETCH,
SESSION_CONTEXT_SEARCH, PREFERENCE_EXTRACTION, GENERIC_DOCUMENT_SEARCH.
- Wrapped previously-unattributed sites: dialectic prefetch, session
context search, preference extraction, conclusions search, vector
sync (×2).
Deterministic event ID + dedup
- generate_event_id folds honcho_version into the hash so cross-deploy
events don't silently collide on ID (Fix 6).
- GetContextEvent resource_id uses empty-string sentinel instead of
"none" so a peer literally named "none" can't collide (Fix 5).
Queue batch cap detection (P2.1)
- hit_batch_token_cap keys on the pre-config-filter SQL boundary so the
"kept=900 of 1000 cap, next=300 excluded by cap" case reports True
while still avoiding the config-filter false positive.
Tool result metadata
- search_messages_temporal returns ToolResult with the same search_meta
shape as search_memory / search_messages (P2.3) — top_k,
used_embedding, embedding_query_count, query_tokens, results_count.
Tests: stream-setup retry, stream-retry attempt sequence, post-stream
output_tokens write-back, is_final_attempt matrix, truncation E2E,
tool-loop hit_input_token_cap propagation, honcho_version in event id,
GetContextEvent disambiguation, queue_items_cleaned round-trip.
* fix(telemetry): address audit findings across LLM/embed/event paths
Four rounds of telemetry audit findings (initial + 3 follow-ups), grouped
by area:
Retry correctness
- Stream LLM setup now runs inside the awaited honcho_llm_call_inner so
tenacity's outer retry catches setup failures (Fix 1). The inner
generator previously deferred execute_stream past the retry wrapper.
- stream_final_response bumps the per-retry attempt index via
dataclasses.replace so emitted events show [1, 2, 3] instead of
[1, 1, 1] (Fix 13).
- Embedding _emit_embedding_call takes is_final_attempt; _process_batch
threads the real retry index (Fix 2).
Token + cost reporting
- HonchoLLMCallResponse.hit_input_token_cap (renamed from
input_was_truncated) uses a token-based rule so single-message
over-cap inputs are correctly flagged — the deriver's prompt-only
path used to silently fly through. Propagated through tool_loop's
per-iteration check (Fix 4) and into RepresentationCompletedEvent.
- DialecticCompletedEvent gains hit_input_token_cap; output_tokens now
folds in the final-stream's cumulative usage via
StreamingResponseWithMetadata.__aiter__ (Fix 7).
Queue batch cap detection
- hit_batch_token_cap previously required total_in_range >= cap, which
produced false negatives whenever the kept range didn't fully exhaust
the budget. Replaced with a pre-config-filter SQL boundary check
(P2.1), then further refined to a queue-item boundary comparison
(Fix 14) so trailing-context trimming doesn't false-negative either.
Event emission completeness
- AgentToolCallCompletedEvent's was_truncated /
result_chars_before_truncation now populated by _truncate_tool_output
via _maybe_truncated_result; 5 handlers migrated (Fix 3).
- DreamSpecialistEvent emits on failure with success=False + new
error_class field, via try/finally (Fix 11). except BaseException
catches cancellations too (Fix 16).
- DeletionCompletedEvent emits on failure paths via try/finally
(Fix 12), and uses ValidationException for unsupported types per
project guideline (Fix 17).
- CleanupStaleItemsCompletedEvent.queue_items_cleaned populated from
deleted_count (Fix 8).
Embedding call attribution (Fix 9)
- embedding_call_purpose accepts parent_category.
- 4 new EmbeddingCallPurpose values: DIALECTIC_PREFETCH,
SESSION_CONTEXT_SEARCH, PREFERENCE_EXTRACTION, GENERIC_DOCUMENT_SEARCH.
- Wrapped previously-unattributed sites: dialectic prefetch, session
context search, preference extraction, conclusions search, vector
sync (×2).
Reconciler no longer holds DB session during embedding (Fix 15)
- _sync_documents and _sync_message_embeddings refactored into
three phases per CLAUDE.md guideline: fetch+detach in a small DB
scope, external embedding call without DB locks, writes in a fresh
short-lived DB scope. New _apply_*_sync helpers; orchestrators
expunge ORM objects before invoking. Vector store upsert + sync_state
updates stay in the apply phase together.
Deterministic event ID + dedup
- generate_event_id folds honcho_version into the hash so cross-deploy
events don't silently collide on ID (Fix 6).
- GetContextEvent resource_id uses empty-string sentinel instead of
"none" so a peer literally named "none" can't collide (Fix 5).
Tool result metadata
- search_messages_temporal returns ToolResult with the same search_meta
shape as search_memory / search_messages (P2.3).
- Dialectic.prefetched_conclusion_count uses Representation.len() so
inductive + contradiction observations count too (Fix 10).
* fix(telemetry): orchestrator emit + review feedback
Three more rounds of audit findings + inline PR review, grouped:
Orchestration / emit reliability
- run_dream wrapped in try/finally so DreamRunEvent always emits, even
on unexpected exceptions including CancelledError (`finally` still
runs while cancellation propagates). Specialist except clauses
broadened from SpecialistExecutionError (never raised in src/) to
Exception so provider/DB/tool failures are recorded with
deduction_success=False / induction_success=False instead of crashing
past the emit.
- BaseSpecialist.run() telemetry state initialization + try/finally
hoisted above the preflight phase (peer lookup, peer-card preload,
create_tool_executor, get_model_config, prompt construction) so
preflight failures emit DreamSpecialistEvent(success=False) instead
of being dropped on the floor.
- Reverted the Round-4 _sync_documents / _sync_message_embeddings
phase split. The split introduced a race: rows were released from
FOR UPDATE SKIP LOCKED before the embed call, allowing two workers
to claim and clobber the same batch. Long-held DB transaction
restored (pre-existing CLAUDE.md violation accepted as a deliberate
trade-off; proper fix requires a claim/in_flight migration tracked
separately).
Schema + naming (PR-internal — none of these have shipped)
- threshold_reason → trigger_reason on DreamRunEvent, DreamPayload, and
every emit/scheduler/router/test call site (~45 src + 21 test lines).
Name now accurately reflects the field's role across "manual",
"surprisal", and "document_threshold" values.
- MessageCreatedEvent reset to schema v1 (was internally bumped to v2
for last_message_id but never shipped at v1 — downstream sees it
for the first time at merge).
- DreamSpecialistEvent gains created_counts_by_level /
deleted_counts_by_level: dict[str, int] keyed on the closed
level taxonomy. Per-tool-call events use list[str] (≤10 items),
but specialist runs aggregate 20+ — dict keeps emissions compact.
- QueueBatchResult marked frozen=True.
Per-call embedding attribution
- Agent tool embedding_call_purpose wraps for search_memory,
search_messages, search_messages_temporal, create_observations now
driven embedding cost rolls up under the right workflow.
- create_observations() signature gains parent_category kwarg
(mirrors existing run_id pattern).
Manual dream scheduling
- Manual /schedule_dream route now passes trigger_reason="manual" and
delay_reason="immediate". Previously both arrived as null in
DreamRunEvent, breaking analytics joins.
Queue-batch SQL perf
- next_exists_check folded into the main CTE query via
bool_or(cumulative_token_count > batch_max_tokens) OVER () in a
nested subquery. Cap detection is now one roundtrip per batch
instead of two.
Code/doc cleanup
- representation.py docstring uses generic "downstream metering key"
language (was "Xatu's Stripe meter"). bench runner --base-url help
uses a generic example host (was "groudon.fly.dev"). Public-facing
code/docs shouldn't reference internal service names.
Tests added for: orchestrator failure-path DreamRunEvent emission,
specialists preflight try/finally coverage, manual-dream
trigger_reason/delay_reason round-trip, dict-rollup accumulation across
multiple tool calls in a specialist run, CTE-fold one-roundtrip
behavior. Full Python suite passes (1236).
* fix(telemetry): correctness + attribution + emitter robustness
- Dreamer iteration count: read response.iterations directly so
one-shot runs no longer report iterations=0 and tool-using runs
include the terminal/synthesis LLM call.
- RepresentationCompletedEvent.observer_count counts successful
saves, not attempts.
- search_memory empty-memory fallback reports the snippet count when
message context is returned (was always 0).
- Wire parent_category through every embedding emit path: message
create (api), save_representation (representation), per-observation
fallback (caller-supplied), and the peer/session context routes
(api). get_working_representation accepts parent_category and
embedding_purpose so the internal fallback embed lands in the same
analytics bucket as the route-level precompute even when the
precompute is suppressed.
- BatchItem carries token_count so _process_batch reuses chunk-prep
counts instead of re-encoding every chunk for the telemetry proxy.
- Drop vestigial EmbeddingCallCompletedEvent.batch_size (always ==
input_count).
- Emitter: release the lock during HTTP send so a failing endpoint's
retry+backoff (~36s worst case) doesn't block other flushers;
edge-trigger the 80%-capacity warning so sustained backpressure
doesn't flood logs; defer event_id generation past the high-volume
sampler for events with run_id so sampled-out children don't pay
the sha256; harden emit() against sync callers with no running
loop; track threshold-flush tasks so shutdown() drains in-flight
sends before closing the HTTP client.
* fix(telemetry): tool cancellation emit, nanoid run_ids, version unification
- execute_tool: wrap post-work in finally so AgentToolCallCompletedEvent
fires on CancelledError; explicit handler sets is_error/result_str
before re-raising.
- run_id: replace str(uuid.uuid4())[:8] with generate_nanoid() across
dialectic/dreamer/specialists; matches project-wide nanoid convention.
- Bump _schema_version on events touched by run_id widening:
DialecticCompletedEvent v1→v2 (also covers hit_input_token_cap field),
AgentIterationEvent v1→v2, AgentToolConclusionsCreatedEvent v1→v2,
AgentToolConclusionsDeletedEvent v2→v3, AgentToolPeerCardUpdatedEvent
v1→v2.
- Unify honcho_version: single HONCHO_VERSION constant in src/_version.py
read from pyproject.toml (importlib.metadata fallback). Drop
TELEMETRY.HONCHO_VERSION setting. Use the constant for the FastAPI app
version (no more hardcoded "3.0.6") and for emitter body injection.
- Delete 17 tautological per-event test_schema_version methods; the
parametrized contract test still enforces version >= 1 across all events.
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
|
|
|
Vineeth Voruganti
|
5b6bd59030
|
Tighten Transaction Scopes (#525)
* fix: further remove extraneous transactions * fix: (search) use 2 phase function to reduce un-needed transaction * fix: refactor agent search to perform external operations before making a transaction * fix: reduce scope of queue manager transaction * fix: (bench) add concurrency to test bench * fix: address review findings for search dedup, webhook idempotency, and bench throttling * Fix Leakage in non-session-scoped chat call (#526) * fix: (search) reduce scope for peer based searches * fix: tests * fix: (test) address coderabbit comment * fix: drop db param from deliver_webhook --------- Co-authored-by: Rajat Ahuja <rahuja445@gmail.com> |
|
|
Vineeth Voruganti
|
78df86dc66
|
fix: Remove noisy sentry error on llm failure and upgrade deps (#396) | |
|
Rajat Ahuja
|
d3637a9d73
|
fix: make FLUSH_ENABLED a config value (#361)
* fix: use cashews for DERIVER_FLUSH_KEY * fix: make FLUSH_ENABLED a config value |
|
|
Rajat Ahuja
|
110787cdca
|
feat: use messages from queue items for rep completed token count (#350) | |
|
Rajat Ahuja
|
2270e5666f
|
switch OTEL metrics to prometheus (#344)
* feat: replace OTEL with Prometheus * fix: second pass of docs and cleanup |
|
doria
|
dce96889bc
|
feat: honcho 3.0, sdks 2.0, excise stainless, update v3 docs, changelogs (#331)
* chore: 3.0 honcho and 2.0 sdks changelog fix: use PeerContextResponse in peer.ts * chore: move docs to /v3/, build SDKs * chore: code review * feat: [WIP] migrate away from stainless in typescript sdk * chore: move api from /v2/ to /v3/ * feat: no-stainless typescript with real tests * feat: migrate python sdk off of stainless * feat: clean typescript sdk * chore: add tests for ts http client * fix: rewrite entire python sdk in new format, update typescript sdk to use `configuration` not `config` for consistency with API * fix: clean up SDKs, synchronize * chore: update sdk examples * chore: update OpenAPI documentation and SDK examples to reflect changes * fix: better test * fix: install deps in test runner, improve robustness of streaming in sdk, coderabbit nits * fix: standardize around camelCase in TS SDK * refactor: update configuration handling in SDKs to use typed models for workspace, session, and peer configurations * docs: clarify queue status usage and remove polling methods from SDKs add claude skills for migrations * chore: fix links in docs * feat: add deriver flush mode to bypass batch token threshold - Introduced `is_deriver_flush_enabled` function to check if flush mode is active. - Updated `QueueManager` to conditionally apply batch token thresholds based on flush mode. - Enhanced `UnifiedTestExecutor` to enable flush mode via Redis. - Added `flush` parameter to test cases to facilitate testing of flush mode behavior. - Updated various test cases to utilize the new flush functionality. * feat: implement schedule_dream functionality in SDKs, use in unified test runner - Added `schedule_dream` method to both Python and TypeScript SDKs for scheduling dream tasks. - Updated HTTP routes to include endpoint for scheduling dreams. - Enhanced test runner to utilize the new `schedule_dream` method for scheduling actions. - Updated TypeScript client to support the new scheduling functionality with appropriate parameters. * feat: update single deriver task to support multiple observers - Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module. - Updated the processing logic to handle multiple observers for representation tasks. - Adjusted related payload and queue management functions to accommodate the new observers structure. - Modified tests to reflect changes in the representation task handling and ensure proper functionality. * refactor: update enqueue tests to support deduplication of queue items with multiple observers - Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers. - Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic. - Removed redundant payload matching logic to streamline test cases and improve clarity. * fix: add backwards compatibility for representation work unit keys and payload observers * feat: update dialectic configuration and introduce cost calculator - Adjusted LLM and dialectic settings in `.env.template`, `config.toml.example`, and `src/config.py` to reduce maximum tool output characters and session history tokens for cost efficiency. - Implemented a new `dialectic_cost_calculator.py` script to estimate costs based on reasoning levels and model pricing. - Enhanced `DialecticAgent` to utilize minimal tools and adjusted output token settings based on reasoning level to optimize performance and reduce costs. * feat: add reasoning level to chat input in unified test runner - Enhanced the `UnifiedTestExecutor` to include a `reasoning_level` parameter in the chat method call. - Updated the `QueryAction` model to support the new `reasoning_level` attribute, allowing for more nuanced chat interactions. * feat: run deriver once for multiple observers (#335) * feat: update single deriver task to support multiple observers - Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module. - Updated the processing logic to handle multiple observers for representation tasks. - Adjusted related payload and queue management functions to accommodate the new observers structure. - Modified tests to reflect changes in the representation task handling and ensure proper functionality. * refactor: update enqueue tests to support deduplication of queue items with multiple observers - Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers. - Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic. - Removed redundant payload matching logic to streamline test cases and improve clarity. * fix: add backwards compatibility for representation work unit keys and payload observers * feat: refactor benchmark runners to share common functionality - Introduced a new `runner_common.py` module containing shared utilities for benchmark test runners, including common argument parsing, client creation, and queue management. - Updated `BEAMRunner`, `LoCoMoRunner`, and `LongMemEvalRunner` to inherit from `RunnerMixin`, leveraging shared functionality for metrics collection and logging. - Added `reasoning_level` and `redis_url` parameters to runner constructors for enhanced configuration. - Streamlined argument parsing by utilizing `add_common_arguments` for shared command-line options across all runners. * fix: update last_user_message handling to use message content instead of ID * fix: standardize config vs configuration --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
Vineeth Voruganti
|
6b3ecef601
|
Telemetry Overhaul (#333)
* chore: Refactor a top-level telemetry folder * fix: Add OTEL push based metrics * chore: cleanup otel to match prometheus implementation * fix: Remove prometheus * feat: Scaffold CloudEvent Emitter * chore: remove dead code * feat: PoC Cloud Events * chore: cleanup test * fix: Event naming conventions and OTEL Settings * fix: Revamped Dream Event Structure * fix: Instrument Event Code * fix: Add Tests for CloudEvent Telemetry * fix: Code Rabbit Nits * fix: Dedupe Event IDs |
|
|
Rajat Ahuja
|
833a89e70a
|
Turbopuffer and LanceDB Integration (#287)
* feat: init turbopuffer and lanceDB * fix: remove destructive embedding migration * fix: bug fixes * fix: LanceDB * fix: turbopuffer * fix: search and add create_observations * fix: use Async clients * fix: search; protect agaainst failed vector create/delete * fix: coderabbit comments * fix: set up compose vector store and reconciliation loop * feat: sync docs without embeddings * fix: reduce batch size; comments; types; add indexes for reconciliation * fix: add message embedding resilience * fix: clean-up and migration test * fix: cleanup 2 * fix: centralize retry logic; bump reconciliation batch; use tracked db; fix soft-delete race condition * fix: skip double query when pgvector is primary * fix: down migration * fix: remove hard-delete from critical path and make PgVectorStore deletions a no-op * fix: use soft-delete pattern for duplicate detection * fix: steps toward deprecating MessageEmbedding table * fix: remove composite and pgvector store -> make more specific * fix: migration order * fix: shorten reconciliation cycle + fix 'IN' equality check * fix: coderabbit comments * fix: add test for migration 7c0d9a4e3b1f * feat: refactor to use ReconcilerScheduler * fix: CR / opus comments * fix: work unit key and reserve system workspace * fix: make workspace_name nullable * fix: clean up sync vectors * fix: delete syntax * fix: hash namespace * External Vector Store Nits (#332) * fix: Migration naming and long held connection * chore: Comment for potential debt * chore: update typescript core package --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
doria
|
b0dc05c909
|
feat: message ingestion forced-batching (#327)
* feat: implement forced batching for representation tasks and adjust max tokens - Updated `REPRESENTATION_BATCH_MAX_TOKENS` to 1024 in `config.py`. - Enhanced `get_and_claim_work_units` in `QueueManager` to enforce batching based on token thresholds. - Added tests to ensure representation work units are only claimed when token counts meet or exceed the threshold. - Introduced a synthesis prompt for tool execution to improve final response generation. * chore: update queue-status docs to match new behavior * fix: make representation work query efficient * fix: Align alembic models and force dreams on for tests * fix: clean queue between tests --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
doria
|
578ef2c665
|
feat: agentic dreamer and agentic dialectic (#309)
* feat: add better params to working representation fetch in SDKs, return messages when added * fix: working representation routes now accepting all parameters properly, with tests * feat: add metadata/config fields to SDK objects where viable * fix: tests * feat: refactor SDKs to use representation config; [TEMP STAINLESS BUILD] update API * feat: add representation object to sdks * fix: use stainless sdk on branch * fix: update TypeScript SDK tsconfig to use node16 module resolution * fix: add isolatedModules = true to tsconfig * fix: lol * chore: coderabbit review * feat: make delete session real * feat: add observations routes with delete endpoints for documents. make session deletion real. * chore: type cleanup * fix: tests * chore: coderabbit review * fix: namespace by workspace * feat: add ability to customize messages_per_summary at both workspace and session level * chore: tests for summary config * chore: coderabbit cleanup * feat: make session and workspace config totally customizeable * feat: add search by peer knowledge (#250) * feat: search by peer perspective * fix: enforce workspace in filters, make messages distinct in join * fix: batch and merge migration steps * fix: add refresh, add config to workspace, add refresh function, make fields readonly * fix: search distinct * fix: merge migrations * fix: merge migrations * fix: batch deletions, improve comments, limit consolidate dream to 100 docs at a time, auth on observations routes * chore: review * chore: coderabbit * chore: review * chore: broken comment * feat: add set peer card route to API * feat: create advanced configuration parameters with message>session>workspace hierarchy * [wip] build unified testing harness * chore: lint * fix: cache invalidation, naming things, etc * feat: longmem tests * chore: peer config refactor * feat: consolidate dream working, refactor representation * fix: Various CR Comment Fixes * feat: Allow configurable Redis port for harness instances and update cleanup methods to be asynchronous. * feat: agentic ingestion task!!! * feat: agentic deriver * feat: dialectic agent and dreamer agent * chore: browbeat tests into passing * fix: nits * chore: remove old code, update config files * fix: simplify deriver * feat: dialectic agent prompt updates, re-introduce non_agent deriver, eval tweaks * feat: fast deriver, dreamer, then dialectic * fix: tweaks across the board * feat: add baseline tests * feat: truncation in tools and client, tweaks for evals * feat: add locomo, fix longmem judge!!! * fix: locomo f1 is trash, use llm judge * feat: trace creation * feat: add first draft of obex benchmark, fix embedding model, fix locomo methodology * fix: locomo session-optimized, better logging of cache usage and better cache usage * chore: use openrouter for baselines * fix: add test for merge migration * chore: opus-powered cleanup * fix: add config for vllm, better client * chore: clean up clients.py a bit * chore: move magic numbers to config, add tests for agent tools * fix: wrong mock in dialectic tests, make ToolContext a dataclass * feat: tweak prompts, make deriver explicit-only * feat: more prompt & tool tweaks * chore: more tweaks * feat: dream with subagents * fix: make dream trigger override scheduled, play around with dream agents * chore: cleanup deriver * chore: cleanup dialectic * chore: cleanup orchestrator * chore: comment out dream stuff, WIPing * fix: inc temp on retry, typechecking * feat: tweak dreaming * feat: contradiction obs * Add dream trees * chore: preserve reasoning_details from openrouter in client * fix: get_observation_context correct params * fix: use correct message id in tool * chore: cleanup longmem runner * chore: clean up tests, remove dream tests for now as rearchitecting around trees * chore: update stainless deps * Update threholding mechanism * chore: pre-commit hooks whitespace * chore: clean up types * feat: add explicit bench * fix: address additional basepyright issues * fix: adding logging as a fixture on honcho_llm_call and supporting dialectic loging. (#305) * fix: lock on db for tool calls * chore: clean up experimental derivers * chore: coderabbit review cleanup * feat: add streaming support to agentic dialectic * feat: prometheus token tracking for deriver and dialectic * fix: self-loops for isolated nodes * chore: PascalCase for prometheus parameter typing * feat: add reasoning levels to dialectic agent * chore: delete old file, add new fake env vars in unittest.yml * fix: all fields needed for dialectic reasoning level configs * feat: track dreaming usage in prometheus * chore: Create backwards compatabile conclusion and queue endpoints * fix: remove redundant try-catch, add trace label, move .limit to end of statement * fix: remove vignettes (for now), review fixes, remove merge migration, config cleanup * chore: code review / cleanup * chore: merge fixes * chore: clean up, remove reasoning_focus, reintroduce peer cards in dreamers * chore: code rabbit nitpicks * fix: add unique index for pending dreams in queue * fix: revert removal of surprisal in dreamer config --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> Co-authored-by: 3un01a <3un01a@plasticlabs.ai> Co-authored-by: 3un01a <3un01a.labs@gmail.com> Co-authored-by: ajspig <46900795+ajspig@users.noreply.github.com> |
|
|
doria
|
e3d345b961
|
API/SDK updates: configurability, more parameters. Unified test harness (#283)
* feat: add better params to working representation fetch in SDKs, return messages when added * fix: working representation routes now accepting all parameters properly, with tests * feat: add metadata/config fields to SDK objects where viable * fix: tests * feat: refactor SDKs to use representation config; [TEMP STAINLESS BUILD] update API * feat: add representation object to sdks * fix: use stainless sdk on branch * fix: update TypeScript SDK tsconfig to use node16 module resolution * fix: add isolatedModules = true to tsconfig * fix: lol * chore: coderabbit review * feat: make delete session real * feat: add observations routes with delete endpoints for documents. make session deletion real. * chore: type cleanup * fix: tests * chore: coderabbit review * fix: namespace by workspace * feat: add ability to customize messages_per_summary at both workspace and session level * chore: tests for summary config * chore: coderabbit cleanup * feat: make session and workspace config totally customizeable * feat: add search by peer knowledge (#250) * feat: search by peer perspective * fix: enforce workspace in filters, make messages distinct in join * fix: batch and merge migration steps * fix: add refresh, add config to workspace, add refresh function, make fields readonly * fix: search distinct * fix: merge migrations * fix: merge migrations * fix: batch deletions, improve comments, limit consolidate dream to 100 docs at a time, auth on observations routes * chore: review * chore: coderabbit * chore: review * chore: broken comment * feat: add set peer card route to API * feat: create advanced configuration parameters with message>session>workspace hierarchy * [wip] build unified testing harness * chore: lint * fix: cache invalidation, naming things, etc * feat: longmem tests * chore: peer config refactor * feat: consolidate dream working, refactor representation * fix: Various CR Comment Fixes * feat: Allow configurable Redis port for harness instances and update cleanup methods to be asynchronous. * fix: version bump, api/sdk updates * fix: observation endpoints, deletion queue, sdk observation implementation * chore: Fix migration order * fix: Use published stainless sdks * chore: (docs) update api-reference * fix: (docs) update based on api and sdk changes * fix: Code Rabbit Comments * fix: Code Rabbit Final Nits * fix: dream scheduler * fix: SDK model type consistency --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
Rajat Ahuja
|
d7bdcc3bc1
|
feat: codify queue columns (#254)
* feat: codify queue columns * fix: batch with python loop control * fix: cleanup merge * fix: down revision * fix: batch delete in migration * feat: only run alembic tests for changed migration / test (#264) * feat: only run alembic tests for changed migration / test * fix: Run full test suite if alembic testing infra changes * feat: codify times_derived + level on Document (#260) * feat: codify times_derived + level on Document * fix: CR comment * fix: CodeRabbit comments * fix: batch migrations; move types; remove fields from payload * fix: rm duplicate table args * fix: add messages.id FK |
|
|
Rajat Ahuja
|
8e3f24d4f8
|
feat: add caching (redis) (#265)
* feat: init redis cache * feat: initialize redis client * fix: add fakeredis client * fix: update docker compose * fix: various coderabbit comments * fix: update docker-compose.yml.example * fix: CR comments 2 * fix: clean up client * fix: cache key class; workspace LRU; constants -> utils * fix: CodeRabbit comments * feat: add redis to harness * fix: use cashews * fix: add locking for cache stampedes * fix: clean up PR * f: unify cache key templates and stop caching None * fix: alter prefixes + prime cache on get_or_create() * fix: re-add fakeredis * chore: Code Rabbit nits * CR comments; make client more resilient * fix: BPR * fix: set redis once but clear cache in between every test * fix: attempt fix contextvar error * fix: make TestClient rely on redis patch * fix: potential working fix * fix: namespace inheritance; address comments * fix: namespace inheritance; address comments --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
Rajat Ahuja
|
16c8d0163a | feat: filter validation error from sentry. centralize config | |
|
doria
|
68acf38134
|
Misc: bug fixes, multi-db test harness, DELETE workspace (#230)
* feat: add optional JWT and webhook secrets to honcho instance creation * chore: ignore spurious warnings * feat: add response format if using gpt-5 model family * feat: add response models to all apis except anthropic * fix: raise NotImplementedError for response models in AsyncAnthropic client * chore: address review * [WIP] representation structure + deriver cleanup * chore: add tests, cleanup * feat: [WIP: semi-working] representation object * fix: alignment * fix: make observations hashable for dedup * fix: datetime formatting, observation counting * fix: switch to int for message id, clean up representation * feat: remove need for metadata working rep * chore: cleanup * fix: use tenacity instead of custom fns * feat: add representation and card to context if desired * feat: add semantically relevant observations * fix: pass all params to streaming, nonblocking streaming * feat: consolidate document saving, make working representation fetching much smarter * chore: add 100% test coverage of representation util * feat: basic dream infra * feat: dream queue item first pass * chore: fixes & cleanup from coderabbit * fix: dreams scheduled when new document count reaches a certain threshold * feat: wip: timed dreams (not working) * fix: test * fix: remove useless pyright ignore * fix: executing dreams * feat: dreaming * feat: [WIP] longmemeval bench * feat: add USE_PEER_CARD setting, fix longmem test driver * feat: get full working rep for dialectic in one swoop -- fix representation_from_documents to use the proper timestamp! * fix: timestamps for real, handle assistant qs in longmem * fix: remove old client, add batching to longmem * perf: remove duplicate detection, will move to background task * feat: track perf metrics on evals * feat: adjust deriver prompt to use peer_id, add question date to question, clean up deriver * fix: label metrics by task for better perf trace * chore: code review * feat: add efficiency score to longmem bench * chore: tuning and cleaning up eval * chore: bring in the big prompts * feat: add support for vllm client * feat: perf: bundle db calls in deriver and dialectic, increase max conns in docker db * feat: add merge-sessions flag to longmemeval, add SUMMARY_ENABLED flag * fix: COLLECT_METRICS default false * chore: display start/end message ids, don't include in metrics * fix: break large messages apart for eval * fix: only get/create collection when needed * feat: properly attribute documents with message id ranges and add session name column to documents * fix: revert move of get_or_create_collection (need for fkey) * fix: always get collection with peer name even if it's none * chore: coderabbit * fix: give peer card its own config, expand document schema, refactor get_context to be parallel, various cleanup chores and bugfixes * chore: refactor: reify observer/observed system across entire codebase, including db migration * refactor: cleanup code organization, make singletons where desired * refactor: replace embeddings store with representation manager * chore: coderabbit cleanup * feat: multi-db longmem harness * Merge branch 'main' into ben/multi-db-harness * [WIP] feat: add delete workspace endpoint, use in bench * chore: move excess logging to debug feat: improve metrics block logs to include more data fix: make longmem db deletion configurable * fix: [CRITICAL] use async genai client * chore: update core sdk, fix tests to use aio as well * fix: rollback prompt changes * chore: update version * fix: cleanup, coderabbit, wrap delete op in try/except --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
Rajat Ahuja
|
f38230fd92
|
Integrate prometheus into Honcho (#226)
* feat: add prometheus to api routes * fix: rm multiproc setup * feat: add prometheus to deriver * fix: rm deriver-specific counters (for now) * fix: dangling issues * feat: skip metrics endpoint; track dialectic calls; track individual message creation * fix: CR comments and use fastapi route template. also fix dialectic call * fix: revert to existing request_id logic * fix: rm session_name from dialectic call * feat: add deriver task metric * fix: rm unused get_namespace * rm: namespace param * feat: add MESSAGE_INPUT_TOKENS and DERIVER_TOKENS_PROCESSED counters * fix: rm message input tokens * fix: PR comments; docstrings; error handle metrics exposing endpoint * chore: Code Rabbit --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
Rajat Ahuja
|
3d0f96d339
|
Add QueueItem cleanup logic (#228)
* feat: start queue cleanup * feat: add Queue cleanup logic * feat: mark messages as errored * fix: rename + commit * fix: rm apscheduler * fix: add default for created_at in migration * fix: consolidate error handling and only throw away first message of an errored batch * fix: move processing to within queue manager * fix: regen migration * fix: CR comment * fix: rename message -> queue_item when dealing with queue_item in the queue manager |
|
|
doria
|
f988aae996
|
create Representation class and use it to unify all formatting (#214)
* feat: add optional JWT and webhook secrets to honcho instance creation * chore: ignore spurious warnings * feat: add response format if using gpt-5 model family * feat: add response models to all apis except anthropic * fix: raise NotImplementedError for response models in AsyncAnthropic client * chore: address review * [WIP] representation structure + deriver cleanup * chore: add tests, cleanup * feat: [WIP: semi-working] representation object * fix: alignment * fix: make observations hashable for dedup * fix: datetime formatting, observation counting * fix: switch to int for message id, clean up representation * feat: remove need for metadata working rep * chore: cleanup * fix: use tenacity instead of custom fns * feat: add representation and card to context if desired * feat: add semantically relevant observations * fix: pass all params to streaming, nonblocking streaming * feat: consolidate document saving, make working representation fetching much smarter * chore: add 100% test coverage of representation util * feat: basic dream infra * feat: dream queue item first pass * chore: fixes & cleanup from coderabbit * fix: dreams scheduled when new document count reaches a certain threshold * feat: wip: timed dreams (not working) * fix: test * fix: remove useless pyright ignore * fix: executing dreams * feat: dreaming * feat: [WIP] longmemeval bench * feat: add USE_PEER_CARD setting, fix longmem test driver * feat: get full working rep for dialectic in one swoop -- fix representation_from_documents to use the proper timestamp! * fix: timestamps for real, handle assistant qs in longmem * fix: remove old client, add batching to longmem * perf: remove duplicate detection, will move to background task * feat: track perf metrics on evals * feat: adjust deriver prompt to use peer_id, add question date to question, clean up deriver * fix: label metrics by task for better perf trace * chore: code review * feat: add efficiency score to longmem bench * chore: tuning and cleaning up eval * chore: bring in the big prompts * feat: add support for vllm client * feat: perf: bundle db calls in deriver and dialectic, increase max conns in docker db * feat: add merge-sessions flag to longmemeval, add SUMMARY_ENABLED flag * fix: COLLECT_METRICS default false * chore: display start/end message ids, don't include in metrics * fix: break large messages apart for eval * fix: only get/create collection when needed * feat: properly attribute documents with message id ranges and add session name column to documents * fix: revert move of get_or_create_collection (need for fkey) * fix: always get collection with peer name even if it's none * chore: coderabbit * fix: give peer card its own config, expand document schema, refactor get_context to be parallel, various cleanup chores and bugfixes * chore: refactor: reify observer/observed system across entire codebase, including db migration * refactor: cleanup code organization, make singletons where desired * refactor: replace embeddings store with representation manager * chore: coderabbit cleanup * chore: update migration to non-null session param in documents, general review and cleanup * chore: merge branch 'main' into ben/deriver-tidy * chore: review fixes |
|
|
Vineeth Voruganti
|
73c5eb1f5a
|
SDK Changes (#223)
* fix: Add top level transactions for tracing in deriver * fix: Add Delete methods to sdks for sessions * feat: Add session deletion and session level deriver status methods * fix: Streaming support to python sdk * fix: Checkpoint Streaming working in SDKs with workarounds for stainless * feat: fix pagination, add message metadata update, and get peer card methods * feat: parity in typescript and python sdks * fix: Address Coderabbit * fix: Address Coderabbit * chore: add basedpyright rules to sdk * fix: Code Rabbit & Changelog * fix: updated docs with api-reference |
|
|
Rajat Ahuja
|
3df044e1b7
|
feat: use token limit from all senders for rollup (#221)
* feat: use token limit from all senders for rollup * fix: send all messages to queue_manager * test: fixe tests and move ownership check to separate query * fix: if first sender message exceeds batch then we should return an empty batch * feat: separate representation and non-representation processing * fix: return msg object and queue items * fix: rollup logic with interleaving messages * fix: rm TODO * Fix langfuse trace and add fallback for weird error in deriver * fix: simplify query * fix: comment out update_current_generation * fix: BPR --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
Rajat Ahuja
|
6e01af2005
|
Track worker ownership (#216)
* fix: simplify; use debug logs * fix: random type error * fix: CR comments * fix: move limit check before opening db connection * fix: use single SQL call to verify ownership and fetch message batch * fix: CR comments 2 |
|
|
Rajat Ahuja
|
1ae2c4599a
|
fix: separate clean and claim work units. skip locked AQS rows on delete (#215) | |
|
Rajat Ahuja
|
57977b541e | fix: Integer -> BigInteger | |
|
Rajat Ahuja
|
70b6965939 | fix: CR comments | |
|
Rajat Ahuja
|
1cc3d9aaaf | perf: use join on messages table instead of storing token_count in queue payload | |
|
Rajat Ahuja
|
5301914847 | fix: CR comments | |
|
Rajat Ahuja
|
dfa7577eb8 | Merge branch 'main' into rajat/DEV-1069 | |
|
doria
|
3f47866ae0
|
feat: add configurable max message size, update tests and docs -- fixes token overflow in deriver (#208)
* feat: add configurable max message size, update tests and docs -- this fixes token overflow in deriver * fix: remove LLMError special handling * chore: bump version --------- Co-authored-by: Rajat Ahuja <rahuja445@gmail.com> Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
Rajat Ahuja
|
4328fb3d2a | fix: actually pass in token count from msg to queue item | |
|
Rajat Ahuja
|
e6c580b660 | refactor: queue manager | |
|
Rajat Ahuja
|
8533256ca0
|
fix: long-held connections in queue manager (#189)
* fix: long-held connections in queue manager * fix: CR comments. tests * fix (deriver): Limit # of work units claimed based on available workers * fix: detached instance * fix: batch claim work units --------- Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com> |
|
|
doria
|
7a9746ca8b
|
Peer Cards (#180)
* chore: fill out missing metadata inputs in python sdk * feat: add get_peer_config to python sdk, thoroughly document ts sdk and remove bad client usage * feat: zod chore: update tests chore: bump version, changelog * chore: python sdk version bump and changelog * [WIP] feat: combine search methods and rework endpoint to include limit param * chore: test new stainless config with library * nits: coderabbit * Merge branch 'ben/sdk-improvements' into ben/search-rrf * chore: pre-commit hooks cleanup * feat: thoroughly document observation config * Update sdks/python/src/honcho/peer.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * chore: v1.3.0 * feat: update version to 2.2.0 and enhance search functionality with arbitrary filters - Remove unused config variables - Added arbitrary filters to all search endpoints. - Pluralize `filters` everywhere in SDKs for consistency - Updated documentation and changelog to reflect these changes. * expose core client in TS and Python SDKs (#150) * expose core client from sdks * align text * fix: resolve get_effective_observe me race condition, default peer config (#176) * fix: resolve get_effective_observe me race condition, default peer config * fix: preserve custom config even after leaving * chore: test cases, enqueue types * Update sdks/typescript/package.json Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: doria <93405247+dr-frmr@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * chore: formatting * chore: revert undesired changes to v1 spec, clean up docs, coderabbit * feat: better search docs, fix worker.ts * fix: correctly make ts params optional in cases, update docs * chore: coderabbit * chore: remove spurious package-lock * feat: [WIP] introduce peer cards * chore: remove search.mdx * chore: clean up deriver * feat: peer cards working in deriver * feat: add basic peer_card_bench * feat: refine peer card prompt, add mini-benchmark, switch to gpt-5-nano * refactor: update peer card handling in dialectic functions and improve error handling - Enhanced `get_peer_card` function to handle `ResourceNotFoundException`. - Updated `dialectic_call` and `dialectic_stream` to accept `peer_card` and `target_peer_card` parameters. - Modified prompt generation to include peer card information. - Cleaned up whitespace in several files for consistency. * chore: update mirascope dependency version in configuration files - Bumped mirascope version from 1.25.1 to 1.25.5 in pyproject.toml and uv.lock. - Added a note in config.py regarding peer card output token handling. - Removed unnecessary comments in clients.py for clarity. * fix: [coderabbit] improve error handling in set_peer_card and enhance logging - Added a check in `set_peer_card` to raise `ResourceNotFoundException` if the peer does not exist. - Updated logging in `CertaintyReasoner` to capture exceptions with Sentry when enabled. - Refined logging messages for clarity and consistency across various functions. - Cleaned up whitespace and formatting in several files for improved readability. * refactor: update working representation handling and improve metadata key usage - Introduced constants for representation collection names to enhance clarity and maintainability. - Updated function signatures in `get_working_representation` and `set_working_representation` to require `session_name`. - Simplified metadata key determination logic by using constants instead of hardcoded strings. - Removed legacy fallback logic for working representation data retrieval. - Refactored `save_working_representation_to_peer` to utilize the new `set_working_representation` function for improved code reuse. * chore: update configuration files and enhance working representation settings - Added new peer card settings and context token limits to `.env.template`, `config.toml.example`, and documentation. - Introduced `WORKING_REPRESENTATION_MAX_OBSERVATIONS` to `DeriverSettings` for better control over observation storage. - Updated `set_working_representation` to merge new observations while respecting the maximum limit. - Improved docstrings for clarity and consistency across functions. * feat: introduce LLMError exception and enhance error handling in deriver - Added LLMError exception to handle failures in LLM calls, normalizing inputs into a JSON-serializable format. - Updated CertaintyReasoner to raise LLMError on exceptions during LLM function calls. - Enhanced QueueManager to log LLMError occurrences and re-queue messages appropriately. - Modified test runner to support asynchronous operations and improved output formatting for test results. - Updated test cases to include session information for better context. * feat: add __repr__ method to QueueItem for improved string representation - Implemented a __repr__ method in the QueueItem class to provide a clear and informative string representation of its attributes. - Updated timeout handling in TestRunner to default to 10000.0 seconds when timeout_seconds is not set, enhancing robustness in polling operations. * refactor: update peer card data structure and improve handling in related functions - Changed return type of `get_peer_card` and `set_peer_card` to use `list[str]` instead of `str | None`. - Updated `peer_card_call` and related functions to accommodate the new list structure for peer cards. - Introduced `PeerCardQuery` model to standardize responses from peer card queries. - Adjusted prompt generation in `peer_card_prompt` to reflect the new data structure. - Modified benchmark tests to align with the updated peer card handling. * refactor: adjust peer card output token settings and update related functions - Increased `PEER_CARD_MAX_OUTPUT_TOKENS` from 2000 to 4000 in `DeriverSettings`. - Updated `critical_analysis_call` to use `json_mode` and removed unused parameters. - Modified benchmark tests to utilize the new `PEER_CARD_MAX_OUTPUT_TOKENS` setting. - Removed obsolete `add_dislike.json` test file. * refactor: update peer card handling in critical analysis and dialectic prompts - Changed `peer_card` parameter type from `str | None` to `list[str] | None` in `critical_analysis_call` and related functions. - Simplified error handling in `process_representation_task` by removing redundant try-except block. - Updated prompt generation in `critical_analysis_prompt` and `dialectic_prompt` to format `peer_card` as a string with newlines. - Adjusted benchmark tests to reflect changes in peer card structure and output formatting. * refactor: update peer card test cases to use list structure - Modified test cases in `test_representation_crud.py` to reflect the change in `peer_card` parameter type from `str` to `list[str]`. - Updated assertions to accommodate the new list format for setting and retrieving peer cards. - Ensured that tests for missing peers correctly handle the list input format. * fix: improve formatting of peer card output in prompts - Updated `peer_card_prompt` to join `old_peer_card` list elements with newlines for better readability. - Removed outdated comment in `dialectic_prompt` regarding handling of non-existent cards. * chore: [coderabbit] enhance docstring and logging in prompts and queue manager - Updated the docstring in `critical_analysis_prompt` to provide detailed type annotations for parameters. - Improved logging in `chat` to differentiate between single and multiple retrieved peer cards. - Adjusted logging format in `QueueManager` to use a more structured approach for shutdown messages. --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Rajat Ahuja <rahuja445@gmail.com> |
|
|
Rajat Ahuja
|
3bea3da169
|
feat: webhooks (#168)
* feat: webhooks * feat: Enhance webhook security and typing, fix validation and encryption bugs * fix: lint / types * fix: rm files * fix: rm mcp * fix: pydantic issue with TypedDict in python version <= 3.11 * fix: pre-commit hook for test coverage * fix: simplify API -- store url on workspace * fix: redo architecture * fix: webhook body * fix: make workspace optional * fix: comments * refactor: add webhook secret * fix: CR comments * feat: use deriver for webhooks * use key-value approach * feat: add work unit key to deriver * fix: add work unit key to webhooks * fix: tests * fix: cr comments #2 * fix: endpoint structure; make webhook delivery into a function; add tests; other general comments * chore: change webhook secret, fix test event and workspace_id, use async with * feat: implement queue.empty and backfill * fix: unique constraint * refactor: queue to use outerjoin and remove skip locked; also fix publish queue.empty * fix: tests * fix: migration - make columns non-nullable |
|
|
Vineeth Voruganti
|
0664410878
|
fix (queue): Add explicit rollback on integrity error (#174) | |
|
doria
|
bc5474c170
|
feat: 2.1.1 release (#167) | |
|
doria
|
a14899521c
|
Honcho 2.1.0 "ROTE" deriver (#160)
* feat: update SDKs to core 1.2.0 * feat: 2.1.0 introduce ROTE deriver/dialectic chore: refactor repo * fix: get typescript sdk tests working again, bump version numbers * chore: cleanup * chore: update unit test provider config * fix: remove "backup" query gen * fix: remove old utils from conftest |