* feat(db): add connection retry, adaptive deriver polling, and pool metrics
Add resilience and visibility for DB connection handling under transaction-
pooler (Supavisor) saturation, where client-connection limits get exhausted
across many tenants.
- get_db/tracked_db now force an eager pool checkout with bounded exponential
backoff (tenacity), retrying SQLAlchemy TimeoutError + OperationalError so
transient pooler rejections degrade gracefully instead of 500ing. Toggle via
DB_CONNECTION_RETRY_ENABLED (+ delay/backoff knobs); ~10s default budget.
- Deriver polling backs off when idle or erroring (base -> max, x2 each cycle)
and snaps back to base on claimed work, cutting steady-state query load.
Toggle via DERIVER_POLLING_BACKOFF_ENABLED (+ max/multiplier).
- Add scrape-time db_pool_connections Prometheus gauge (checked_out/checked_in/
size/overflow, labeled api|deriver), registered in both the API lifespan and
the deriver metrics server.
- Make SqlalchemyIntegration explicit in both Sentry inits; wrap connection
acquisition in a db.pool.acquire span and capture live pool stats on
retry-exhaustion.
* feat(db): add acquisition counter and in-flight query gauge
Build on the pool-connection metrics with two signals that turn detection
into diagnosis under transaction-pooler saturation:
- db_connection_acquisitions{outcome=ok|retried|exhausted}: counts how often
connection checkout retries through pooler rejection — the alertable early
warning before requests start failing.
- db_queries_in_flight: statements actually executing on the wire (via
SQLAlchemy cursor-execute events, drift-proof across query errors). Pairs
with checked_out: the gap reveals connections held but parked (the "idle in
transaction during an external call" antipattern). Labeled namespace +
instance_type only; gated on METRICS.ENABLED for zero overhead when off.
Add DB-free unit tests for retry outcomes, polling backoff, and in-flight
gauge drift handling.
* fix: address CodeRabbit review on PR #758
- db: roll back the session on a retryable checkout failure before
retrying — a failed autobegin can leave it pending-rollback, making the
next db.connection() raise instead of re-checking-out cleanly. Cheap
Python-side cleanup when no connection was bound.
- metrics: guard DBPoolCollector.collect() so a pool-read/import hiccup
can't raise and abort the whole /metrics scrape (Prometheus drops ALL
metrics if any collector raises) — log and fall back to empty.
* fix(db): lazy retrying session + review fixes for connection backoff
Address Codex/CodeRabbit review on PR #758.
- Replace eager checkout with HonchoAsyncSession: a lazy AsyncSession that
checks out its connection (with retry) on the first DB-touching call, not at
construction. Request handlers doing non-DB work (embedding/file/LLM) before
their first query no longer pin a connection across it, while the API path
still gets checkout retry. Only the checkout is retried — the statement runs
once via super(), so writes are never duplicated. Tracing's set_config moves
into the same lazy acquire hook.
- Roll the session back on a retryable checkout failure before retrying, so a
failed autobegin can't leave it pending-rollback.
- Lower default POOL_TIMEOUT to 5s and validate it stays under the retry budget
for pooled (non-null) POOL_CLASS; update config.toml.example and v2/v3 docs.
- Clamp pool overflow gauge to >= 0 (was negative before the pool fills).
- Remove double-sleep in the deriver idle poll (true backoff cap, not 2x);
make in-flight instrumentation registration idempotent.
- Tests: HonchoAsyncSession lazy/idempotent acquire, statement-runs-once,
tracing, commit/rollback flag reset, get_db no-acquire-at-entry, polling-loop
single-sleep, and the POOL_TIMEOUT/retry-budget validator.
* fix(db): cover all DB-touching session methods; clear flag on close/reset
Address Codex follow-up review on PR #758 (polish, no behavior-critical bug).
- HonchoAsyncSession: wrap get/get_one/stream/stream_scalars/delete in addition
to execute/scalar/scalars/flush/merge/refresh/commit, so the "lazy checkout
with retry on first DB use" guarantee has no holes. connection() stays
unwrapped (acquire_connection_with_retry calls it — wrapping would recurse).
- Reset the acquired flag on close()/reset() too, so a session reused after
close/reset re-acquires (and re-wraps retry) on its next DB use.
- Fix stale comments: connection retry now applies lazily to the request path
via HonchoAsyncSession (config.py), and the FakeSession helper note.
- Tests: close/reset flag reset, and get/delete route through acquisition.
* fix: Inconsistencies in Docs, health endpoint, troubleshooting guide
* fix: (docs) maintain consistency on postgres db name
* chore: (docs) update v2 contributing docs with updates db paths
* docs: overhaul self-hosting docs for provider-agnostic setup
- .env.template: lead with provider options (custom, vllm, google,
anthropic, openai, groq) instead of baking in vendor-specific keys.
All provider/model settings commented out so server fails fast until
configured. Separate endpoint config from per-feature provider+model
from tuning knobs.
- docker-compose.yml.example: fix healthcheck -d honcho -> -d postgres
to match POSTGRES_DB=postgres.
- config.toml.example: reorder and document LLM key section with
OpenRouter and vLLM examples.
- self-hosting.mdx: replace multi-vendor key table with provider options
table. Add examples for OpenRouter, vLLM/Ollama, and direct vendor
keys. Remove duplicated key lists from Docker/manual setup sections.
- configuration.mdx: replace scattered provider docs with provider types
table. Fix Docker Compose snippet to match actual compose file. Note
code defaults as fallback, not recommended path.
- troubleshooting.mdx: add alternative provider issues section (custom
provider config, model name format, Docker localhost, structured
output failures).
* docs: add Docker build troubleshooting for permission errors
- Document BuildKit requirement (RUN --mount syntax)
- AppArmor/SELinux blocking Docker builds on Linux
- Volume mount UID mismatch between host and container app user
- Note in self-hosting docs that Docker path builds from source
* docs: reframe self-hosting as contributor/dev path, point to cloud service
* Revert "docs: reframe self-hosting as contributor/dev path, point to cloud service"
This reverts commit 3e766eb1a9.
* docs: add production compose, model guidance, thinking budget docs
- Add docker-compose.prod.yml for VM/server deployment: no source
mounts, restart policies, 127.0.0.1-bound ports, cache enabled
- Add model tier guidance and community quick-start link to self-hosting
- Document THINKING_BUDGET_TOKENS gotcha for non-Anthropic providers
- Add reverse proxy examples (Caddy + nginx) to production section
- Add backup/restore commands to production considerations
* docs: simplify self-hosting to single provider, restructure config guide
Self-hosting page now defaults to one OpenAI-compatible endpoint
with one model for all features. Moved model tiers, alternative
providers, and per-feature tuning into the configuration guide.
Eliminated duplicate config priority sections, dev/prod split,
and redundant TOML examples.
* docs: merge compose files, restore provider/model to feature sections in .env.template
Single docker-compose.yml.example with dev sections commented out.
Moved PROVIDER and MODEL back alongside each feature in .env.template
so settings stay colocated with their module. Updated self-hosting
docs to reference single compose file.
* fix: broken anchor links, redundant migration step, minor inconsistencies
Fix 4 broken internal links (#llm-provider-setup, #llm-api-keys,
#which-api-keys-do-i-need, #alternative-providers) to point to
correct headings. Remove redundant Docker migration step (entrypoint
already runs alembic). Fix cache URL missing ?suppress=true in
reference config. Fix uv install command to use official method.
* docs: env template ready to use, simplify self-hosting flow
.env.template now has provider/model lines uncommented with
placeholder values — user just sets endpoint, key, and model name.
Thinking budgets default to 0 for non-Anthropic providers.
Self-hosting page: removed 30-line env var wall, LLM setup now
points to the template. Merged duplicate verify sections.
Removed api_key from SDK examples (auth off by default).
* docs: reorder next steps, configuration guide first
* fix: default embedding provider to openrouter for single-endpoint setup
Without this, embeddings default to openai which requires a separate
LLM_OPENAI_API_KEY. Setting to openrouter routes embeddings through
the same OpenAI-compatible endpoint as everything else.
* fix: review issues — hermes page, thinking budget, production wording
Hermes integration page: replaced inline Docker/manual setup with
link to self-hosting guide, added elkimek community link. Removed
old env var names (OPENAI_API_KEY without LLM_ prefix).
Troubleshooting: removed "or 1" from thinking budget guidance.
Self-hosting: softened "production-ready" to "production-oriented"
since auth is disabled by default.
* docs: model examples in template, expanded LLM setup, better verify flow
.env.template: added "e.g. google/gemini-2.5-flash" hints next to
model placeholders so users know the expected format.
Self-hosting: expanded LLM Setup to show the 3 things users need to
set (endpoint, key, model name) with find-replace tip. Added build
time note, deriver log check, and real smoke test (create workspace)
to verify section. Health check now notes it doesn't verify DB/LLM.
* fix: smoke test uses v3 API path, not v1
* docs: clarify deriver metrics port vs Prometheus host port
* fix: remove deprecated memoryMode from hermes config example
* docs: update hermes page to match current memory provider config
Updated config to match hermes-agent docs: removed apiKey (not needed
for self-hosted), added hermes memory setup CLI command, added config
fields table (recallMode, writeFrequency, sessionStrategy, etc.).
Better verification tests: store-and-recall across sessions, direct
tool calling test. Links to upstream hermes docs for full field list.
* fix: invalid THINKING_BUDGET_TOKENS=0 and missing docker/ in image
Comment out THINKING_BUDGET_TOKENS=0 in .env.template — deriver,
summary, and dream validators require gt=0. Dialectic levels also
commented out since non-thinking models don't need the override.
Add COPY for docker/ directory in Dockerfile so entrypoint.sh is
available when docker-compose.yml.example references it.
* chore: Additional troubleshooting step
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* feat: retry on more httpx exceptions
* fix: Add retry parity to typescript and update docs
* chore: (skills) update skills to match latest state of the sdk
* chore: (docs) update stale sdk code
* chore: (docs) clean up inconsistencies in docs
* chore: Rebuild Package
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* fix: sum input tokens and set session context cutoff to MAX_INPUT_TOKENS - sum
* fix: CR comments + remove context var
* fix: CR comments 2
* fix: use cache decorator; rm logger
* refactor: re-fetch formatted history after getting global working rep
* chore: fill out missing metadata inputs in python sdk
* feat: add get_peer_config to python sdk, thoroughly document ts sdk and remove bad client usage
* feat: zod
chore: update tests
chore: bump version, changelog
* chore: python sdk version bump and changelog
* [WIP] feat: combine search methods and rework endpoint to include limit param
* chore: test new stainless config with library
* nits: coderabbit
* Merge branch 'ben/sdk-improvements' into ben/search-rrf
* chore: pre-commit hooks cleanup
* feat: thoroughly document observation config
* Update sdks/python/src/honcho/peer.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* chore: v1.3.0
* feat: update version to 2.2.0 and enhance search functionality with arbitrary filters
- Remove unused config variables
- Added arbitrary filters to all search endpoints.
- Pluralize `filters` everywhere in SDKs for consistency
- Updated documentation and changelog to reflect these changes.
* expose core client in TS and Python SDKs (#150)
* expose core client from sdks
* align text
* fix: resolve get_effective_observe me race condition, default peer config (#176)
* fix: resolve get_effective_observe me race condition, default peer config
* fix: preserve custom config even after leaving
* chore: test cases, enqueue types
* Update sdks/typescript/package.json
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: doria <93405247+dr-frmr@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* chore: formatting
* chore: revert undesired changes to v1 spec, clean up docs, coderabbit
* feat: better search docs, fix worker.ts
* fix: correctly make ts params optional in cases, update docs
* chore: coderabbit
* chore: remove spurious package-lock
* feat: [WIP] introduce peer cards
* chore: remove search.mdx
* chore: clean up deriver
* feat: peer cards working in deriver
* feat: add basic peer_card_bench
* feat: refine peer card prompt, add mini-benchmark, switch to gpt-5-nano
* refactor: update peer card handling in dialectic functions and improve error handling
- Enhanced `get_peer_card` function to handle `ResourceNotFoundException`.
- Updated `dialectic_call` and `dialectic_stream` to accept `peer_card` and `target_peer_card` parameters.
- Modified prompt generation to include peer card information.
- Cleaned up whitespace in several files for consistency.
* chore: update mirascope dependency version in configuration files
- Bumped mirascope version from 1.25.1 to 1.25.5 in pyproject.toml and uv.lock.
- Added a note in config.py regarding peer card output token handling.
- Removed unnecessary comments in clients.py for clarity.
* fix: [coderabbit] improve error handling in set_peer_card and enhance logging
- Added a check in `set_peer_card` to raise `ResourceNotFoundException` if the peer does not exist.
- Updated logging in `CertaintyReasoner` to capture exceptions with Sentry when enabled.
- Refined logging messages for clarity and consistency across various functions.
- Cleaned up whitespace and formatting in several files for improved readability.
* refactor: update working representation handling and improve metadata key usage
- Introduced constants for representation collection names to enhance clarity and maintainability.
- Updated function signatures in `get_working_representation` and `set_working_representation` to require `session_name`.
- Simplified metadata key determination logic by using constants instead of hardcoded strings.
- Removed legacy fallback logic for working representation data retrieval.
- Refactored `save_working_representation_to_peer` to utilize the new `set_working_representation` function for improved code reuse.
* chore: update configuration files and enhance working representation settings
- Added new peer card settings and context token limits to `.env.template`, `config.toml.example`, and documentation.
- Introduced `WORKING_REPRESENTATION_MAX_OBSERVATIONS` to `DeriverSettings` for better control over observation storage.
- Updated `set_working_representation` to merge new observations while respecting the maximum limit.
- Improved docstrings for clarity and consistency across functions.
* feat: introduce LLMError exception and enhance error handling in deriver
- Added LLMError exception to handle failures in LLM calls, normalizing inputs into a JSON-serializable format.
- Updated CertaintyReasoner to raise LLMError on exceptions during LLM function calls.
- Enhanced QueueManager to log LLMError occurrences and re-queue messages appropriately.
- Modified test runner to support asynchronous operations and improved output formatting for test results.
- Updated test cases to include session information for better context.
* feat: add __repr__ method to QueueItem for improved string representation
- Implemented a __repr__ method in the QueueItem class to provide a clear and informative string representation of its attributes.
- Updated timeout handling in TestRunner to default to 10000.0 seconds when timeout_seconds is not set, enhancing robustness in polling operations.
* refactor: update peer card data structure and improve handling in related functions
- Changed return type of `get_peer_card` and `set_peer_card` to use `list[str]` instead of `str | None`.
- Updated `peer_card_call` and related functions to accommodate the new list structure for peer cards.
- Introduced `PeerCardQuery` model to standardize responses from peer card queries.
- Adjusted prompt generation in `peer_card_prompt` to reflect the new data structure.
- Modified benchmark tests to align with the updated peer card handling.
* refactor: adjust peer card output token settings and update related functions
- Increased `PEER_CARD_MAX_OUTPUT_TOKENS` from 2000 to 4000 in `DeriverSettings`.
- Updated `critical_analysis_call` to use `json_mode` and removed unused parameters.
- Modified benchmark tests to utilize the new `PEER_CARD_MAX_OUTPUT_TOKENS` setting.
- Removed obsolete `add_dislike.json` test file.
* refactor: update peer card handling in critical analysis and dialectic prompts
- Changed `peer_card` parameter type from `str | None` to `list[str] | None` in `critical_analysis_call` and related functions.
- Simplified error handling in `process_representation_task` by removing redundant try-except block.
- Updated prompt generation in `critical_analysis_prompt` and `dialectic_prompt` to format `peer_card` as a string with newlines.
- Adjusted benchmark tests to reflect changes in peer card structure and output formatting.
* refactor: update peer card test cases to use list structure
- Modified test cases in `test_representation_crud.py` to reflect the change in `peer_card` parameter type from `str` to `list[str]`.
- Updated assertions to accommodate the new list format for setting and retrieving peer cards.
- Ensured that tests for missing peers correctly handle the list input format.
* fix: improve formatting of peer card output in prompts
- Updated `peer_card_prompt` to join `old_peer_card` list elements with newlines for better readability.
- Removed outdated comment in `dialectic_prompt` regarding handling of non-existent cards.
* chore: [coderabbit] enhance docstring and logging in prompts and queue manager
- Updated the docstring in `critical_analysis_prompt` to provide detailed type annotations for parameters.
- Improved logging in `chat` to differentiate between single and multiple retrieved peer cards.
- Adjusted logging format in `QueueManager` to use a more structured approach for shutdown messages.
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Rajat Ahuja <rahuja445@gmail.com>
Vineeth Voruganti
Eri Barrett
Rajat Ahuja
Lalit Aditya Julapalli
doria