Commit Graph

162 Commits

Author SHA1 Message Date
adavyas bd801f4964 Fix basedpyright regressions for custom instructions 2026-04-07 12:33:34 -04:00
adavyas c6ca57b9d8 Fix schema validation typing test coverage 2026-04-07 12:25:20 -04:00
adavyas de82b80227 Require explicit deriver custom instruction token cap 2026-04-07 12:11:26 -04:00
adavyas cd1c35c720 feat: cap deriver custom instruction tokens 2026-04-07 10:21:09 -04:00
adavyas 1b27bff625 Align custom instruction tests with split deriver prompts 2026-04-06 15:55:08 -04:00
adavyas f763f32b04 Fix custom instruction test typing 2026-04-06 15:55:08 -04:00
adavyas 6a746f1de4 Implement deriver custom instructions 2026-04-06 15:55:08 -04:00
adavyas 8a42a9b5b9 Stabilize prompt cache layout tests 2026-04-06 15:39:39 -04:00
adavyas db94ac050f Fix empty search query validation 2026-03-28 20:10:47 -07:00
adavyas 131905d14a Merge branch 'main' into codex/fix/prefix-based-cache-optim
# Conflicts:
#	src/schemas/api.py
2026-03-28 19:59:43 -07:00
adavyas aa326c8870 Harden schema sanitization and prompt cache tests 2026-03-27 15:08:59 -07:00
adavyas 2c4d88c316 Tune dialectic cache layout for Gemini 2026-03-27 07:45:56 -07:00
adavyas 3ea34064e1 Fix Gemini streaming and deriver cacheability 2026-03-25 00:00:34 -07:00
Rajat Ahuja 1cbcbc0263
fix: populate test harness DB config from docker compose (#435) 2026-03-19 17:39:57 -04:00
LRRuan 2097c2cbcf
fix(files): handle empty json uploads safely (#434)
* fix(files): handle empty json uploads safely

* fix(files): normalize invalid json upload errors

* fix(files): restore file processing error import

---------

Co-authored-by: LRRuan <lrruan@users.noreply.github.com>
2026-03-18 18:36:34 -04:00
Vineeth Voruganti 09a980c2fb
Sanitization and Memory Bug Fixes (#419)
* fix: use WeakValueDictionary for _observation_locks to prevent memory leak

* fix: harden input sanitization across API surface (DEV-1400)

- Parameterize SQL in set_config calls to prevent injection via request context
- Strip NUL bytes from string inputs (message content, queries, peer cards)
- Add JSONB metadata validation (100 key limit, 5 depth limit)
- Add filter recursion depth limit (max 5) to prevent stack overflow
- Update changelogs with unreleased entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: Refactor Schemas into separate files

* fix: Code Rabbit Comments

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 15:01:52 -04:00
adavyas 6dc080aeaf Use Gemini function names for tool responses 2026-03-14 17:01:20 -07:00
adavyas d2ce88737d Preserve Gemini tool context during provider fallback 2026-03-14 16:49:54 -07:00
adavyas 56924eb956 Fix remaining prefix cache review issues 2026-03-14 16:29:56 -07:00
adavyas 4ceec9f778 Address cache probe and Gemini review feedback 2026-03-14 14:44:08 -07:00
adavyas 482bb6cb16 Fix summarizer cache test typing 2026-03-14 13:12:52 -07:00
adavyas e5841882ad Track Gemini cache token metrics 2026-03-14 12:59:55 -07:00
adavyas 1b99469335 Optimize prompt prefix caching 2026-03-13 17:36:06 -07:00
Vineeth Voruganti 10ef7b96a8
Add Stricter limits to Summary & Peer Card (#400)
* fix: Add bounds to gemini client

* fix: Prevent empty summaries from being saved to DB (HONCHO-M7)

Raise LLMError on blocked Gemini responses (SAFETY, RECITATION, etc.)
so retry/backup-provider logic triggers. Treat empty LLM responses in
the summarizer as fallback instead of persisting empty strings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: Summary Eval via Locomo

* fix: Code Rabbit Comments

* fix: Code Rabbit Comments

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 15:08:09 -05:00
Vineeth Voruganti 3b1e964372
Modify Queue Status to only relevant tasks (#398)
* feat: Update Honcho system benchmarks

* fix: Align with runner common functions and fixed basedpyright issues

* fix: Address coderabbit issues

* chore: Address Ruff Errors

* fix: Queue Status to remove unhelpful info

* chore: (docs) Add docs for dreaming and buffering

* chore: Remove claude workflow

---------

Co-authored-by: 3un01a <3un01a@plasticlabs.ai>
2026-02-23 22:54:21 -05:00
3un01a 2631209256
feat: Update Honcho system benchmarks (#393)
* feat: Update Honcho system benchmarks

* fix: Align with runner common functions and fixed basedpyright issues

* fix: Address coderabbit issues

* chore: Address Ruff Errors

---------

Co-authored-by: 3un01a <3un01a@plasticlabs.ai>
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-02-23 21:30:30 -05:00
3un01a eba9279af2
Oolong Benchmark (#323)
* (feat) Add Oolong Benchmarks

* (fix) Address issues to fix basedpyright and coderabbit comments

* (fix) Address basedpyrwright additional warnings

* (fix) Address additional coderabbit issues

* (fix) Replace huggingface data loading to local filesystem-based

* (fix) Address coderabbit issues regarding data paths

* fix: Align with test harness conventions

* fix: Code Review Comments

* fix: stream data rather than load all at once

---------

Co-authored-by: 3un01a <3un01a@plasticlabs.ai>
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-02-23 16:55:59 -05:00
Vineeth Voruganti 78df86dc66
fix: Remove noisy sentry error on llm failure and upgrade deps (#396) 2026-02-23 15:53:29 -05:00
Vineeth Voruganti beb282bfbc
fix: Various Codex Audits (#386)
* fix: Various Codex Audits

* fix: Address Comments
2026-02-13 12:00:15 -05:00
Rajat Ahuja 97df0a80cd
feat: consolidate db calls in session context (#380)
* feat: consolidate db calls in session context

* fix: guard against embedding failures

* Parallelize Async Calls (#383)

* fix: parallelize db calls in context()

* fix: (test) use session factory to further isolate tests

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-02-13 11:54:41 -05:00
Rajat Ahuja 33ef0f8eab
perf: add redis caching and reduce embedding API calls (#372)
* feat: add redis to get collection + get session peer config

* fix: reduce embedding calls

* fix: delete session caches when soft-deleting session

* fix: batch embeddings in create observatiosn tool

* fix: batch embeddings call in extract_preferences

* fix: double embed in fallback for _handle_search_memory

* fix: claude comments

* fix: return ObservationResult

* fix: make cache delete/set retryable. remove session peer config cache

* chore: claude nitpicks

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-02-13 00:53:46 -05:00
doria 3aaced2cd1
feat: implement async workspace deletion with active session checks (#378)
* feat: implement async workspace deletion with active session checks

- Updated the DELETE /workspaces/:id endpoint to return 202 Accepted, indicating that the deletion request is processed in the background.
- Added a check for active sessions before allowing workspace deletion, raising a ConflictException if any exist.
- Updated related tests to ensure proper handling of active sessions during workspace deletion.

* fix: Address review issues

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-02-12 18:48:00 -05:00
doria 2522cc5ee6
sdks: add set peer card function (#371)
* feat: add set peer card to SDK, bump version, document

* fix: pytest -> pytest -x

* chore: deprecate .card(), move to .getCard() / .get_card()

* fix: get_or_create when crudding peer cards

* chore: review nits

* chore: document .get_card / .set_card

* chore: (docs) update language from deriver to dreamer agent

* fix: get_peer_card should not create peer/workspace

* fix: change api contract to return ResourceNotFoundException (#375)

* fix: PR nitpicks

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-02-09 15:35:57 -05:00
doria 3eab54374c
chore: parallelize tests for speed (#374)
* chore: parallelize tests for speed

* fix: truncate all tables after each test

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-02-06 21:10:34 -05:00
doria c9dee51fb9
refactor: enhance specialist exploration and observation processes (#357)
* refactor: enhance specialist exploration and observation processes

- Updated the orchestration logic to allow specialists to explore freely with optional hints from high-surprisal observations.
- Removed predefined probing questions, enabling a more dynamic approach to observation gathering.
- Adjusted the deduction and induction specialists to utilize the current peer card context and exploration hints in their prompts.
- Improved documentation within the code to clarify the roles and responsibilities of specialists in the observation process.

* feat: add thorough dream test

* refactor: standardize hint terminology and dedup peer card context
2026-02-03 11:02:07 -05:00
Rajat Ahuja d3637a9d73
fix: make FLUSH_ENABLED a config value (#361)
* fix: use cashews for DERIVER_FLUSH_KEY

* fix: make FLUSH_ENABLED a config value
2026-02-03 10:53:14 -05:00
doria 8ae21bb4fa
chore: delete old unused test system (#358) 2026-01-29 17:40:27 -05:00
doria a4a341e8b5
fix: add migration for 'deriver' to 'reasoning' in ResolvedConfiguration (#352) 2026-01-27 11:17:49 -05:00
Rajat Ahuja 110787cdca
feat: use messages from queue items for rep completed token count (#350) 2026-01-26 18:00:53 -05:00
Vineeth Voruganti 911aa6029f
v3.0.0 Release Candidate (#346)
* fix: (docs) update changelog and examples

* chore: (docs) Add note on Vector Stores

* fix: (mcp) WIP to sync with 3.0.0 conventions

* fix: (mcp) Sync MCP with final API changes for 3.0.0

* chore: (docs) sync changelog and openapi spec

* fix: fixing .mdx files to match openapi.json spec

* chore: Add script to estimate event creation

---------

Co-authored-by: ajspig <dragon@monstercode.com>
2026-01-26 15:06:02 -05:00
doria 5371463f39
Merge pull request #348 from plastic-labs/ben/trim-test-trial-tools
feat: give bench tools access to remote honchos
2026-01-26 14:18:39 -05:00
doria d79a9f21d2
feat: make session_name nullable for documents and update related SDKs (#347)
* feat: make session_name nullable for documents and update related SDKs

- Introduced a migration to make the `session_name` column in the documents table nullable, allowing for sessionless dreams.
- Updated Python and TypeScript SDKs to reflect the optional nature of `session_id` in conclusion creation and related methods.
- Enhanced tests to cover scenarios for creating conclusions without a session ID, ensuring proper handling of sessionless conclusions.
- Adjusted documentation and type definitions to clarify the optional session context in various components.

* chore: add migration test

* fix: ensure orphaned sessions exist during downgrade for nullable session_name migration
2026-01-26 13:33:11 -05:00
Rajat Ahuja 2270e5666f
switch OTEL metrics to prometheus (#344)
* feat: replace OTEL with Prometheus

* fix: second pass of docs and cleanup
2026-01-25 17:26:42 -05:00
Benjamin McCormick 6c3f671636 fix: clean Redis URL by removing query parameters in cache client and update flush mode logging in benchmark runner 2026-01-23 17:57:58 -05:00
Benjamin McCormick 258fd3736b fix: turn off summaries on test runs directly 2026-01-23 16:13:00 -05:00
Benjamin McCormick ffff6afb1a chore: refactor test runners to be cleaner, deduplicate code, add params for testing remote servers 2026-01-23 15:47:51 -05:00
doria e9d8b8759d
refactor: update semantic search parameter from `last_user_message` to `search_query` across documentation and SDKs (#341)
* refactor: update semantic search parameter from `last_user_message` to `search_query` across documentation and SDKs

- Changed references in documentation and code to use `search_query` instead of `last_user_message` for fetching semantically relevant observations and conclusions.
- Updated related function signatures and descriptions in Python and TypeScript SDKs to reflect this change.
- Adjusted tests to ensure compatibility with the new parameter naming.

* chore: openapi v3 formatted how we like it

* fix: reorder docs, update examples in README, update skills

* fix: message type option in sdk reference

* chore: update remaining getcontext and representation language

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-01-23 13:11:20 -05:00
Rajat Ahuja afa9f7b589
fix: add _total suffix to metrics (OpenMetrics convention) (#340) 2026-01-23 11:49:45 -05:00
doria dce96889bc
feat: honcho 3.0, sdks 2.0, excise stainless, update v3 docs, changelogs (#331)
* chore: 3.0 honcho and 2.0 sdks changelog

fix: use PeerContextResponse in peer.ts

* chore: move docs to /v3/, build SDKs

* chore: code review

* feat: [WIP] migrate away from stainless in typescript sdk

* chore: move api from /v2/ to /v3/

* feat: no-stainless typescript with real tests

* feat: migrate python sdk off of stainless

* feat: clean typescript sdk

* chore: add tests for ts http client

* fix: rewrite entire python sdk in new format, update typescript sdk to use `configuration` not `config` for consistency with API

* fix: clean up SDKs, synchronize

* chore: update sdk examples

* chore: update OpenAPI documentation and SDK examples to reflect changes

* fix: better test

* fix: install deps in test runner, improve robustness of streaming in sdk, coderabbit nits

* fix: standardize around camelCase in TS SDK

* refactor: update configuration handling in SDKs to use typed models for workspace, session, and peer configurations

* docs: clarify queue status usage and remove polling methods from SDKs

add claude skills for migrations

* chore: fix links in docs

* feat: add deriver flush mode to bypass batch token threshold

- Introduced `is_deriver_flush_enabled` function to check if flush mode is active.
- Updated `QueueManager` to conditionally apply batch token thresholds based on flush mode.
- Enhanced `UnifiedTestExecutor` to enable flush mode via Redis.
- Added `flush` parameter to test cases to facilitate testing of flush mode behavior.
- Updated various test cases to utilize the new flush functionality.

* feat: implement schedule_dream functionality in SDKs, use in unified test runner

- Added `schedule_dream` method to both Python and TypeScript SDKs for scheduling dream tasks.
- Updated HTTP routes to include endpoint for scheduling dreams.
- Enhanced test runner to utilize the new `schedule_dream` method for scheduling actions.
- Updated TypeScript client to support the new scheduling functionality with appropriate parameters.

* feat: update single deriver task to support multiple observers

- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.

* refactor: update enqueue tests to support deduplication of queue items with multiple observers

- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.

* fix: add backwards compatibility for representation work unit keys and payload observers

* feat: update dialectic configuration and introduce cost calculator

- Adjusted LLM and dialectic settings in `.env.template`, `config.toml.example`, and `src/config.py` to reduce maximum tool output characters and session history tokens for cost efficiency.
- Implemented a new `dialectic_cost_calculator.py` script to estimate costs based on reasoning levels and model pricing.
- Enhanced `DialecticAgent` to utilize minimal tools and adjusted output token settings based on reasoning level to optimize performance and reduce costs.

* feat: add reasoning level to chat input in unified test runner

- Enhanced the `UnifiedTestExecutor` to include a `reasoning_level` parameter in the chat method call.
- Updated the `QueryAction` model to support the new `reasoning_level` attribute, allowing for more nuanced chat interactions.

* feat: run deriver once for multiple observers (#335)

* feat: update single deriver task to support multiple observers

- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.

* refactor: update enqueue tests to support deduplication of queue items with multiple observers

- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.

* fix: add backwards compatibility for representation work unit keys and payload observers

* feat: refactor benchmark runners to share common functionality

- Introduced a new `runner_common.py` module containing shared utilities for benchmark test runners, including common argument parsing, client creation, and queue management.
- Updated `BEAMRunner`, `LoCoMoRunner`, and `LongMemEvalRunner` to inherit from `RunnerMixin`, leveraging shared functionality for metrics collection and logging.
- Added `reasoning_level` and `redis_url` parameters to runner constructors for enhanced configuration.
- Streamlined argument parsing by utilizing `add_common_arguments` for shared command-line options across all runners.

* fix: update last_user_message handling to use message content instead of ID

* fix: standardize config vs configuration

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
2026-01-22 15:16:28 -05:00
Vineeth Voruganti 6b3ecef601
Telemetry Overhaul (#333)
* chore: Refactor a top-level telemetry folder

* fix: Add OTEL push based metrics

* chore: cleanup otel to match prometheus implementation

* fix: Remove prometheus

* feat: Scaffold CloudEvent Emitter

* chore: remove dead code

* feat: PoC Cloud Events

* chore: cleanup test

* fix: Event naming conventions and OTEL Settings

* fix: Revamped Dream Event Structure

* fix: Instrument Event Code

* fix: Add Tests for CloudEvent Telemetry

* fix: Code Rabbit Nits

* fix: Dedupe Event IDs
2026-01-21 16:49:14 -05:00