3.0 KiB

Raw Blame History

Unified Honcho Test System

This system allows for defining comprehensive, step-based tests for Honcho in a unified JSON format. It supports testing configuration hierarchy, multi-turn interactions, and complex assertions including LLM-as-a-judge.

Running Tests

# Run all tests in the test_cases directory
python -m tests.unified.run

# Run a specific test file
python -m tests.unified.run --test-dir tests/unified/test_cases

Test Schema

Tests are defined in JSON files. A test definition consists of a name, optional description, and a list of steps.

Structure

{
  "name": "my_test",
  "workspace_config": { ... },
  "steps": [
    { "step_type": "..." },
    ...
  ]
}

Actions

Configuration:
- set_workspace_config: Update workspace settings.
- set_session_config: Update session settings.
Interaction:
- create_session: Create a new session, optionally with peers and config.
- add_message: Add a single message.
- add_messages: Add multiple messages.
- create_scope: Create a scope and optionally add member sessions. Add the sessions before the messages you want in scope — membership only affects messages ingested after a session joins.
Waiting:
- wait: Wait for duration or "queue_empty".
Querying & Assertions:
- query: Perform an action and assert on the result.
  - target: "chat", "get_context", "get_peer_card", "get_representation"
  - scope: confine the read to a scope (or, for chat/representation, to the union of several). Valid for "chat", "get_representation" and "get_context"; the latter takes a single scope and requires observed_peer_id.

Raw HTTP vs the SDK

Most steps drive the Honcho Python SDK. create_scope and any query carrying scope go over raw HTTP instead, because the published SDK trails the API and exposes neither. Calling the API directly also tests the contract the SDK is generated from, so a wrong status code or response shape surfaces here rather than being masked by client-side validation.

Assertions

llm_judge: Use Claude to evaluate the result against a natural language prompt.
contains / not_contains: Substring matching.
exact_match: Strict equality.
json_match: specific key-value checks.

Example

{
  "name": "demo_config_flow",
  "steps": [
    {
      "step_type": "create_session",
      "session_id": "s1",
      "peer_configs": {
        "user": { "observe_me": true },
        "agent": { "observe_others": true }
      }
    },
    {
      "step_type": "add_message",
      "session_id": "s1",
      "peer_id": "user",
      "content": "My name is Alice."
    },
    {
      "step_type": "wait",
      "target": "queue_empty"
    },
    {
      "step_type": "query",
      "target": "chat",
      "peer_id": "agent",
      "session_id": "s1",
      "input": "Who am I?",
      "assertions": [
        {
          "assertion_type": "contains",
          "text": "Alice"
        }
      ]
    }
  ]
}

3.0 KiB Raw Blame History