honcho/tests/unified/README.md

Unified Honcho Test System

This system allows for defining comprehensive, step-based tests for Honcho in a unified JSON format. It supports testing configuration hierarchy, multi-turn interactions, and complex assertions including LLM-as-a-judge.

Running Tests

# Run all tests in the test_cases directory
python -m tests.unified.run

# Run a specific test file
python -m tests.unified.run --test-dir tests/unified/test_cases

Test Schema

Tests are defined in JSON files. A test definition consists of a name, optional description, and a list of steps.

Structure

{
  "name": "my_test",
  "workspace_config": { ... },
  "steps": [
    { "step_type": "..." },
    ...
  ]
}

Actions

1. Configuration:

   • set_workspace_config: Update workspace settings.
   • set_session_config: Update session settings.

2. Interaction:

   • create_session: Create a new session, optionally with peers and config.
   • add_message: Add a single message.
   • add_messages: Add multiple messages.

3. Waiting:

   • wait: Wait for duration or "queue_empty".

4. Querying & Assertions:

   • query: Perform an action and assert on the result.
     • target: "chat", "get_context", "get_peer_card", "get_representation"

Assertions

• llm_judge: Use Claude to evaluate the result against a natural language prompt.
• contains / not_contains: Substring matching.
• exact_match: Strict equality.
• json_match: specific key-value checks.

Example

{
  "name": "demo_config_flow",
  "steps": [
    {
      "step_type": "create_session",
      "session_id": "s1",
      "peer_configs": {
        "user": { "observe_me": true },
        "agent": { "observe_others": true }
      }
    },
    {
      "step_type": "add_message",
      "session_id": "s1",
      "peer_id": "user",
      "content": "My name is Alice."
    },
    {
      "step_type": "wait",
      "target": "queue_empty"
    },
    {
      "step_type": "query",
      "target": "chat",
      "peer_id": "agent",
      "session_id": "s1",
      "input": "Who am I?",
      "assertions": [
        {
          "assertion_type": "contains",
          "text": "Alice"
        }
      ]
    }
  ]
}