From 846ecd19f677780caa5f879fc8747576fc4631bf Mon Sep 17 00:00:00 2001 From: Eugene Eisenstein Date: Mon, 31 Aug 2026 19:53:48 -0400 Subject: [PATCH] feat(sandbox): committed ephemeral Honcho sandbox with one-command reset A local Honcho that comes up already seeded and resets to that exact state in under a second, so harness integration testing stops depending on machine state. `docker-compose.yml.example` gave the topology but was an example, unseeded, and had no teardown path, so every harness change was verified against whatever state a given laptop was in. The blocker to fixing that was the deriver's provider call, which made a sandbox neither free nor deterministic; src/mock_provider removes it. Reset restores a Postgres template database rather than re-deriving: `seed` snapshots the finished state, `reset` drops the live database and recreates it from that snapshot, then flushes Redis. Nothing is stopped or restarted -- DROP DATABASE ... WITH (FORCE) evicts the connection pools and both services reconnect on their own. Measured 0.86s in mock mode, 1.29s in real mode against a 361s seed. Two provider modes, both first class. Mock is the default: deterministic, free, no egress. Real points the same stack at a configured provider, reading credentials from one gitignored file rather than ambient environment. The difference matters and is documented: mock embeddings are hash-derived and carry no semantic similarity, so vector-recall assertions are impossible there, and mock conclusions come out explicit-only. On the committed fixture, mock yields 4 synthetic explicit conclusions and real yields 22 across all three levels. The sandbox is configured only by what Compose injects. PYTHON_DOTENV_DISABLED and HONCHO_CONFIG_TOML_DISABLED are both set because src/config.py calls load_dotenv(override=True) at import and the Dockerfile bakes any local config.toml into the image; without them a developer's own provider config wins silently. Deriver scheduling is pinned for the same reason: on stock settings a sandbox seeded with a handful of messages produces zero conclusions and gives no indication why, because work units wait for a 512-token batch or 30 minutes and startup jitter delays the first poll by up to 30s. Snapshots can go stale, so each carries a fingerprint -- Alembic revision, fixture hash, provider mode -- and reset refuses on a mismatch instead of restoring a state that predates a migration. Templates are per-mode, so both can coexist. Note: the pinned digest in sandbox/image.env predates the mock provider, so `--build` is required until that lands and a new image is published. sandbox.sh detects this and says so rather than crash-looping on a missing module. --- .gitignore | 4 + sandbox/README.md | 165 ++++++++++++++++++++ sandbox/compose.mock.yml | 50 ++++++ sandbox/compose.real.yml | 22 +++ sandbox/compose.yml | 133 ++++++++++++++++ sandbox/fixture.json | 41 +++++ sandbox/image.env | 16 ++ sandbox/init.sql | 9 ++ sandbox/real.env.example | 21 +++ sandbox/sandbox.sh | 326 +++++++++++++++++++++++++++++++++++++++ sandbox/seed.py | 184 ++++++++++++++++++++++ 11 files changed, 971 insertions(+) create mode 100644 sandbox/README.md create mode 100644 sandbox/compose.mock.yml create mode 100644 sandbox/compose.real.yml create mode 100644 sandbox/compose.yml create mode 100644 sandbox/fixture.json create mode 100644 sandbox/image.env create mode 100644 sandbox/init.sql create mode 100644 sandbox/real.env.example create mode 100755 sandbox/sandbox.sh create mode 100644 sandbox/seed.py diff --git a/.gitignore b/.gitignore index d7ad4cda..aac48e68 100644 --- a/.gitignore +++ b/.gitignore @@ -196,3 +196,7 @@ grafana-data/ # Claude Code addon stuff .omc + +# Sandbox real-provider credentials (see sandbox/real.env.example) +sandbox/real.env +sandbox/.state.env diff --git a/sandbox/README.md b/sandbox/README.md new file mode 100644 index 00000000..3e183db3 --- /dev/null +++ b/sandbox/README.md @@ -0,0 +1,165 @@ +# Honcho sandbox + +A local Honcho you can wipe and rebuild in seconds, so harness testing stops depending on +whatever state your laptop happens to be in. + +It comes up **already seeded** — a known workspace, peers, session, messages, and conclusions the +deriver has actually produced — and `reset` returns it to that exact state without re-deriving. + +```bash +sandbox/sandbox.sh up # start + seed (first run also builds/pulls) +sandbox/sandbox.sh reset # back to the seeded state, under a second +sandbox/sandbox.sh status # what's running, which templates exist +sandbox/sandbox.sh down # stop and delete the volumes +``` + +The API lands on . Postgres is on 15432 and Redis on 16379 — all +deliberately off the defaults, so the sandbox coexists with a normal local stack instead of +fighting it for ports and volumes. + +> **Right now you need `--build`.** The mock provider comes from PR #1094, which is not merged, so +> no published image contains it. `sandbox.sh` checks and tells you. Once #1094 lands and an image +> is published, bump the digest in `image.env` and plain `up` works. + +## Provider modes + +The deriver calls a model provider, which is what made a sandbox neither free nor deterministic. +Both ways of resolving that ship here; `mock` is the default. + +| | `mock` (default) | `real` | +| --------------------- | ---------------------------------------------------------------- | ---------------------------------------------------- | +| **Determinism** | Same conclusions every run | Non-deterministic | +| **Cost** | Zero, and no network egress | Real spend | +| **Seed time** | ~15s | ~6 min | +| **Reset time** | 0.86s | 1.29s | +| **Vector recall** | **Untestable** — see below | Testable | +| **What you get** | 4 conclusions, all `explicit`, synthetic text | 22 conclusions — 16 `explicit`, 4 `deductive`, 2 `inductive` | +| **Use it for** | CI, harness smoke tests, anything that needs a repeatable answer | Recall quality, validating against real model output | + +Those counts are from the committed fixture, measured on both paths. They are the sharpest +illustration of the difference: mock gives you four copies of `[mock] mock dummy placeholder …`, +while real mode reasons its way from "my cat Marzipan is named after the pylon's colour" to the +deductive conclusion *"alice's cat is named after a feature of alice's pedestrian bridge project."* +Anything asserting on that kind of inference needs real mode. + +```bash +sandbox/sandbox.sh up # mock +sandbox/sandbox.sh up --provider real # real +``` + +Each mode keeps its own seeded template, so both can exist at once and switching between them +costs nothing — about 10 seconds, and no re-derivation, because `up` restores that mode's existing +template instead of reseeding. + +### Two things that will otherwise cost you a day + +**Mock embeddings carry no semantic similarity.** They are hash-derived, so two paraphrases are +exactly as far apart as two unrelated strings. Any recall assertion built on mock mode must use +lexical or full-text search. A vector-ranking assertion will fail there for reasons that have +nothing to do with the code you are testing — that is what `real` mode is for. + +**Mock conclusions are synthetic.** The text is derived from the request, not from the meaning of +your messages, and the level is always `explicit` (the Dreamer's specialists write via tool calls, +which the mock deliberately never emits). Assert that conclusions *exist*; do not assert on what +they say or on the level mix, or your test will pass in one mode and fail in the other. + +### Real mode + +```bash +cp sandbox/real.env.example sandbox/real.env # then add a key +sandbox/sandbox.sh up --provider real +``` + +Credentials come from that one gitignored file, never from your ambient shell environment, so a +real-mode run has a single auditable input and the sandbox still doesn't inherit machine state. +`sandbox.sh` refuses to start if the file is missing or still carries the placeholder key — a +real-mode stack with no key boots perfectly well and then derives nothing, which is indistinguishable +from "the deriver found nothing". + +Real mode gets an unexpected benefit from the reset design: derivation is slow and costs money, and +the snapshot means you pay for it **once per seed** and reset for free after that. Measured: 361s to +seed, 1.29s to reset back to that exact state with no provider calls at all. + +## How reset is fast + +Seeding runs the deriver, and a naive reset would have to run it again. Instead, `seed` snapshots +the finished database as a Postgres template: + +``` +CREATE DATABASE honcho_sandbox_seeded_mock TEMPLATE honcho_sandbox +``` + +and `reset` drops the live database and re-creates it from that template, then flushes Redis. No +migrations, no re-derivation, no LLM calls — **measured at 0.86s**, and byte-identical every time. + +Nothing is stopped or restarted. `DROP DATABASE ... WITH (FORCE)` evicts the api and deriver +connection pools, and both reconnect by themselves — the api on its next request, the deriver on its +next poll a quarter-second later. Each logs one `OperationalError` as its in-flight connection dies; +that is expected, and it is the price of not paying ~9 seconds of container restart on every reset. + +The cost of a snapshot is that it can go stale. Two guards: + +- **Per-mode template names**, so a mock-seeded template can never be restored into a real-mode run. +- **A fingerprint recorded inside each template** — the Alembic revision it was seeded at, a hash of + `fixture.json` + `seed.py`, and the provider mode. `reset` compares and refuses with a "reseed" + message rather than silently restoring a state that predates a migration. + +If you hit that refusal: `sandbox/sandbox.sh seed`. + +## What's in the fixture + +`fixture.json` — two peers with **explicitly stated** observation topology (`alice` is observed and +does not observe; `assistant` observes and is not observed), one session, six messages carrying +distinctive lexically-searchable facts, and one scheduled cross-peer dream. + +Topology is written down rather than defaulted because it is the thing that silently breaks, and +`seed.py` re-reads it back from the server to confirm it took. + +Edit `fixture.json`, then `sandbox/sandbox.sh seed` to rebuild the template. `seed` always starts +from an empty, freshly migrated database, so it means the same thing whatever state you run it +from — running it twice gives the same result as running it once. + +Dreams never fire on their own here — the document threshold is 50 and the minimum gap between +dreams is 8 hours — so the seed schedules one directly. + +## Layout + +| File | Purpose | +| ------------------- | ---------------------------------------------------------------------- | +| `sandbox.sh` | The entry point. Owns Docker and Postgres. | +| `compose.yml` | Provider-agnostic base. **Not a working stack alone.** | +| `compose.mock.yml` | Provider overlay: adds the mock-provider service and wiring. | +| `compose.real.yml` | Provider overlay: reads `real.env`, no mock service. | +| `image.env` | The pinned image digest. One line, bump deliberately. | +| `init.sql` | Creates the `honcho_sandbox` database. | +| `fixture.json` | The seeded conversation. | +| `seed.py` | Populates and verifies. Talks only to the API. | +| `real.env.example` | Template for real-mode credentials. | + +Exactly one provider overlay is always composed on top of the base, so the choice is visible in the +`-f` list rather than buried in a default. That is also why `compose.yml` alone does not run: the +`depends_on` edge to `mock-provider` lives in the mock overlay, because Compose merges `depends_on` +additively and an override file cannot remove one. + +## Configuration + +The sandbox is configured **only** by what Compose injects. `PYTHON_DOTENV_DISABLED` and +`HONCHO_CONFIG_TOML_DISABLED` are both set, because `src/config.py` calls `load_dotenv(override=True)` +at import and the Dockerfile's `COPY config.toml* /app/` bakes a local `config.toml` into any +locally built image. Without those two flags your machine's leftovers would quietly win. + +`compose.yml` also pins the deriver's scheduling so derivation happens *now*. On stock settings a +sandbox seeded with a handful of messages produces zero conclusions and fails silently: work units +aren't claimed until a batch reaches 512 tokens or 30 minutes pass, and startup jitter delays the +first poll by up to 30 seconds. Those are turned off here — see the comments in `compose.yml`. + +## Building from the working tree + +```bash +sandbox/sandbox.sh up --build +``` + +Builds the repo instead of pulling the pinned digest, and points every service at the result so +they stay in sync. Use it when testing a change to Honcho itself. It trades reproducibility for +currency — you are running your tree, not the pinned bytes, which is right while iterating and +wrong when reproducing someone else's result. diff --git a/sandbox/compose.mock.yml b/sandbox/compose.mock.yml new file mode 100644 index 00000000..55fa8ea7 --- /dev/null +++ b/sandbox/compose.mock.yml @@ -0,0 +1,50 @@ +# Provider overlay: deterministic, free, no network egress. The sandbox default. +# +# Adds src/mock_provider (PR #1094) as its own service. It runs from the standard +# Honcho image under a different entrypoint, the way api and deriver already differ, +# so there is no second image to build or keep in digest-sync. + +services: + mock-provider: + image: ${HONCHO_SANDBOX_IMAGE:?set by sandbox.sh from sandbox/image.env} + entrypoint: + ["/app/.venv/bin/fastapi", "run", "--host", "0.0.0.0", "src/mock_provider/main.py"] + healthcheck: + # The app exposes only POST /chat/completions and POST /embeddings — no health + # or models route — so liveness is checked against the schema FastAPI always + # serves. A GET keeps the check side-effect free. + test: + [ + "CMD", + "/app/.venv/bin/python", + "-c", + "import urllib.request; urllib.request.urlopen('http://localhost:8000/openapi.json', timeout=2).read()", + ] + interval: 3s + timeout: 5s + retries: 20 + start_period: 5s + restart: "no" + + api: &provider-wiring + depends_on: + mock-provider: + condition: service_healthy + environment: + # All three are required (src/mock_provider/main.py). + # + # The key's value is never checked — the mock reads no Authorization header, + # and Honcho only tests it for truthiness before building the client + # (src/llm/registry.py). Set the base URL without it and the client is never + # constructed, so the base URL is silently ignored. Keep the value obviously + # fake: a module that ever escapes the override then 401s rather than spends. + LLM_OPENAI_API_KEY: sandbox-not-a-real-key + LLM_OPENAI_BASE_URL: http://mock-provider:8000/v1 + # Embeddings resolve through a separate client that reads the base URL only + # from this per-module override. Without it, embedding calls go to + # api.openai.com for real. Do not add a per-module credential override + # (..._OVERRIDES__API_KEY / API_KEY_ENV) — that makes the module ignore the + # global base URL above. + EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL: http://mock-provider:8000/v1 + + deriver: *provider-wiring diff --git a/sandbox/compose.real.yml b/sandbox/compose.real.yml new file mode 100644 index 00000000..d35a3ca3 --- /dev/null +++ b/sandbox/compose.real.yml @@ -0,0 +1,22 @@ +# Provider overlay: a real, configured model provider. +# +# Non-deterministic and it costs money. Use it when you need something the mock +# cannot give you — chiefly vector recall, since mock embeddings are hash-derived +# and carry no semantic similarity, and the full conclusion level mix. +# +# Credentials come from sandbox/real.env (gitignored), never from ambient host +# environment, so a real-mode run has one auditable input and the sandbox still does +# not depend on whatever happens to be exported in your shell. +# +# Copy sandbox/real.env.example to sandbox/real.env to get started. sandbox.sh +# refuses to start real mode if that file is missing or carries no key. +# +# No mock-provider service is defined here, so nothing depends on one. + +services: + api: &provider-wiring + env_file: + - path: real.env + required: true + + deriver: *provider-wiring diff --git a/sandbox/compose.yml b/sandbox/compose.yml new file mode 100644 index 00000000..c101f0ab --- /dev/null +++ b/sandbox/compose.yml @@ -0,0 +1,133 @@ +# Honcho sandbox — an ephemeral, seeded stack for harness integration testing. +# +# This file is provider-agnostic and is NOT a working stack on its own. Exactly one +# provider overlay must be composed on top of it: +# +# docker compose -f sandbox/compose.yml -f sandbox/compose.mock.yml up -d # default +# docker compose -f sandbox/compose.yml -f sandbox/compose.real.yml up -d +# +# Use sandbox/sandbox.sh rather than driving compose directly. + +name: honcho-sandbox + +# Everything the api and deriver need in order to behave identically and promptly. +# Provider wiring lives in the overlays; this block is provider-independent. +x-honcho-env: &honcho-env + DB_CONNECTION_URI: postgresql+psycopg://postgres:postgres@database:5432/honcho_sandbox + CACHE_URL: redis://redis:6379/0?suppress=true + CACHE_ENABLED: "true" + AUTH_USE_AUTH: "false" + LOG_LEVEL: INFO + + # The sandbox is configured only by what Compose injects here. + # + # src/config.py calls load_dotenv(override=True) at import, so a stray .env inside + # the image would beat this block. Dockerfile's `COPY config.toml* /app/` means a + # developer's local config.toml gets baked into any locally built image. Both are + # switched off so the stack cannot inherit machine state. + PYTHON_DOTENV_DISABLED: "1" + HONCHO_CONFIG_TOML_DISABLED: "1" + + # Derivation has to happen now, not eventually. On stock settings a sandbox seeded + # with a handful of messages produces zero conclusions, and it fails silently — it + # reads as "the deriver found nothing" rather than "the deriver never ran". + # + # Three separate gates have to come off (src/config.py, DeriverSettings): + # - work units are not claimed until the batch reaches 512 tokens... + # - ...or REPRESENTATION_BATCH_MAX_AGE_SECONDS (30 min) elapses + # - startup jitter delays the first poll by up to 30s, then backoff stretches + # the interval out to 30s + DERIVER_FLUSH_ENABLED: "true" + DERIVER_REPRESENTATION_BATCH_WORK_UNIT_TARGET_TOKENS: "0" + DERIVER_REPRESENTATION_BATCH_MAX_AGE_SECONDS: "1" + DERIVER_POLLING_STARTUP_JITTER_SECONDS: "0" + DERIVER_POLLING_JITTER_RATIO: "0" + DERIVER_POLLING_BACKOFF_ENABLED: "false" + DERIVER_POLLING_SLEEP_INTERVAL_SECONDS: "0.25" + DERIVER_STALE_WORK_UNIT_CLEANUP_INTERVAL_SECONDS: "5" + + # Dreams never fire on their own here: the document threshold is 50 and the + # minimum gap between dreams is 8 hours. seed.py calls schedule_dream directly, + # which bypasses both — but the subsystem still has to be enabled. + DREAM_ENABLED: "true" + +services: + api: + image: ${HONCHO_SANDBOX_IMAGE:?set by sandbox.sh from sandbox/image.env} + entrypoint: ["sh", "docker/entrypoint.sh"] + depends_on: + database: + condition: service_healthy + redis: + condition: service_healthy + ports: + - "127.0.0.1:${SANDBOX_API_PORT:-18000}:8000" + healthcheck: + test: + [ + "CMD", + "/app/.venv/bin/python", + "-c", + "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).read()", + ] + interval: 3s + timeout: 5s + retries: 20 + start_period: 10s + environment: *honcho-env + restart: "no" + + deriver: + image: ${HONCHO_SANDBOX_IMAGE:?set by sandbox.sh from sandbox/image.env} + entrypoint: ["/app/.venv/bin/python", "-m", "src.deriver"] + depends_on: + api: + condition: service_healthy + database: + condition: service_healthy + redis: + condition: service_healthy + environment: *honcho-env + restart: "no" + + database: + image: pgvector/pgvector:pg15 + ports: + - "127.0.0.1:${SANDBOX_DB_PORT:-15432}:5432" + command: ["postgres", "-c", "max_connections=200"] + environment: + POSTGRES_DB: postgres + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + # The port is bound to 127.0.0.1 and the whole database is disposable. + POSTGRES_HOST_AUTH_METHOD: trust + PGDATA: /var/lib/postgresql/data/pgdata + volumes: + # Creates the honcho_sandbox database. database/init.sql only adds the vector + # extension to `postgres`, and the sandbox needs its own database so that + # reset can DROP and re-CREATE it from a template. + - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro + - pgdata:/var/lib/postgresql/data/ + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d honcho_sandbox"] + interval: 3s + timeout: 5s + retries: 20 + restart: "no" + + redis: + image: redis:8.2 + ports: + - "127.0.0.1:${SANDBOX_REDIS_PORT:-16379}:6379" + healthcheck: + test: ["CMD-SHELL", "redis-cli ping"] + interval: 3s + timeout: 5s + retries: 20 + restart: "no" + +# Named volumes are prefixed with the project name (honcho-sandbox_*), so they +# never collide with a developer's existing local stack. `sandbox.sh down` removes +# them. Redis is deliberately unbacked — its state is rebuilt by reset. +volumes: + pgdata: diff --git a/sandbox/fixture.json b/sandbox/fixture.json new file mode 100644 index 00000000..39894862 --- /dev/null +++ b/sandbox/fixture.json @@ -0,0 +1,41 @@ +{ + "workspace": "sandbox", + "session": "sandbox-session", + + "_comment": "Peer topology is stated explicitly because it is the thing that silently breaks (DEV-2462 asserts on it). alice is observed and does not observe; the assistant observes and is not observed - the standard harness shape.", + "peers": [ + { "id": "alice", "observe_me": true, "observe_others": false }, + { "id": "assistant", "observe_me": false, "observe_others": true } + ], + + "_messages_comment": "Facts are deliberately distinctive and lexically searchable, so a recall assertion can look for an exact token. Under the mock provider, embeddings carry no semantic similarity, so full-text is the only thing that can work.", + "messages": [ + { + "peer": "alice", + "content": "I'm Alice Nakamura, I work as a structural engineer in Rotterdam." + }, + { + "peer": "assistant", + "content": "Good to meet you, Alice. What are you working on in Rotterdam?" + }, + { + "peer": "alice", + "content": "A pedestrian bridge over the Nieuwe Maas. We're using weathering steel because the maintenance budget is basically zero." + }, + { + "peer": "assistant", + "content": "Weathering steel is a reasonable call for a low-maintenance span. How long is the crossing?" + }, + { + "peer": "alice", + "content": "About 240 metres, with a single cable-stayed pylon. My cat Marzipan is named after the pylon's colour, which tells you how much I think about this bridge." + }, + { + "peer": "assistant", + "content": "240 metres on a single pylon is ambitious. And Marzipan is a good name for a cat." + } + ], + + "_dreams_comment": "Dreams never fire on their own here - the document threshold is 50 and the minimum gap is 8 hours - so the seed schedules one directly.", + "dreams": [{ "observer": "assistant", "observed": "alice" }] +} diff --git a/sandbox/image.env b/sandbox/image.env new file mode 100644 index 00000000..32606781 --- /dev/null +++ b/sandbox/image.env @@ -0,0 +1,16 @@ +# The image the sandbox runs, pinned by digest so a run is reproducible. +# +# Bump this deliberately. The whole point of the sandbox is that two people on two +# machines exercise the same bytes, which a floating tag does not give you. +# +# This is the multi-arch index digest (linux/amd64 + linux/arm64), so it resolves on +# both Apple Silicon and CI. +# +# NOTE: this digest predates PR #1094, so it does NOT contain src/mock_provider. +# Until #1094 lands and a new image is published, mock mode needs --build: +# +# sandbox/sandbox.sh up --build +# +# sandbox.sh checks for the module and tells you this rather than failing obscurely. +# When #1094 is merged and published, bump the digest here and drop this note. +HONCHO_SANDBOX_IMAGE=ghcr.io/plastic-labs/honcho@sha256:3e62ce949cd020155e59d58302e11a20465c79815d80e2d4eecd106e9c8f9ab4 diff --git a/sandbox/init.sql b/sandbox/init.sql new file mode 100644 index 00000000..928e4c48 --- /dev/null +++ b/sandbox/init.sql @@ -0,0 +1,9 @@ +-- Runs once, on first initialisation of an empty PGDATA. +-- +-- The sandbox uses its own database rather than `postgres` so that reset can DROP +-- and re-CREATE it from a seeded template. You cannot drop the database you are +-- connected to, and `postgres` is the default connection target for maintenance. +CREATE DATABASE honcho_sandbox; + +\connect honcho_sandbox +CREATE EXTENSION IF NOT EXISTS vector; diff --git a/sandbox/real.env.example b/sandbox/real.env.example new file mode 100644 index 00000000..aa28486d --- /dev/null +++ b/sandbox/real.env.example @@ -0,0 +1,21 @@ +# Copy to sandbox/real.env and fill in. real.env is gitignored; this file is not. +# +# cp sandbox/real.env.example sandbox/real.env +# sandbox/sandbox.sh up --provider real +# +# Only needed for real mode. Mock mode needs nothing here. + +# One provider key. Honcho tests it for truthiness before building the client. +LLM_OPENAI_API_KEY=sk-replace-me +# LLM_ANTHROPIC_API_KEY= +# LLM_GEMINI_API_KEY= + +# Optional: a cheaper model than the gpt-5.4-mini default. The sandbox derives on +# every seed, so this is the knob that decides what a seed costs. +# DERIVER_MODEL_CONFIG__TRANSPORT=openai +# DERIVER_MODEL_CONFIG__MODEL=gpt-5.4-mini + +# Deliberately absent: EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL. +# Leaving it unset is what sends embeddings to the real provider — which is the +# entire reason to run real mode, since mock embeddings carry no semantic +# similarity and cannot support a vector-recall assertion. diff --git a/sandbox/sandbox.sh b/sandbox/sandbox.sh new file mode 100755 index 00000000..2f5a19d3 --- /dev/null +++ b/sandbox/sandbox.sh @@ -0,0 +1,326 @@ +#!/usr/bin/env bash +# +# Honcho sandbox — an ephemeral, seeded stack that resets to a known state in +# seconds, so harness testing stops depending on machine state. +# +# sandbox.sh up [--provider mock|real] [--build] +# sandbox.sh seed [--provider ...] +# sandbox.sh reset [--provider ...] +# sandbox.sh status [--provider ...] +# sandbox.sh down [--provider ...] +# +# See README.md. + +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$HERE/.." && pwd)" + +DB=honcho_sandbox +BUILT_IMAGE=honcho-sandbox/honcho:built + +PROVIDER="${HONCHO_SANDBOX_PROVIDER:-mock}" +BUILD=0 + +die() { echo "error: $*" >&2; exit 1; } +say() { echo "==> $*"; } + +# -------------------------------------------------------------------------- +# Argument parsing +# -------------------------------------------------------------------------- + +COMMAND="${1:-}" +[ -n "$COMMAND" ] || die "usage: sandbox.sh {up|seed|reset|status|down} [--provider mock|real] [--build]" +shift + +while [ $# -gt 0 ]; do + case "$1" in + --provider) PROVIDER="${2:-}"; shift 2 ;; + --provider=*) PROVIDER="${1#*=}"; shift ;; + --build) BUILD=1; shift ;; + *) die "unknown argument: $1" ;; + esac +done + +case "$PROVIDER" in + mock|real) ;; + *) die "unknown provider '$PROVIDER' (expected 'mock' or 'real')" ;; +esac + +# Each mode keeps its own template, so both can coexist and switching modes is +# free rather than forcing a reseed. +TEMPLATE="${DB}_seeded_${PROVIDER}" + +# -------------------------------------------------------------------------- +# Compose invocation +# +# The base file is provider-agnostic and is not a working stack on its own; +# exactly one provider overlay is always composed on top. +# -------------------------------------------------------------------------- + +# The pinned default... +# shellcheck disable=SC1091 +set -a; . "$HERE/image.env"; set +a + +# ...then whatever the running stack was actually started with. Without this, +# `up --build` followed by a plain `reset` would flip the image back to the pinned +# digest, and Compose would helpfully recreate every container from it mid-reset. +STATE="$HERE/.state.env" +if [ "$BUILD" = 1 ]; then + HONCHO_SANDBOX_IMAGE="$BUILT_IMAGE" +elif [ -f "$STATE" ]; then + # shellcheck disable=SC1090 + set -a; . "$STATE"; set +a +fi +export HONCHO_SANDBOX_IMAGE + +compose() { + docker compose \ + -f "$HERE/compose.yml" \ + -f "$HERE/compose.$PROVIDER.yml" \ + --project-directory "$HERE" \ + "$@" +} + +psql_db() { + # Maintenance connections go to `postgres`, never to $DB — you cannot drop a + # database you are connected to. + compose exec -T database psql -v ON_ERROR_STOP=1 -U postgres -d postgres "$@" +} + +db_exists() { + [ "$(psql_db -tAc "SELECT 1 FROM pg_database WHERE datname='$1'" 2>/dev/null)" = "1" ] +} + +# -------------------------------------------------------------------------- +# Preconditions +# -------------------------------------------------------------------------- + +preflight() { + docker info >/dev/null 2>&1 || die "the Docker daemon is not running" + + if [ "$PROVIDER" = real ]; then + # Fail before starting anything. A real-mode stack with no key boots fine and + # then derives nothing, which reads as "the deriver found nothing". + [ -f "$HERE/real.env" ] || die \ + "real mode needs $HERE/real.env. Copy real.env.example and add a provider key." + grep -Eq '^\s*LLM_[A-Z]+_API_KEY=.+' "$HERE/real.env" || die \ + "no LLM_*_API_KEY with a value in real.env." + if grep -Eq '^\s*LLM_[A-Z]+_API_KEY=(sk-replace-me|your-api-key-here)\s*$' "$HERE/real.env"; then + die "real.env still has the placeholder key from real.env.example." + fi + fi + # Must not end on a failed test. Under `set -e`, a non-zero return from the last + # command in this function aborts the script with no output at all — which is + # exactly how the first real-mode `up` "failed": silently, before doing anything. + return 0 +} + +# The pinned digest predates PR #1094, so it carries no src/mock_provider. Say so +# precisely instead of letting the service crash-loop on a missing module. +check_mock_provider_present() { + [ "$PROVIDER" = mock ] || return 0 + if ! docker run --rm --entrypoint test "$HONCHO_SANDBOX_IMAGE" -d /app/src/mock_provider; then + die "$(cat </dev/null 2>&1; then + # Already here — either pulled on an earlier run, or built locally by --build + # and recorded in .state.env, in which case pulling it would fail outright + # because the tag exists in no registry. + say "using local image $HONCHO_SANDBOX_IMAGE" + else + say "pulling $HONCHO_SANDBOX_IMAGE" + docker pull --quiet "$HONCHO_SANDBOX_IMAGE" >/dev/null + fi + + check_mock_provider_present + + say "starting the $PROVIDER stack" + # --remove-orphans because both modes share one Compose project: switching from + # mock to real leaves mock-provider running and otherwise unreferenced, where it + # would sit idle and make `status` misleading about what the stack is using. + compose up -d --wait --remove-orphans + echo "HONCHO_SANDBOX_IMAGE=$HONCHO_SANDBOX_IMAGE" > "$STATE" + + if db_exists "$TEMPLATE"; then + say "template $TEMPLATE already present — resetting to it" + cmd_reset + else + cmd_seed + fi + + echo + say "sandbox ready at http://127.0.0.1:${SANDBOX_API_PORT:-18000} (provider: $PROVIDER)" +} + +cmd_seed() { + preflight + # Seeding an already-seeded database appends to it — the fixture would land a + # second time and the "expected 6 messages, found 12" check in seed.py would + # (correctly) fail. Start from an empty, freshly migrated database every time so + # `seed` means the same thing regardless of what was there before. + say "clearing the database before seeding" + compose stop api deriver >/dev/null + psql_db -c "DROP DATABASE IF EXISTS $DB WITH (FORCE)" >/dev/null + psql_db -c "CREATE DATABASE $DB" >/dev/null + compose exec -T database psql -q -v ON_ERROR_STOP=1 -U postgres -d "$DB" \ + -c "CREATE EXTENSION IF NOT EXISTS vector" >/dev/null + # Redis still holds entries keyed to the database that was just dropped; without + # this the api serves cached peers that no longer exist and seeding conflicts. + compose exec -T redis redis-cli FLUSHALL >/dev/null + # The api entrypoint runs scripts/provision_db.py, so starting it is what applies + # the Alembic migrations to the new database. + compose start api deriver >/dev/null + compose up -d --wait --no-recreate >/dev/null + + say "seeding ($PROVIDER)" + SANDBOX_BASE_URL="http://127.0.0.1:${SANDBOX_API_PORT:-18000}" \ + uv run --no-project --with-editable "$REPO/sdks/python" \ + python "$HERE/seed.py" + + say "snapshotting to $TEMPLATE" + stamp_fingerprint + # api and deriver hold pooled connections; CREATE DATABASE ... TEMPLATE needs + # the source to have none. + compose stop api deriver >/dev/null + disconnect_db "$DB" + psql_db -c "DROP DATABASE IF EXISTS $TEMPLATE" >/dev/null + psql_db -c "CREATE DATABASE $TEMPLATE TEMPLATE $DB" >/dev/null + compose start api deriver >/dev/null + compose up -d --wait --no-recreate >/dev/null + say "snapshot ready — reset restores this state without re-deriving" +} + +cmd_reset() { + preflight + db_exists "$TEMPLATE" || die \ + "no template for $PROVIDER mode. Run: sandbox/sandbox.sh seed --provider $PROVIDER" + + check_fingerprint + + # Nothing is stopped or restarted. DROP ... WITH (FORCE) evicts the api and + # deriver connection pools itself, and both reconnect on their next use + # (db.POOL_PRE_PING is on) — the api on its next request, the deriver on its + # next poll a quarter-second later. + # + # Both log one OperationalError as their in-flight connection dies. That noise + # is the cost of not paying ~9s of container stop/start on every reset, which + # would defeat the point. Restarting them instead is the fallback if this ever + # stops being true. + # + # CREATE DATABASE ... TEMPLATE additionally needs the *source* to be + # connectionless, and check_fingerprint just read from it. + disconnect_db "$TEMPLATE" + psql_db -c "DROP DATABASE $DB WITH (FORCE)" >/dev/null + psql_db -c "CREATE DATABASE $DB TEMPLATE $TEMPLATE" >/dev/null + compose exec -T redis redis-cli FLUSHALL >/dev/null + say "reset to $TEMPLATE" +} + +cmd_status() { + compose ps + echo + echo "provider: $PROVIDER" + echo "image: $HONCHO_SANDBOX_IMAGE" + echo "api: http://127.0.0.1:${SANDBOX_API_PORT:-18000}" + for mode in mock real; do + if db_exists "${DB}_seeded_${mode}"; then + echo "template: ${DB}_seeded_${mode} present" + else + echo "template: ${DB}_seeded_${mode} absent" + fi + done +} + +cmd_down() { + say "tearing down the $PROVIDER stack and its volumes" + compose down -v + rm -f "$STATE" +} + +# -------------------------------------------------------------------------- +# Snapshot staleness +# +# A template that predates a migration, or was built against a different +# provider, restores silently and wrongly. Both are recorded inside the template +# itself so reset can refuse rather than mislead. +# -------------------------------------------------------------------------- + +fingerprint() { + local alembic + alembic="$(compose exec -T database psql -tAX -U postgres -d "$DB" \ + -c "SELECT version_num FROM alembic_version" 2>/dev/null | tr -d '[:space:]')" + local files + files="$(cat "$HERE/fixture.json" "$HERE/seed.py" | shasum -a 256 | cut -c1-16)" + echo "alembic=$alembic fixture=$files provider=$PROVIDER" +} + +stamp_fingerprint() { + compose exec -T database psql -v ON_ERROR_STOP=1 -U postgres -d "$DB" >/dev/null </dev/null | tr -d '\r')" + + # The live database carries whatever revision the api container migrated it to + # on boot; the template carries the revision it was seeded at. A difference is + # exactly the migration drift this guard exists to catch, so it is compared, + # not normalised away. + [ "$want" = "$have" ] || die "$(cat <} + current: $want + +Reseed: sandbox/sandbox.sh seed --provider $PROVIDER +MSG +)" +} + +disconnect_db() { + # Defaults to the live database. CREATE DATABASE ... TEMPLATE also requires the + # *source* to have no connections, and reading the fingerprint just opened one. + local target="${1:-$DB}" + psql_db -c \ + "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='$target' AND pid <> pg_backend_pid()" \ + >/dev/null +} + +case "$COMMAND" in + up) cmd_up ;; + seed) cmd_seed ;; + reset) cmd_reset ;; + status) cmd_status ;; + down) cmd_down ;; + *) die "unknown command '$COMMAND' (expected up, seed, reset, status, or down)" ;; +esac diff --git a/sandbox/seed.py b/sandbox/seed.py new file mode 100644 index 00000000..9ec544b3 --- /dev/null +++ b/sandbox/seed.py @@ -0,0 +1,184 @@ +"""Populate the sandbox with a known fixture and verify it actually landed. + +Run through sandbox.sh, which supplies the base URL and owns the Docker and +Postgres side. This script only talks to the API: + + uv run --no-project --with-editable ./sdks/python python sandbox/seed.py + +It is provider-agnostic. Under the mock provider the *text* of a conclusion is +synthetic and unrelated to the messages, so nothing here asserts on conclusion +content - only that derivation ran and produced something. What it does assert +strictly is the round trip that harness integrations actually get wrong: which +messages landed, on which peers, with which observation topology. + +Exits non-zero on any failure. An empty sandbox that reports success is the exact +failure mode this whole thing exists to prevent. +""" + +from __future__ import annotations + +import json +import os +import sys +import time +from pathlib import Path +from typing import Any + +from honcho import Honcho +from honcho.api_types import MessageCreateParams, SessionPeerConfig + +FIXTURE = Path(__file__).with_name("fixture.json") + +# Generous: a cold deriver on a real provider is slow, and the failure we care +# about (nothing is processing at all) shows up as a timeout either way. +DRAIN_TIMEOUT_SECONDS = float(os.environ.get("SANDBOX_DRAIN_TIMEOUT", "300")) +DRAIN_POLL_SECONDS = 0.5 + + +class SeedError(RuntimeError): + """Seeding did not reach a usable state.""" + + +def log(message: str) -> None: + print(f"[seed] {message}", flush=True) + + +def drain(honcho: Honcho, what: str) -> None: + """Block until the deriver queue is empty, or fail loudly. + + Silence is the dangerous outcome here. A queue that never drains and a queue + that never had anything queued look identical from the outside, so the timeout + reports the last status it saw rather than just giving up. + """ + deadline = time.monotonic() + DRAIN_TIMEOUT_SECONDS + last: Any = None + while time.monotonic() < deadline: + last = honcho.queue_status() + outstanding = last.pending_work_units + last.in_progress_work_units + if outstanding == 0 and last.completed_work_units > 0: + log(f"drained after {what} ({last.completed_work_units} work units)") + return + time.sleep(DRAIN_POLL_SECONDS) + + raise SeedError( + f"queue did not drain after {what} within {DRAIN_TIMEOUT_SECONDS:.0f}s. " + f"Last status: {last}. " + "If completed_work_units is 0, nothing was ever enqueued - check that the " + "deriver is running and that the provider base URL resolves." + ) + + +def main() -> int: + fixture = json.loads(FIXTURE.read_text()) + base_url = os.environ.get("SANDBOX_BASE_URL", "http://127.0.0.1:18000") + workspace_id = fixture["workspace"] + + log(f"seeding workspace {workspace_id!r} at {base_url}") + honcho = Honcho(base_url=base_url, workspace_id=workspace_id, api_key="sandbox") + + peers = {spec["id"]: honcho.peer(spec["id"]) for spec in fixture["peers"]} + session = honcho.session(fixture["session"]) + + session.add_peers( + [ + ( + peers[spec["id"]], + SessionPeerConfig( + observe_me=spec["observe_me"], + observe_others=spec["observe_others"], + ), + ) + for spec in fixture["peers"] + ] + ) + log(f"created {len(peers)} peers with explicit observation topology") + + session.add_messages( + [ + MessageCreateParams(peer_id=msg["peer"], content=msg["content"]) + for msg in fixture["messages"] + ] + ) + log(f"posted {len(fixture['messages'])} messages") + drain(honcho, "messages") + + for dream in fixture.get("dreams", []): + honcho.schedule_dream( + observer=dream["observer"], + observed=dream["observed"], + session=session, + ) + log(f"scheduled dream: {dream['observer']} -> {dream['observed']}") + if fixture.get("dreams"): + drain(honcho, "dream") + + verify(honcho, session, peers, fixture) + log("seed complete") + return 0 + + +def verify( + honcho: Honcho, + session: Any, + peers: dict[str, Any], + fixture: dict[str, Any], +) -> None: + """Assert the fixture round-tripped. Raises SeedError on any mismatch.""" + stored = list(session.messages(size=100)) + if len(stored) != len(fixture["messages"]): + raise SeedError( + f"expected {len(fixture['messages'])} messages, found {len(stored)}" + ) + + expected_authors = sorted(msg["peer"] for msg in fixture["messages"]) + actual_authors = sorted(msg.peer_id for msg in stored) + if actual_authors != expected_authors: + raise SeedError( + f"messages landed on the wrong peers: expected {expected_authors}, " + f"got {actual_authors}" + ) + log(f"verified {len(stored)} messages on the expected peers") + + # Topology is the thing that silently breaks, so it is checked against the + # server's view rather than assumed from the create call. + for spec in fixture["peers"]: + config = session.get_peer_configuration(spec["id"]) + if (config.observe_me, config.observe_others) != ( + spec["observe_me"], + spec["observe_others"], + ): + raise SeedError( + f"peer {spec['id']!r} topology mismatch: expected " + f"observe_me={spec['observe_me']} " + f"observe_others={spec['observe_others']}, got {config}" + ) + log("verified observation topology") + + # Presence only. Under the mock provider, conclusion text is synthetic and + # says nothing about the messages, and the level mix is explicit-only because + # the Dreamer specialists write via tool calls the mock never emits. Asserting + # on either would pass in real mode and fail in mock mode. + total = 0 + for observer_id, observer in peers.items(): + for observed_id in peers: + found = len(list(observer.conclusions_of(observed_id).list(size=100))) + if found: + log(f" {observer_id} -> {observed_id}: {found} conclusions") + total += found + + if total == 0: + raise SeedError( + "no conclusions were derived. The queue drained, so the deriver ran and " + "produced nothing - check the provider wiring " + "(LLM_OPENAI_BASE_URL / EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL) " + "and the deriver logs." + ) + log(f"verified {total} conclusions present") + + +if __name__ == "__main__": + try: + sys.exit(main()) + except SeedError as exc: + print(f"[seed] FAILED: {exc}", file=sys.stderr) + sys.exit(1)