feat(sandbox): committed ephemeral Honcho sandbox with one-command reset
A local Honcho that comes up already seeded and resets to that exact state in under a second, so harness integration testing stops depending on machine state. `docker-compose.yml.example` gave the topology but was an example, unseeded, and had no teardown path, so every harness change was verified against whatever state a given laptop was in. The blocker to fixing that was the deriver's provider call, which made a sandbox neither free nor deterministic; src/mock_provider removes it. Reset restores a Postgres template database rather than re-deriving: `seed` snapshots the finished state, `reset` drops the live database and recreates it from that snapshot, then flushes Redis. Nothing is stopped or restarted -- DROP DATABASE ... WITH (FORCE) evicts the connection pools and both services reconnect on their own. Measured 0.86s in mock mode, 1.29s in real mode against a 361s seed. Two provider modes, both first class. Mock is the default: deterministic, free, no egress. Real points the same stack at a configured provider, reading credentials from one gitignored file rather than ambient environment. The difference matters and is documented: mock embeddings are hash-derived and carry no semantic similarity, so vector-recall assertions are impossible there, and mock conclusions come out explicit-only. On the committed fixture, mock yields 4 synthetic explicit conclusions and real yields 22 across all three levels. The sandbox is configured only by what Compose injects. PYTHON_DOTENV_DISABLED and HONCHO_CONFIG_TOML_DISABLED are both set because src/config.py calls load_dotenv(override=True) at import and the Dockerfile bakes any local config.toml into the image; without them a developer's own provider config wins silently. Deriver scheduling is pinned for the same reason: on stock settings a sandbox seeded with a handful of messages produces zero conclusions and gives no indication why, because work units wait for a 512-token batch or 30 minutes and startup jitter delays the first poll by up to 30s. Snapshots can go stale, so each carries a fingerprint -- Alembic revision, fixture hash, provider mode -- and reset refuses on a mismatch instead of restoring a state that predates a migration. Templates are per-mode, so both can coexist. Note: the pinned digest in sandbox/image.env predates the mock provider, so `--build` is required until that lands and a new image is published. sandbox.sh detects this and says so rather than crash-looping on a missing module.
This commit is contained in:
Eugene Eisenstein
parent
699c99368d
commit
846ecd19f6
|
|
@ -196,3 +196,7 @@ grafana-data/
|
|||
|
||||
# Claude Code addon stuff
|
||||
.omc
|
||||
|
||||
# Sandbox real-provider credentials (see sandbox/real.env.example)
|
||||
sandbox/real.env
|
||||
sandbox/.state.env
|
||||
|
|
|
|||
|
|
@ -0,0 +1,165 @@
|
|||
# Honcho sandbox
|
||||
|
||||
A local Honcho you can wipe and rebuild in seconds, so harness testing stops depending on
|
||||
whatever state your laptop happens to be in.
|
||||
|
||||
It comes up **already seeded** — a known workspace, peers, session, messages, and conclusions the
|
||||
deriver has actually produced — and `reset` returns it to that exact state without re-deriving.
|
||||
|
||||
```bash
|
||||
sandbox/sandbox.sh up # start + seed (first run also builds/pulls)
|
||||
sandbox/sandbox.sh reset # back to the seeded state, under a second
|
||||
sandbox/sandbox.sh status # what's running, which templates exist
|
||||
sandbox/sandbox.sh down # stop and delete the volumes
|
||||
```
|
||||
|
||||
The API lands on <http://127.0.0.1:18000>. Postgres is on 15432 and Redis on 16379 — all
|
||||
deliberately off the defaults, so the sandbox coexists with a normal local stack instead of
|
||||
fighting it for ports and volumes.
|
||||
|
||||
> **Right now you need `--build`.** The mock provider comes from PR #1094, which is not merged, so
|
||||
> no published image contains it. `sandbox.sh` checks and tells you. Once #1094 lands and an image
|
||||
> is published, bump the digest in `image.env` and plain `up` works.
|
||||
|
||||
## Provider modes
|
||||
|
||||
The deriver calls a model provider, which is what made a sandbox neither free nor deterministic.
|
||||
Both ways of resolving that ship here; `mock` is the default.
|
||||
|
||||
| | `mock` (default) | `real` |
|
||||
| --------------------- | ---------------------------------------------------------------- | ---------------------------------------------------- |
|
||||
| **Determinism** | Same conclusions every run | Non-deterministic |
|
||||
| **Cost** | Zero, and no network egress | Real spend |
|
||||
| **Seed time** | ~15s | ~6 min |
|
||||
| **Reset time** | 0.86s | 1.29s |
|
||||
| **Vector recall** | **Untestable** — see below | Testable |
|
||||
| **What you get** | 4 conclusions, all `explicit`, synthetic text | 22 conclusions — 16 `explicit`, 4 `deductive`, 2 `inductive` |
|
||||
| **Use it for** | CI, harness smoke tests, anything that needs a repeatable answer | Recall quality, validating against real model output |
|
||||
|
||||
Those counts are from the committed fixture, measured on both paths. They are the sharpest
|
||||
illustration of the difference: mock gives you four copies of `[mock] mock dummy placeholder …`,
|
||||
while real mode reasons its way from "my cat Marzipan is named after the pylon's colour" to the
|
||||
deductive conclusion *"alice's cat is named after a feature of alice's pedestrian bridge project."*
|
||||
Anything asserting on that kind of inference needs real mode.
|
||||
|
||||
```bash
|
||||
sandbox/sandbox.sh up # mock
|
||||
sandbox/sandbox.sh up --provider real # real
|
||||
```
|
||||
|
||||
Each mode keeps its own seeded template, so both can exist at once and switching between them
|
||||
costs nothing — about 10 seconds, and no re-derivation, because `up` restores that mode's existing
|
||||
template instead of reseeding.
|
||||
|
||||
### Two things that will otherwise cost you a day
|
||||
|
||||
**Mock embeddings carry no semantic similarity.** They are hash-derived, so two paraphrases are
|
||||
exactly as far apart as two unrelated strings. Any recall assertion built on mock mode must use
|
||||
lexical or full-text search. A vector-ranking assertion will fail there for reasons that have
|
||||
nothing to do with the code you are testing — that is what `real` mode is for.
|
||||
|
||||
**Mock conclusions are synthetic.** The text is derived from the request, not from the meaning of
|
||||
your messages, and the level is always `explicit` (the Dreamer's specialists write via tool calls,
|
||||
which the mock deliberately never emits). Assert that conclusions *exist*; do not assert on what
|
||||
they say or on the level mix, or your test will pass in one mode and fail in the other.
|
||||
|
||||
### Real mode
|
||||
|
||||
```bash
|
||||
cp sandbox/real.env.example sandbox/real.env # then add a key
|
||||
sandbox/sandbox.sh up --provider real
|
||||
```
|
||||
|
||||
Credentials come from that one gitignored file, never from your ambient shell environment, so a
|
||||
real-mode run has a single auditable input and the sandbox still doesn't inherit machine state.
|
||||
`sandbox.sh` refuses to start if the file is missing or still carries the placeholder key — a
|
||||
real-mode stack with no key boots perfectly well and then derives nothing, which is indistinguishable
|
||||
from "the deriver found nothing".
|
||||
|
||||
Real mode gets an unexpected benefit from the reset design: derivation is slow and costs money, and
|
||||
the snapshot means you pay for it **once per seed** and reset for free after that. Measured: 361s to
|
||||
seed, 1.29s to reset back to that exact state with no provider calls at all.
|
||||
|
||||
## How reset is fast
|
||||
|
||||
Seeding runs the deriver, and a naive reset would have to run it again. Instead, `seed` snapshots
|
||||
the finished database as a Postgres template:
|
||||
|
||||
```
|
||||
CREATE DATABASE honcho_sandbox_seeded_mock TEMPLATE honcho_sandbox
|
||||
```
|
||||
|
||||
and `reset` drops the live database and re-creates it from that template, then flushes Redis. No
|
||||
migrations, no re-derivation, no LLM calls — **measured at 0.86s**, and byte-identical every time.
|
||||
|
||||
Nothing is stopped or restarted. `DROP DATABASE ... WITH (FORCE)` evicts the api and deriver
|
||||
connection pools, and both reconnect by themselves — the api on its next request, the deriver on its
|
||||
next poll a quarter-second later. Each logs one `OperationalError` as its in-flight connection dies;
|
||||
that is expected, and it is the price of not paying ~9 seconds of container restart on every reset.
|
||||
|
||||
The cost of a snapshot is that it can go stale. Two guards:
|
||||
|
||||
- **Per-mode template names**, so a mock-seeded template can never be restored into a real-mode run.
|
||||
- **A fingerprint recorded inside each template** — the Alembic revision it was seeded at, a hash of
|
||||
`fixture.json` + `seed.py`, and the provider mode. `reset` compares and refuses with a "reseed"
|
||||
message rather than silently restoring a state that predates a migration.
|
||||
|
||||
If you hit that refusal: `sandbox/sandbox.sh seed`.
|
||||
|
||||
## What's in the fixture
|
||||
|
||||
`fixture.json` — two peers with **explicitly stated** observation topology (`alice` is observed and
|
||||
does not observe; `assistant` observes and is not observed), one session, six messages carrying
|
||||
distinctive lexically-searchable facts, and one scheduled cross-peer dream.
|
||||
|
||||
Topology is written down rather than defaulted because it is the thing that silently breaks, and
|
||||
`seed.py` re-reads it back from the server to confirm it took.
|
||||
|
||||
Edit `fixture.json`, then `sandbox/sandbox.sh seed` to rebuild the template. `seed` always starts
|
||||
from an empty, freshly migrated database, so it means the same thing whatever state you run it
|
||||
from — running it twice gives the same result as running it once.
|
||||
|
||||
Dreams never fire on their own here — the document threshold is 50 and the minimum gap between
|
||||
dreams is 8 hours — so the seed schedules one directly.
|
||||
|
||||
## Layout
|
||||
|
||||
| File | Purpose |
|
||||
| ------------------- | ---------------------------------------------------------------------- |
|
||||
| `sandbox.sh` | The entry point. Owns Docker and Postgres. |
|
||||
| `compose.yml` | Provider-agnostic base. **Not a working stack alone.** |
|
||||
| `compose.mock.yml` | Provider overlay: adds the mock-provider service and wiring. |
|
||||
| `compose.real.yml` | Provider overlay: reads `real.env`, no mock service. |
|
||||
| `image.env` | The pinned image digest. One line, bump deliberately. |
|
||||
| `init.sql` | Creates the `honcho_sandbox` database. |
|
||||
| `fixture.json` | The seeded conversation. |
|
||||
| `seed.py` | Populates and verifies. Talks only to the API. |
|
||||
| `real.env.example` | Template for real-mode credentials. |
|
||||
|
||||
Exactly one provider overlay is always composed on top of the base, so the choice is visible in the
|
||||
`-f` list rather than buried in a default. That is also why `compose.yml` alone does not run: the
|
||||
`depends_on` edge to `mock-provider` lives in the mock overlay, because Compose merges `depends_on`
|
||||
additively and an override file cannot remove one.
|
||||
|
||||
## Configuration
|
||||
|
||||
The sandbox is configured **only** by what Compose injects. `PYTHON_DOTENV_DISABLED` and
|
||||
`HONCHO_CONFIG_TOML_DISABLED` are both set, because `src/config.py` calls `load_dotenv(override=True)`
|
||||
at import and the Dockerfile's `COPY config.toml* /app/` bakes a local `config.toml` into any
|
||||
locally built image. Without those two flags your machine's leftovers would quietly win.
|
||||
|
||||
`compose.yml` also pins the deriver's scheduling so derivation happens *now*. On stock settings a
|
||||
sandbox seeded with a handful of messages produces zero conclusions and fails silently: work units
|
||||
aren't claimed until a batch reaches 512 tokens or 30 minutes pass, and startup jitter delays the
|
||||
first poll by up to 30 seconds. Those are turned off here — see the comments in `compose.yml`.
|
||||
|
||||
## Building from the working tree
|
||||
|
||||
```bash
|
||||
sandbox/sandbox.sh up --build
|
||||
```
|
||||
|
||||
Builds the repo instead of pulling the pinned digest, and points every service at the result so
|
||||
they stay in sync. Use it when testing a change to Honcho itself. It trades reproducibility for
|
||||
currency — you are running your tree, not the pinned bytes, which is right while iterating and
|
||||
wrong when reproducing someone else's result.
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
# Provider overlay: deterministic, free, no network egress. The sandbox default.
|
||||
#
|
||||
# Adds src/mock_provider (PR #1094) as its own service. It runs from the standard
|
||||
# Honcho image under a different entrypoint, the way api and deriver already differ,
|
||||
# so there is no second image to build or keep in digest-sync.
|
||||
|
||||
services:
|
||||
mock-provider:
|
||||
image: ${HONCHO_SANDBOX_IMAGE:?set by sandbox.sh from sandbox/image.env}
|
||||
entrypoint:
|
||||
["/app/.venv/bin/fastapi", "run", "--host", "0.0.0.0", "src/mock_provider/main.py"]
|
||||
healthcheck:
|
||||
# The app exposes only POST /chat/completions and POST /embeddings — no health
|
||||
# or models route — so liveness is checked against the schema FastAPI always
|
||||
# serves. A GET keeps the check side-effect free.
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"/app/.venv/bin/python",
|
||||
"-c",
|
||||
"import urllib.request; urllib.request.urlopen('http://localhost:8000/openapi.json', timeout=2).read()",
|
||||
]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
start_period: 5s
|
||||
restart: "no"
|
||||
|
||||
api: &provider-wiring
|
||||
depends_on:
|
||||
mock-provider:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
# All three are required (src/mock_provider/main.py).
|
||||
#
|
||||
# The key's value is never checked — the mock reads no Authorization header,
|
||||
# and Honcho only tests it for truthiness before building the client
|
||||
# (src/llm/registry.py). Set the base URL without it and the client is never
|
||||
# constructed, so the base URL is silently ignored. Keep the value obviously
|
||||
# fake: a module that ever escapes the override then 401s rather than spends.
|
||||
LLM_OPENAI_API_KEY: sandbox-not-a-real-key
|
||||
LLM_OPENAI_BASE_URL: http://mock-provider:8000/v1
|
||||
# Embeddings resolve through a separate client that reads the base URL only
|
||||
# from this per-module override. Without it, embedding calls go to
|
||||
# api.openai.com for real. Do not add a per-module credential override
|
||||
# (..._OVERRIDES__API_KEY / API_KEY_ENV) — that makes the module ignore the
|
||||
# global base URL above.
|
||||
EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL: http://mock-provider:8000/v1
|
||||
|
||||
deriver: *provider-wiring
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
# Provider overlay: a real, configured model provider.
|
||||
#
|
||||
# Non-deterministic and it costs money. Use it when you need something the mock
|
||||
# cannot give you — chiefly vector recall, since mock embeddings are hash-derived
|
||||
# and carry no semantic similarity, and the full conclusion level mix.
|
||||
#
|
||||
# Credentials come from sandbox/real.env (gitignored), never from ambient host
|
||||
# environment, so a real-mode run has one auditable input and the sandbox still does
|
||||
# not depend on whatever happens to be exported in your shell.
|
||||
#
|
||||
# Copy sandbox/real.env.example to sandbox/real.env to get started. sandbox.sh
|
||||
# refuses to start real mode if that file is missing or carries no key.
|
||||
#
|
||||
# No mock-provider service is defined here, so nothing depends on one.
|
||||
|
||||
services:
|
||||
api: &provider-wiring
|
||||
env_file:
|
||||
- path: real.env
|
||||
required: true
|
||||
|
||||
deriver: *provider-wiring
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
# Honcho sandbox — an ephemeral, seeded stack for harness integration testing.
|
||||
#
|
||||
# This file is provider-agnostic and is NOT a working stack on its own. Exactly one
|
||||
# provider overlay must be composed on top of it:
|
||||
#
|
||||
# docker compose -f sandbox/compose.yml -f sandbox/compose.mock.yml up -d # default
|
||||
# docker compose -f sandbox/compose.yml -f sandbox/compose.real.yml up -d
|
||||
#
|
||||
# Use sandbox/sandbox.sh rather than driving compose directly.
|
||||
|
||||
name: honcho-sandbox
|
||||
|
||||
# Everything the api and deriver need in order to behave identically and promptly.
|
||||
# Provider wiring lives in the overlays; this block is provider-independent.
|
||||
x-honcho-env: &honcho-env
|
||||
DB_CONNECTION_URI: postgresql+psycopg://postgres:postgres@database:5432/honcho_sandbox
|
||||
CACHE_URL: redis://redis:6379/0?suppress=true
|
||||
CACHE_ENABLED: "true"
|
||||
AUTH_USE_AUTH: "false"
|
||||
LOG_LEVEL: INFO
|
||||
|
||||
# The sandbox is configured only by what Compose injects here.
|
||||
#
|
||||
# src/config.py calls load_dotenv(override=True) at import, so a stray .env inside
|
||||
# the image would beat this block. Dockerfile's `COPY config.toml* /app/` means a
|
||||
# developer's local config.toml gets baked into any locally built image. Both are
|
||||
# switched off so the stack cannot inherit machine state.
|
||||
PYTHON_DOTENV_DISABLED: "1"
|
||||
HONCHO_CONFIG_TOML_DISABLED: "1"
|
||||
|
||||
# Derivation has to happen now, not eventually. On stock settings a sandbox seeded
|
||||
# with a handful of messages produces zero conclusions, and it fails silently — it
|
||||
# reads as "the deriver found nothing" rather than "the deriver never ran".
|
||||
#
|
||||
# Three separate gates have to come off (src/config.py, DeriverSettings):
|
||||
# - work units are not claimed until the batch reaches 512 tokens...
|
||||
# - ...or REPRESENTATION_BATCH_MAX_AGE_SECONDS (30 min) elapses
|
||||
# - startup jitter delays the first poll by up to 30s, then backoff stretches
|
||||
# the interval out to 30s
|
||||
DERIVER_FLUSH_ENABLED: "true"
|
||||
DERIVER_REPRESENTATION_BATCH_WORK_UNIT_TARGET_TOKENS: "0"
|
||||
DERIVER_REPRESENTATION_BATCH_MAX_AGE_SECONDS: "1"
|
||||
DERIVER_POLLING_STARTUP_JITTER_SECONDS: "0"
|
||||
DERIVER_POLLING_JITTER_RATIO: "0"
|
||||
DERIVER_POLLING_BACKOFF_ENABLED: "false"
|
||||
DERIVER_POLLING_SLEEP_INTERVAL_SECONDS: "0.25"
|
||||
DERIVER_STALE_WORK_UNIT_CLEANUP_INTERVAL_SECONDS: "5"
|
||||
|
||||
# Dreams never fire on their own here: the document threshold is 50 and the
|
||||
# minimum gap between dreams is 8 hours. seed.py calls schedule_dream directly,
|
||||
# which bypasses both — but the subsystem still has to be enabled.
|
||||
DREAM_ENABLED: "true"
|
||||
|
||||
services:
|
||||
api:
|
||||
image: ${HONCHO_SANDBOX_IMAGE:?set by sandbox.sh from sandbox/image.env}
|
||||
entrypoint: ["sh", "docker/entrypoint.sh"]
|
||||
depends_on:
|
||||
database:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "127.0.0.1:${SANDBOX_API_PORT:-18000}:8000"
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"/app/.venv/bin/python",
|
||||
"-c",
|
||||
"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).read()",
|
||||
]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
start_period: 10s
|
||||
environment: *honcho-env
|
||||
restart: "no"
|
||||
|
||||
deriver:
|
||||
image: ${HONCHO_SANDBOX_IMAGE:?set by sandbox.sh from sandbox/image.env}
|
||||
entrypoint: ["/app/.venv/bin/python", "-m", "src.deriver"]
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
database:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
environment: *honcho-env
|
||||
restart: "no"
|
||||
|
||||
database:
|
||||
image: pgvector/pgvector:pg15
|
||||
ports:
|
||||
- "127.0.0.1:${SANDBOX_DB_PORT:-15432}:5432"
|
||||
command: ["postgres", "-c", "max_connections=200"]
|
||||
environment:
|
||||
POSTGRES_DB: postgres
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
# The port is bound to 127.0.0.1 and the whole database is disposable.
|
||||
POSTGRES_HOST_AUTH_METHOD: trust
|
||||
PGDATA: /var/lib/postgresql/data/pgdata
|
||||
volumes:
|
||||
# Creates the honcho_sandbox database. database/init.sql only adds the vector
|
||||
# extension to `postgres`, and the sandbox needs its own database so that
|
||||
# reset can DROP and re-CREATE it from a template.
|
||||
- ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro
|
||||
- pgdata:/var/lib/postgresql/data/
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres -d honcho_sandbox"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
restart: "no"
|
||||
|
||||
redis:
|
||||
image: redis:8.2
|
||||
ports:
|
||||
- "127.0.0.1:${SANDBOX_REDIS_PORT:-16379}:6379"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "redis-cli ping"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
restart: "no"
|
||||
|
||||
# Named volumes are prefixed with the project name (honcho-sandbox_*), so they
|
||||
# never collide with a developer's existing local stack. `sandbox.sh down` removes
|
||||
# them. Redis is deliberately unbacked — its state is rebuilt by reset.
|
||||
volumes:
|
||||
pgdata:
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
{
|
||||
"workspace": "sandbox",
|
||||
"session": "sandbox-session",
|
||||
|
||||
"_comment": "Peer topology is stated explicitly because it is the thing that silently breaks (DEV-2462 asserts on it). alice is observed and does not observe; the assistant observes and is not observed - the standard harness shape.",
|
||||
"peers": [
|
||||
{ "id": "alice", "observe_me": true, "observe_others": false },
|
||||
{ "id": "assistant", "observe_me": false, "observe_others": true }
|
||||
],
|
||||
|
||||
"_messages_comment": "Facts are deliberately distinctive and lexically searchable, so a recall assertion can look for an exact token. Under the mock provider, embeddings carry no semantic similarity, so full-text is the only thing that can work.",
|
||||
"messages": [
|
||||
{
|
||||
"peer": "alice",
|
||||
"content": "I'm Alice Nakamura, I work as a structural engineer in Rotterdam."
|
||||
},
|
||||
{
|
||||
"peer": "assistant",
|
||||
"content": "Good to meet you, Alice. What are you working on in Rotterdam?"
|
||||
},
|
||||
{
|
||||
"peer": "alice",
|
||||
"content": "A pedestrian bridge over the Nieuwe Maas. We're using weathering steel because the maintenance budget is basically zero."
|
||||
},
|
||||
{
|
||||
"peer": "assistant",
|
||||
"content": "Weathering steel is a reasonable call for a low-maintenance span. How long is the crossing?"
|
||||
},
|
||||
{
|
||||
"peer": "alice",
|
||||
"content": "About 240 metres, with a single cable-stayed pylon. My cat Marzipan is named after the pylon's colour, which tells you how much I think about this bridge."
|
||||
},
|
||||
{
|
||||
"peer": "assistant",
|
||||
"content": "240 metres on a single pylon is ambitious. And Marzipan is a good name for a cat."
|
||||
}
|
||||
],
|
||||
|
||||
"_dreams_comment": "Dreams never fire on their own here - the document threshold is 50 and the minimum gap is 8 hours - so the seed schedules one directly.",
|
||||
"dreams": [{ "observer": "assistant", "observed": "alice" }]
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
# The image the sandbox runs, pinned by digest so a run is reproducible.
|
||||
#
|
||||
# Bump this deliberately. The whole point of the sandbox is that two people on two
|
||||
# machines exercise the same bytes, which a floating tag does not give you.
|
||||
#
|
||||
# This is the multi-arch index digest (linux/amd64 + linux/arm64), so it resolves on
|
||||
# both Apple Silicon and CI.
|
||||
#
|
||||
# NOTE: this digest predates PR #1094, so it does NOT contain src/mock_provider.
|
||||
# Until #1094 lands and a new image is published, mock mode needs --build:
|
||||
#
|
||||
# sandbox/sandbox.sh up --build
|
||||
#
|
||||
# sandbox.sh checks for the module and tells you this rather than failing obscurely.
|
||||
# When #1094 is merged and published, bump the digest here and drop this note.
|
||||
HONCHO_SANDBOX_IMAGE=ghcr.io/plastic-labs/honcho@sha256:3e62ce949cd020155e59d58302e11a20465c79815d80e2d4eecd106e9c8f9ab4
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
-- Runs once, on first initialisation of an empty PGDATA.
|
||||
--
|
||||
-- The sandbox uses its own database rather than `postgres` so that reset can DROP
|
||||
-- and re-CREATE it from a seeded template. You cannot drop the database you are
|
||||
-- connected to, and `postgres` is the default connection target for maintenance.
|
||||
CREATE DATABASE honcho_sandbox;
|
||||
|
||||
\connect honcho_sandbox
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
# Copy to sandbox/real.env and fill in. real.env is gitignored; this file is not.
|
||||
#
|
||||
# cp sandbox/real.env.example sandbox/real.env
|
||||
# sandbox/sandbox.sh up --provider real
|
||||
#
|
||||
# Only needed for real mode. Mock mode needs nothing here.
|
||||
|
||||
# One provider key. Honcho tests it for truthiness before building the client.
|
||||
LLM_OPENAI_API_KEY=sk-replace-me
|
||||
# LLM_ANTHROPIC_API_KEY=
|
||||
# LLM_GEMINI_API_KEY=
|
||||
|
||||
# Optional: a cheaper model than the gpt-5.4-mini default. The sandbox derives on
|
||||
# every seed, so this is the knob that decides what a seed costs.
|
||||
# DERIVER_MODEL_CONFIG__TRANSPORT=openai
|
||||
# DERIVER_MODEL_CONFIG__MODEL=gpt-5.4-mini
|
||||
|
||||
# Deliberately absent: EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL.
|
||||
# Leaving it unset is what sends embeddings to the real provider — which is the
|
||||
# entire reason to run real mode, since mock embeddings carry no semantic
|
||||
# similarity and cannot support a vector-recall assertion.
|
||||
|
|
@ -0,0 +1,326 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# Honcho sandbox — an ephemeral, seeded stack that resets to a known state in
|
||||
# seconds, so harness testing stops depending on machine state.
|
||||
#
|
||||
# sandbox.sh up [--provider mock|real] [--build]
|
||||
# sandbox.sh seed [--provider ...]
|
||||
# sandbox.sh reset [--provider ...]
|
||||
# sandbox.sh status [--provider ...]
|
||||
# sandbox.sh down [--provider ...]
|
||||
#
|
||||
# See README.md.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd "$HERE/.." && pwd)"
|
||||
|
||||
DB=honcho_sandbox
|
||||
BUILT_IMAGE=honcho-sandbox/honcho:built
|
||||
|
||||
PROVIDER="${HONCHO_SANDBOX_PROVIDER:-mock}"
|
||||
BUILD=0
|
||||
|
||||
die() { echo "error: $*" >&2; exit 1; }
|
||||
say() { echo "==> $*"; }
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Argument parsing
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
COMMAND="${1:-}"
|
||||
[ -n "$COMMAND" ] || die "usage: sandbox.sh {up|seed|reset|status|down} [--provider mock|real] [--build]"
|
||||
shift
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--provider) PROVIDER="${2:-}"; shift 2 ;;
|
||||
--provider=*) PROVIDER="${1#*=}"; shift ;;
|
||||
--build) BUILD=1; shift ;;
|
||||
*) die "unknown argument: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
case "$PROVIDER" in
|
||||
mock|real) ;;
|
||||
*) die "unknown provider '$PROVIDER' (expected 'mock' or 'real')" ;;
|
||||
esac
|
||||
|
||||
# Each mode keeps its own template, so both can coexist and switching modes is
|
||||
# free rather than forcing a reseed.
|
||||
TEMPLATE="${DB}_seeded_${PROVIDER}"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Compose invocation
|
||||
#
|
||||
# The base file is provider-agnostic and is not a working stack on its own;
|
||||
# exactly one provider overlay is always composed on top.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
# The pinned default...
|
||||
# shellcheck disable=SC1091
|
||||
set -a; . "$HERE/image.env"; set +a
|
||||
|
||||
# ...then whatever the running stack was actually started with. Without this,
|
||||
# `up --build` followed by a plain `reset` would flip the image back to the pinned
|
||||
# digest, and Compose would helpfully recreate every container from it mid-reset.
|
||||
STATE="$HERE/.state.env"
|
||||
if [ "$BUILD" = 1 ]; then
|
||||
HONCHO_SANDBOX_IMAGE="$BUILT_IMAGE"
|
||||
elif [ -f "$STATE" ]; then
|
||||
# shellcheck disable=SC1090
|
||||
set -a; . "$STATE"; set +a
|
||||
fi
|
||||
export HONCHO_SANDBOX_IMAGE
|
||||
|
||||
compose() {
|
||||
docker compose \
|
||||
-f "$HERE/compose.yml" \
|
||||
-f "$HERE/compose.$PROVIDER.yml" \
|
||||
--project-directory "$HERE" \
|
||||
"$@"
|
||||
}
|
||||
|
||||
psql_db() {
|
||||
# Maintenance connections go to `postgres`, never to $DB — you cannot drop a
|
||||
# database you are connected to.
|
||||
compose exec -T database psql -v ON_ERROR_STOP=1 -U postgres -d postgres "$@"
|
||||
}
|
||||
|
||||
db_exists() {
|
||||
[ "$(psql_db -tAc "SELECT 1 FROM pg_database WHERE datname='$1'" 2>/dev/null)" = "1" ]
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Preconditions
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
preflight() {
|
||||
docker info >/dev/null 2>&1 || die "the Docker daemon is not running"
|
||||
|
||||
if [ "$PROVIDER" = real ]; then
|
||||
# Fail before starting anything. A real-mode stack with no key boots fine and
|
||||
# then derives nothing, which reads as "the deriver found nothing".
|
||||
[ -f "$HERE/real.env" ] || die \
|
||||
"real mode needs $HERE/real.env. Copy real.env.example and add a provider key."
|
||||
grep -Eq '^\s*LLM_[A-Z]+_API_KEY=.+' "$HERE/real.env" || die \
|
||||
"no LLM_*_API_KEY with a value in real.env."
|
||||
if grep -Eq '^\s*LLM_[A-Z]+_API_KEY=(sk-replace-me|your-api-key-here)\s*$' "$HERE/real.env"; then
|
||||
die "real.env still has the placeholder key from real.env.example."
|
||||
fi
|
||||
fi
|
||||
# Must not end on a failed test. Under `set -e`, a non-zero return from the last
|
||||
# command in this function aborts the script with no output at all — which is
|
||||
# exactly how the first real-mode `up` "failed": silently, before doing anything.
|
||||
return 0
|
||||
}
|
||||
|
||||
# The pinned digest predates PR #1094, so it carries no src/mock_provider. Say so
|
||||
# precisely instead of letting the service crash-loop on a missing module.
|
||||
check_mock_provider_present() {
|
||||
[ "$PROVIDER" = mock ] || return 0
|
||||
if ! docker run --rm --entrypoint test "$HONCHO_SANDBOX_IMAGE" -d /app/src/mock_provider; then
|
||||
die "$(cat <<MSG
|
||||
the image $HONCHO_SANDBOX_IMAGE does not contain src/mock_provider.
|
||||
|
||||
PR #1094 adds it and is not merged yet, so no published image has it. Build from
|
||||
the working tree instead:
|
||||
|
||||
sandbox/sandbox.sh up --build
|
||||
|
||||
Once #1094 is merged and published, bump the digest in sandbox/image.env.
|
||||
MSG
|
||||
)"
|
||||
fi
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Commands
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
cmd_up() {
|
||||
preflight
|
||||
|
||||
compose pull --quiet database redis
|
||||
if [ "$BUILD" = 1 ]; then
|
||||
say "building $BUILT_IMAGE from the working tree"
|
||||
docker build -t "$BUILT_IMAGE" "$REPO"
|
||||
elif docker image inspect "$HONCHO_SANDBOX_IMAGE" >/dev/null 2>&1; then
|
||||
# Already here — either pulled on an earlier run, or built locally by --build
|
||||
# and recorded in .state.env, in which case pulling it would fail outright
|
||||
# because the tag exists in no registry.
|
||||
say "using local image $HONCHO_SANDBOX_IMAGE"
|
||||
else
|
||||
say "pulling $HONCHO_SANDBOX_IMAGE"
|
||||
docker pull --quiet "$HONCHO_SANDBOX_IMAGE" >/dev/null
|
||||
fi
|
||||
|
||||
check_mock_provider_present
|
||||
|
||||
say "starting the $PROVIDER stack"
|
||||
# --remove-orphans because both modes share one Compose project: switching from
|
||||
# mock to real leaves mock-provider running and otherwise unreferenced, where it
|
||||
# would sit idle and make `status` misleading about what the stack is using.
|
||||
compose up -d --wait --remove-orphans
|
||||
echo "HONCHO_SANDBOX_IMAGE=$HONCHO_SANDBOX_IMAGE" > "$STATE"
|
||||
|
||||
if db_exists "$TEMPLATE"; then
|
||||
say "template $TEMPLATE already present — resetting to it"
|
||||
cmd_reset
|
||||
else
|
||||
cmd_seed
|
||||
fi
|
||||
|
||||
echo
|
||||
say "sandbox ready at http://127.0.0.1:${SANDBOX_API_PORT:-18000} (provider: $PROVIDER)"
|
||||
}
|
||||
|
||||
cmd_seed() {
|
||||
preflight
|
||||
# Seeding an already-seeded database appends to it — the fixture would land a
|
||||
# second time and the "expected 6 messages, found 12" check in seed.py would
|
||||
# (correctly) fail. Start from an empty, freshly migrated database every time so
|
||||
# `seed` means the same thing regardless of what was there before.
|
||||
say "clearing the database before seeding"
|
||||
compose stop api deriver >/dev/null
|
||||
psql_db -c "DROP DATABASE IF EXISTS $DB WITH (FORCE)" >/dev/null
|
||||
psql_db -c "CREATE DATABASE $DB" >/dev/null
|
||||
compose exec -T database psql -q -v ON_ERROR_STOP=1 -U postgres -d "$DB" \
|
||||
-c "CREATE EXTENSION IF NOT EXISTS vector" >/dev/null
|
||||
# Redis still holds entries keyed to the database that was just dropped; without
|
||||
# this the api serves cached peers that no longer exist and seeding conflicts.
|
||||
compose exec -T redis redis-cli FLUSHALL >/dev/null
|
||||
# The api entrypoint runs scripts/provision_db.py, so starting it is what applies
|
||||
# the Alembic migrations to the new database.
|
||||
compose start api deriver >/dev/null
|
||||
compose up -d --wait --no-recreate >/dev/null
|
||||
|
||||
say "seeding ($PROVIDER)"
|
||||
SANDBOX_BASE_URL="http://127.0.0.1:${SANDBOX_API_PORT:-18000}" \
|
||||
uv run --no-project --with-editable "$REPO/sdks/python" \
|
||||
python "$HERE/seed.py"
|
||||
|
||||
say "snapshotting to $TEMPLATE"
|
||||
stamp_fingerprint
|
||||
# api and deriver hold pooled connections; CREATE DATABASE ... TEMPLATE needs
|
||||
# the source to have none.
|
||||
compose stop api deriver >/dev/null
|
||||
disconnect_db "$DB"
|
||||
psql_db -c "DROP DATABASE IF EXISTS $TEMPLATE" >/dev/null
|
||||
psql_db -c "CREATE DATABASE $TEMPLATE TEMPLATE $DB" >/dev/null
|
||||
compose start api deriver >/dev/null
|
||||
compose up -d --wait --no-recreate >/dev/null
|
||||
say "snapshot ready — reset restores this state without re-deriving"
|
||||
}
|
||||
|
||||
cmd_reset() {
|
||||
preflight
|
||||
db_exists "$TEMPLATE" || die \
|
||||
"no template for $PROVIDER mode. Run: sandbox/sandbox.sh seed --provider $PROVIDER"
|
||||
|
||||
check_fingerprint
|
||||
|
||||
# Nothing is stopped or restarted. DROP ... WITH (FORCE) evicts the api and
|
||||
# deriver connection pools itself, and both reconnect on their next use
|
||||
# (db.POOL_PRE_PING is on) — the api on its next request, the deriver on its
|
||||
# next poll a quarter-second later.
|
||||
#
|
||||
# Both log one OperationalError as their in-flight connection dies. That noise
|
||||
# is the cost of not paying ~9s of container stop/start on every reset, which
|
||||
# would defeat the point. Restarting them instead is the fallback if this ever
|
||||
# stops being true.
|
||||
#
|
||||
# CREATE DATABASE ... TEMPLATE additionally needs the *source* to be
|
||||
# connectionless, and check_fingerprint just read from it.
|
||||
disconnect_db "$TEMPLATE"
|
||||
psql_db -c "DROP DATABASE $DB WITH (FORCE)" >/dev/null
|
||||
psql_db -c "CREATE DATABASE $DB TEMPLATE $TEMPLATE" >/dev/null
|
||||
compose exec -T redis redis-cli FLUSHALL >/dev/null
|
||||
say "reset to $TEMPLATE"
|
||||
}
|
||||
|
||||
cmd_status() {
|
||||
compose ps
|
||||
echo
|
||||
echo "provider: $PROVIDER"
|
||||
echo "image: $HONCHO_SANDBOX_IMAGE"
|
||||
echo "api: http://127.0.0.1:${SANDBOX_API_PORT:-18000}"
|
||||
for mode in mock real; do
|
||||
if db_exists "${DB}_seeded_${mode}"; then
|
||||
echo "template: ${DB}_seeded_${mode} present"
|
||||
else
|
||||
echo "template: ${DB}_seeded_${mode} absent"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
cmd_down() {
|
||||
say "tearing down the $PROVIDER stack and its volumes"
|
||||
compose down -v
|
||||
rm -f "$STATE"
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Snapshot staleness
|
||||
#
|
||||
# A template that predates a migration, or was built against a different
|
||||
# provider, restores silently and wrongly. Both are recorded inside the template
|
||||
# itself so reset can refuse rather than mislead.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
fingerprint() {
|
||||
local alembic
|
||||
alembic="$(compose exec -T database psql -tAX -U postgres -d "$DB" \
|
||||
-c "SELECT version_num FROM alembic_version" 2>/dev/null | tr -d '[:space:]')"
|
||||
local files
|
||||
files="$(cat "$HERE/fixture.json" "$HERE/seed.py" | shasum -a 256 | cut -c1-16)"
|
||||
echo "alembic=$alembic fixture=$files provider=$PROVIDER"
|
||||
}
|
||||
|
||||
stamp_fingerprint() {
|
||||
compose exec -T database psql -v ON_ERROR_STOP=1 -U postgres -d "$DB" >/dev/null <<SQL
|
||||
CREATE TABLE IF NOT EXISTS sandbox_fingerprint (value text PRIMARY KEY);
|
||||
TRUNCATE sandbox_fingerprint;
|
||||
INSERT INTO sandbox_fingerprint VALUES ('$(fingerprint)');
|
||||
SQL
|
||||
}
|
||||
|
||||
check_fingerprint() {
|
||||
local want have
|
||||
want="$(fingerprint)"
|
||||
have="$(compose exec -T database psql -tAX -U postgres -d "$TEMPLATE" \
|
||||
-c "SELECT value FROM sandbox_fingerprint" 2>/dev/null | tr -d '\r')"
|
||||
|
||||
# The live database carries whatever revision the api container migrated it to
|
||||
# on boot; the template carries the revision it was seeded at. A difference is
|
||||
# exactly the migration drift this guard exists to catch, so it is compared,
|
||||
# not normalised away.
|
||||
[ "$want" = "$have" ] || die "$(cat <<MSG
|
||||
$TEMPLATE is stale and would restore the wrong state.
|
||||
|
||||
template: ${have:-<no fingerprint>}
|
||||
current: $want
|
||||
|
||||
Reseed: sandbox/sandbox.sh seed --provider $PROVIDER
|
||||
MSG
|
||||
)"
|
||||
}
|
||||
|
||||
disconnect_db() {
|
||||
# Defaults to the live database. CREATE DATABASE ... TEMPLATE also requires the
|
||||
# *source* to have no connections, and reading the fingerprint just opened one.
|
||||
local target="${1:-$DB}"
|
||||
psql_db -c \
|
||||
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='$target' AND pid <> pg_backend_pid()" \
|
||||
>/dev/null
|
||||
}
|
||||
|
||||
case "$COMMAND" in
|
||||
up) cmd_up ;;
|
||||
seed) cmd_seed ;;
|
||||
reset) cmd_reset ;;
|
||||
status) cmd_status ;;
|
||||
down) cmd_down ;;
|
||||
*) die "unknown command '$COMMAND' (expected up, seed, reset, status, or down)" ;;
|
||||
esac
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
"""Populate the sandbox with a known fixture and verify it actually landed.
|
||||
|
||||
Run through sandbox.sh, which supplies the base URL and owns the Docker and
|
||||
Postgres side. This script only talks to the API:
|
||||
|
||||
uv run --no-project --with-editable ./sdks/python python sandbox/seed.py
|
||||
|
||||
It is provider-agnostic. Under the mock provider the *text* of a conclusion is
|
||||
synthetic and unrelated to the messages, so nothing here asserts on conclusion
|
||||
content - only that derivation ran and produced something. What it does assert
|
||||
strictly is the round trip that harness integrations actually get wrong: which
|
||||
messages landed, on which peers, with which observation topology.
|
||||
|
||||
Exits non-zero on any failure. An empty sandbox that reports success is the exact
|
||||
failure mode this whole thing exists to prevent.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from honcho import Honcho
|
||||
from honcho.api_types import MessageCreateParams, SessionPeerConfig
|
||||
|
||||
FIXTURE = Path(__file__).with_name("fixture.json")
|
||||
|
||||
# Generous: a cold deriver on a real provider is slow, and the failure we care
|
||||
# about (nothing is processing at all) shows up as a timeout either way.
|
||||
DRAIN_TIMEOUT_SECONDS = float(os.environ.get("SANDBOX_DRAIN_TIMEOUT", "300"))
|
||||
DRAIN_POLL_SECONDS = 0.5
|
||||
|
||||
|
||||
class SeedError(RuntimeError):
|
||||
"""Seeding did not reach a usable state."""
|
||||
|
||||
|
||||
def log(message: str) -> None:
|
||||
print(f"[seed] {message}", flush=True)
|
||||
|
||||
|
||||
def drain(honcho: Honcho, what: str) -> None:
|
||||
"""Block until the deriver queue is empty, or fail loudly.
|
||||
|
||||
Silence is the dangerous outcome here. A queue that never drains and a queue
|
||||
that never had anything queued look identical from the outside, so the timeout
|
||||
reports the last status it saw rather than just giving up.
|
||||
"""
|
||||
deadline = time.monotonic() + DRAIN_TIMEOUT_SECONDS
|
||||
last: Any = None
|
||||
while time.monotonic() < deadline:
|
||||
last = honcho.queue_status()
|
||||
outstanding = last.pending_work_units + last.in_progress_work_units
|
||||
if outstanding == 0 and last.completed_work_units > 0:
|
||||
log(f"drained after {what} ({last.completed_work_units} work units)")
|
||||
return
|
||||
time.sleep(DRAIN_POLL_SECONDS)
|
||||
|
||||
raise SeedError(
|
||||
f"queue did not drain after {what} within {DRAIN_TIMEOUT_SECONDS:.0f}s. "
|
||||
f"Last status: {last}. "
|
||||
"If completed_work_units is 0, nothing was ever enqueued - check that the "
|
||||
"deriver is running and that the provider base URL resolves."
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
fixture = json.loads(FIXTURE.read_text())
|
||||
base_url = os.environ.get("SANDBOX_BASE_URL", "http://127.0.0.1:18000")
|
||||
workspace_id = fixture["workspace"]
|
||||
|
||||
log(f"seeding workspace {workspace_id!r} at {base_url}")
|
||||
honcho = Honcho(base_url=base_url, workspace_id=workspace_id, api_key="sandbox")
|
||||
|
||||
peers = {spec["id"]: honcho.peer(spec["id"]) for spec in fixture["peers"]}
|
||||
session = honcho.session(fixture["session"])
|
||||
|
||||
session.add_peers(
|
||||
[
|
||||
(
|
||||
peers[spec["id"]],
|
||||
SessionPeerConfig(
|
||||
observe_me=spec["observe_me"],
|
||||
observe_others=spec["observe_others"],
|
||||
),
|
||||
)
|
||||
for spec in fixture["peers"]
|
||||
]
|
||||
)
|
||||
log(f"created {len(peers)} peers with explicit observation topology")
|
||||
|
||||
session.add_messages(
|
||||
[
|
||||
MessageCreateParams(peer_id=msg["peer"], content=msg["content"])
|
||||
for msg in fixture["messages"]
|
||||
]
|
||||
)
|
||||
log(f"posted {len(fixture['messages'])} messages")
|
||||
drain(honcho, "messages")
|
||||
|
||||
for dream in fixture.get("dreams", []):
|
||||
honcho.schedule_dream(
|
||||
observer=dream["observer"],
|
||||
observed=dream["observed"],
|
||||
session=session,
|
||||
)
|
||||
log(f"scheduled dream: {dream['observer']} -> {dream['observed']}")
|
||||
if fixture.get("dreams"):
|
||||
drain(honcho, "dream")
|
||||
|
||||
verify(honcho, session, peers, fixture)
|
||||
log("seed complete")
|
||||
return 0
|
||||
|
||||
|
||||
def verify(
|
||||
honcho: Honcho,
|
||||
session: Any,
|
||||
peers: dict[str, Any],
|
||||
fixture: dict[str, Any],
|
||||
) -> None:
|
||||
"""Assert the fixture round-tripped. Raises SeedError on any mismatch."""
|
||||
stored = list(session.messages(size=100))
|
||||
if len(stored) != len(fixture["messages"]):
|
||||
raise SeedError(
|
||||
f"expected {len(fixture['messages'])} messages, found {len(stored)}"
|
||||
)
|
||||
|
||||
expected_authors = sorted(msg["peer"] for msg in fixture["messages"])
|
||||
actual_authors = sorted(msg.peer_id for msg in stored)
|
||||
if actual_authors != expected_authors:
|
||||
raise SeedError(
|
||||
f"messages landed on the wrong peers: expected {expected_authors}, "
|
||||
f"got {actual_authors}"
|
||||
)
|
||||
log(f"verified {len(stored)} messages on the expected peers")
|
||||
|
||||
# Topology is the thing that silently breaks, so it is checked against the
|
||||
# server's view rather than assumed from the create call.
|
||||
for spec in fixture["peers"]:
|
||||
config = session.get_peer_configuration(spec["id"])
|
||||
if (config.observe_me, config.observe_others) != (
|
||||
spec["observe_me"],
|
||||
spec["observe_others"],
|
||||
):
|
||||
raise SeedError(
|
||||
f"peer {spec['id']!r} topology mismatch: expected "
|
||||
f"observe_me={spec['observe_me']} "
|
||||
f"observe_others={spec['observe_others']}, got {config}"
|
||||
)
|
||||
log("verified observation topology")
|
||||
|
||||
# Presence only. Under the mock provider, conclusion text is synthetic and
|
||||
# says nothing about the messages, and the level mix is explicit-only because
|
||||
# the Dreamer specialists write via tool calls the mock never emits. Asserting
|
||||
# on either would pass in real mode and fail in mock mode.
|
||||
total = 0
|
||||
for observer_id, observer in peers.items():
|
||||
for observed_id in peers:
|
||||
found = len(list(observer.conclusions_of(observed_id).list(size=100)))
|
||||
if found:
|
||||
log(f" {observer_id} -> {observed_id}: {found} conclusions")
|
||||
total += found
|
||||
|
||||
if total == 0:
|
||||
raise SeedError(
|
||||
"no conclusions were derived. The queue drained, so the deriver ran and "
|
||||
"produced nothing - check the provider wiring "
|
||||
"(LLM_OPENAI_BASE_URL / EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL) "
|
||||
"and the deriver logs."
|
||||
)
|
||||
log(f"verified {total} conclusions present")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except SeedError as exc:
|
||||
print(f"[seed] FAILED: {exc}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
Loading…
Reference in New Issue