diff --git a/sandbox/README.md b/sandbox/README.md index 608c45a8..ca2cd8de 100644 --- a/sandbox/README.md +++ b/sandbox/README.md @@ -3,8 +3,9 @@ A local Honcho you can wipe and rebuild in seconds, so harness testing stops depending on whatever state your laptop happens to be in. -It comes up **already seeded** — a known workspace, peers, session, messages, and conclusions the -deriver has actually produced — and `reset` returns it to that exact state without re-deriving. +It comes up **already seeded** — a known workspace, peers, session, messages, conclusions the +deriver has actually produced, and committed conclusions at every reasoning level — and `reset` +returns it to that exact state without re-deriving. ```bash sandbox/sandbox.sh up # start + seed (first run also builds/pulls) @@ -29,14 +30,19 @@ Both ways of resolving that ship here; `mock` is the default. | **Seed time** | ~15s | ~6 min | | **Reset time** | 0.86s | 1.29s | | **Vector recall** | **Untestable** — see below | Testable | -| **What you get** | 4 conclusions, all `explicit`, synthetic text | 22 conclusions — 16 `explicit`, 4 `deductive`, 2 `inductive` | +| **Derived** | 4 conclusions, all `explicit`, synthetic text | 22 conclusions — 16 `explicit`, 4 `deductive`, 2 `inductive` | +| **Seeded** | 7 — 4 `explicit`, 2 `deductive`, 1 `inductive`, committed text | the same 7, identical bytes | | **Use it for** | CI, harness smoke tests, anything that needs a repeatable answer | Recall quality, validating against real model output | -Those counts are from the committed fixture, measured on both paths. They are the sharpest +The derived counts are from the committed fixture, measured on both paths. They are the sharpest illustration of the difference: mock gives you four copies of `[mock] mock dummy placeholder …`, while real mode reasons its way from "my cat Marzipan is named after the pylon's colour" to the deductive conclusion *"alice's cat is named after a feature of alice's pedestrian bridge project."* -Anything asserting on that kind of inference needs real mode. +Anything asserting on a model's *own* inference needs real mode. + +The seeded row is why you may not need to. Seeded conclusions are written straight into the +collection rather than derived, so their text and level are identical in both modes — see +[Seeded conclusions](#seeded-conclusions). ```bash sandbox/sandbox.sh up # mock @@ -54,10 +60,11 @@ exactly as far apart as two unrelated strings. Any recall assertion built on moc lexical or full-text search. A vector-ranking assertion will fail there for reasons that have nothing to do with the code you are testing — that is what `real` mode is for. -**Mock conclusions are synthetic.** The text is derived from the request, not from the meaning of -your messages, and the level is always `explicit` (the Dreamer's specialists write via tool calls, -which the mock deliberately never emits). Assert that conclusions *exist*; do not assert on what -they say or on the level mix, or your test will pass in one mode and fail in the other. +**Mock *derived* conclusions are synthetic.** The text comes from the request, not from the meaning +of your messages, and the level is always `explicit` (the Dreamer's specialists write via tool +calls, which the mock deliberately never emits). Assert that derived conclusions *exist*; do not +assert on what they say or on the level mix, or your test will pass in one mode and fail in the +other. Assert against the seeded conclusions instead — that is what they are for. ### Real mode @@ -109,6 +116,77 @@ The cost of a snapshot is that it can go stale. Three guards: If you hit the staleness refusal: `sandbox/sandbox.sh seed`. +## Seeded conclusions + +`fixture.json` can carry pre-made conclusions per peer, at any of the three reasoning levels. +All three keys are optional; with none of them the sandbox behaves exactly as it would without +this feature. + +```json +{ + "id": "alice", + "observe_me": true, + "observe_others": false, + + "explicit": [ + "alice is a structural engineer based in Rotterdam", + "alice has a cat called Marzipan, named after the colour of the bridge's pylon" + ], + "deductive": [ + { "content": "alice named her cat after a feature of the bridge she is designing", + "premises": [0, 1] } + ], + "inductive": ["alice optimises for low lifetime maintenance cost"] +} +``` + +**The peer carrying the keys is the one being observed.** Conclusions are keyed by an +`(observer, observed)` pair, and the observer is inferred: every fixture peer with +`observe_others` set, excluding the observed itself. For the standard harness shape that is +exactly `assistant -> alice` — the same pair the dream uses and the one a harness reads from. A +peer nobody else observes is an **error**, not a silent no-op; add an explicit `"observer"` to +that peer to override the inference, which is also how you reach self-representation +(`alice -> alice`). + +**Items are a bare string, or `{content, premises}`.** Each premise is an index into the same +peer's `explicit` list. Premises are optional, but a derived conclusion without them is not +something the Dreamer would ever write, so the committed fixture always supplies them. + +**Contents must be unique.** Honcho collapses a conclusion whose content matches something +already stored — case- and whitespace-insensitively — and that cannot be switched off. The +seeder asserts exact counts, so a near-duplicate fails the seed rather than quietly vanishing. + +### What premises buy you, and what they don't + +The premise indices become real `Document.source_ids` pointing at the actual rows from the +`explicit` pass, so the reasoning tree genuinely traverses in both directions. Two caveats on +where that is visible: + +- **Premise text renders in the representation.** `peer.representation(target=…)` prints each + premise indented under its conclusion, which is the practical payoff and what `seed.py verify` + asserts. +- **Premise *links* are not on the API.** The conclusion response carries `level` but not + `source_ids`, and `get_reasoning_chain` is a Dialectic tool rather than a route. So a test can + only reach the links through the Dialectic, or through SQL. Because of that, a broken + reasoning tree is invisible from outside — so the seeder checks its own links before the + snapshot is taken, resolving every premise id and confirming each is reachable from its + children. A dangling or mis-filed premise fails the seed. + +### Why this is not done over the API + +The public create-conclusions endpoint always writes `level="explicit"` with no premises, and +neither the schema nor the SDKs have a field for either. The columns exist; only an in-process +caller can set them. So `sandbox.sh` runs `inject_conclusions.py` **inside the api container**, +which already is Honcho's venv with the api's settings and a live embedding client — the script +goes in on stdin and the fixture through the environment, so nothing is mounted and nothing is +left behind. + +The cost is that the injector calls Honcho *internals*, which carry no stability contract, out +of the image pinned in `image.env`. It checks the signatures it depends on before writing and +fails with a "bump `image.env` and update this script together" message, rather than half-seeding +a database. Widening the public API to accept a level was the alternative and was deliberately +rejected: it would let any client assert a conclusion is `deductive` with premises it invented. + ## What's in the fixture `fixture.json` — two peers with **explicitly stated** observation topology (`alice` is observed and @@ -137,6 +215,7 @@ dreams is 8 hours — so the seed schedules one directly. | `init.sql` | Creates the `honcho_sandbox` database. | | `fixture.json` | The seeded conversation. | | `seed.py` | Populates and verifies. Talks only to the API. | +| `inject_conclusions.py` | Writes the seeded conclusions. Runs inside the api container. | | `real.env.example` | Template for real-mode credentials. | Exactly one provider overlay is always composed on top of the base, so the choice is visible in the diff --git a/sandbox/fixture.json b/sandbox/fixture.json index 39894862..bcd7dddb 100644 --- a/sandbox/fixture.json +++ b/sandbox/fixture.json @@ -3,8 +3,37 @@ "session": "sandbox-session", "_comment": "Peer topology is stated explicitly because it is the thing that silently breaks (DEV-2462 asserts on it). alice is observed and does not observe; the assistant observes and is not observed - the standard harness shape.", + + "_conclusions_comment": "The explicit/deductive/inductive keys are optional seeded memory, written straight into the (observer, observed) collection rather than derived. They exist because derived conclusions are only assertable in real mode: the mock provider's conclusion text is synthetic and its level is always explicit. Seeded text is identical in both modes, so a harness test can assert on content and level for free. Premise indices point into the same peer's explicit list, and they are what make the conclusion's reasoning chain resolve - excadrill's equivalent fixtures use synthetic source ids that never resolve to a real document. A premise-less derived conclusion is allowed by the format but is not something the Dreamer would ever write, so the committed fixture does not use one.", "peers": [ - { "id": "alice", "observe_me": true, "observe_others": false }, + { + "id": "alice", + "observe_me": true, + "observe_others": false, + + "explicit": [ + "alice is a structural engineer based in Rotterdam", + "alice is designing a pedestrian bridge over the Nieuwe Maas spanning 240 metres on a single cable-stayed pylon", + "alice specified weathering steel for the bridge because its maintenance budget is close to zero", + "alice has a cat called Marzipan, named after the colour of the bridge's pylon" + ], + "deductive": [ + { + "content": "alice named her cat after a feature of the bridge she is designing", + "premises": [1, 3] + }, + { + "content": "alice selects materials on lifetime maintenance cost rather than capital cost", + "premises": [2] + } + ], + "inductive": [ + { + "content": "alice lets the Nieuwe Maas bridge project shape decisions well outside her working hours", + "premises": [1, 3] + } + ] + }, { "id": "assistant", "observe_me": false, "observe_others": true } ], diff --git a/sandbox/inject_conclusions.py b/sandbox/inject_conclusions.py new file mode 100644 index 00000000..b75d07f8 --- /dev/null +++ b/sandbox/inject_conclusions.py @@ -0,0 +1,502 @@ +"""Seed conclusions at every reasoning level, from inside the api container. + +Run by sandbox.sh, never from the host: + + compose exec -T -e SANDBOX_FIXTURE_JSON="$(cat fixture.json)" api \ + /app/.venv/bin/python - < sandbox/inject_conclusions.py + +Level and premise links are not reachable from the public API. `crud.create_observations` +hardcodes `level="explicit"`, `source_ids=NULL`, `internal_metadata={}`, and `ConclusionCreate` +has no field for any of them. The columns exist on Document; only an in-process caller can set +them. So this script imports Honcho's own write helpers, which is why it runs in the container +rather than beside seed.py: the container already *is* Honcho's venv, with the api's settings +and a correctly wired embedding client. + +That comes at a price worth stating. These are internal APIs with no stability contract, and +they are imported out of the *pinned* image, not the working tree — so a digest bump can move +them underneath us. `check_signatures` fails loudly on that rather than letting a broken seed +look like a working one. + +Two passes, not three: premise indices point into the same peer's `explicit` list, so both +derived levels reference pass-1 ids and nothing has to reference a derived id. + +Everything here is asserted exactly, because every silent-failure mode in this path reduces a +count without raising: exact-content dedup is always on and cannot be switched off, semantic +dedup replaces rows, per-item embedding failures drop rows, and the session-purity invariant +skips explicit rows with no session. A seed that quietly wrote nothing is the failure this +whole sandbox exists to prevent. +""" + +from __future__ import annotations + +import asyncio +import inspect +import json +import os +import sys +from typing import Any + +LEVELS = ("explicit", "deductive", "inductive") + +# Levels whose premise text renders under a different metadata key. The working representation +# prints DocumentMetadata.premises for deductive and .sources for inductive, and each is read +# only for its own level, so putting the text under the wrong one renders nothing. +PREMISE_FIELD = {"deductive": "premises", "inductive": "sources"} + + +class InjectError(RuntimeError): + """Seeding conclusions did not reach a usable state.""" + + +def log(message: str) -> None: + print(f"[conclusions] {message}", flush=True) + + +def check_signatures(crud: Any, schemas: Any) -> None: + """Fail loudly if the image's internals moved. + + Cheap insurance against the real hazard of importing unversioned internals out of a pinned + image: without this, a renamed keyword surfaces as a TypeError mid-seed, or worse, as a + seed that silently wrote fewer rows. + """ + expected = { + "create_documents": ( + "documents", + "workspace_name", + "observer", + "observed", + "deduplicate", + ), + "create_observations": ("observations", "workspace_name"), + "get_or_create_collection": ("workspace_name", "observer", "observed"), + "get_documents_by_ids": ("workspace_name", "document_ids"), + "get_child_observations": ( + "workspace_name", + "parent_id", + "observer", + "observed", + ), + } + for name, params in expected.items(): + signature = inspect.signature(getattr(crud, name)) + missing = [p for p in params if p not in signature.parameters] + if missing: + raise InjectError( + f"crud.{name} is missing expected parameters {missing} in this image. " + "The sandbox seeds conclusions through Honcho internals, which carry no " + "stability contract; bump sandbox/image.env and update this script together." + ) + + for model, fields in ( + ( + schemas.DocumentCreate, + ("content", "level", "metadata", "embedding", "source_ids"), + ), + ( + schemas.DocumentMetadata, + ("message_ids", "message_created_at", "premises", "sources"), + ), + ): + missing = [f for f in fields if f not in model.model_fields] + if missing: + raise InjectError( + f"{model.__name__} is missing expected fields {missing} in this image. " + "Bump sandbox/image.env and update this script together." + ) + + +def normalize( + items: list[Any], level: str, peer_id: str, explicit_count: int +) -> list[dict[str, Any]]: + """Accept either a bare string or {content, premises}, and validate premise indices. + + An out-of-range index is rejected here rather than written as a dangling source id. Nothing + downstream validates source_ids -- a bad one resolves to "referenced N premise IDs but none + found in database" at read time, which is precisely the quiet wrongness to avoid. + """ + normalized: list[dict[str, Any]] = [] + for position, item in enumerate(items): + if isinstance(item, str): + content, premises = item, [] + elif isinstance(item, dict): + content = item.get("content") + premises = item.get("premises", []) + unknown = set(item) - {"content", "premises"} + if unknown: + raise InjectError( + f"{peer_id}.{level}[{position}] has unknown keys {sorted(unknown)}" + ) + else: + raise InjectError( + f"{peer_id}.{level}[{position}] must be a string or an object, got {type(item).__name__}" + ) + + if not content or not content.strip(): + raise InjectError(f"{peer_id}.{level}[{position}] has empty content") + if premises and level == "explicit": + raise InjectError( + f"{peer_id}.explicit[{position}] declares premises. Explicit conclusions come " + "straight from messages and are the premises other levels point at." + ) + for index in premises: + if not isinstance(index, int) or not 0 <= index < explicit_count: + raise InjectError( + f"{peer_id}.{level}[{position}] premise {index!r} is not a valid index into " + f"{peer_id}.explicit (which has {explicit_count} entries)" + ) + + normalized.append({"content": content, "premises": premises}) + return normalized + + +def resolve_observers(spec: dict[str, Any], peers: list[dict[str, Any]]) -> list[str]: + """Who holds conclusions about this peer. + + The peer carrying the keys is the observed. Absent an explicit override, the observers are + every peer configured to observe others -- which for the standard harness shape is exactly + the assistant. Resolving to nobody is an error: the conclusions would be written into no + collection at all and the seed would report success having stored nothing. + """ + observed = spec["id"] + override = spec.get("observer") + if override: + return [override] + + observers = [ + peer["id"] + for peer in peers + if peer.get("observe_others") and peer["id"] != observed + ] + if not observers: + raise InjectError( + f"peer {observed!r} has seeded conclusions but no other peer observes it: " + "no fixture peer besides itself sets observe_others, and it declares no " + f'explicit "observer". The conclusions would be written nowhere. Either give ' + f'{observed!r} an "observer", or set observe_others on the peer that should ' + "hold them." + ) + return observers + + +async def latest_message_timestamp( + db: Any, models: Any, workspace: str, session: str +) -> str | None: + """The conversation's own clock, for the representation to render. + + Derived conclusions are stamped with the last ingested message time rather than the seed's + wall-clock, so the representation shows when the conversation happened. Honcho's own dream + path back-dates the same way. + """ + from sqlalchemy import func, select + + result = await db.execute( + select(func.max(models.Message.created_at)).where( + models.Message.workspace_name == workspace, + models.Message.session_name == session, + ) + ) + newest = result.scalar_one_or_none() + return newest.strftime("%Y-%m-%dT%H:%M:%SZ") if newest else None + + +def assert_clean(result: Any, requested: int, label: str) -> None: + """No row may be dropped, deduped, or replaced. + + create_documents reports these as counters rather than raising, so without this the seed + reports success while having written fewer conclusions than the fixture declares. Exact + content dedup cannot be disabled, so this fires on a reseed over existing content too -- + which is correct: seed.py always starts from an empty database. + """ + created = len(result.created_documents) + counters = { + "exact duplicates within the batch": result.exact_dup_in_batch_count, + "exact duplicates already stored": result.exact_dup_existing_count, + "semantically rejected": result.semantic_dup_rejected_count, + "semantically replaced": result.semantic_dup_replaced_count, + } + dropped = {name: count for name, count in counters.items() if count} + if created != requested or dropped: + raise InjectError( + f"{label}: asked for {requested} conclusions, stored {created}" + + ( + f" ({', '.join(f'{n}: {c}' for n, c in dropped.items())})" + if dropped + else "" + ) + + ". Conclusion contents must be unique; near-identical text is collapsed by " + "Honcho's dedup before it reaches the database." + ) + + +async def inject_pair( + modules: dict[str, Any], + workspace: str, + session: str, + observer: str, + observed: str, + conclusions: dict[str, list[dict[str, Any]]], +) -> dict[str, int]: + """Seed one (observer, observed) collection. Returns per-level counts actually stored.""" + crud = modules["crud"] + schemas = modules["schemas"] + models = modules["models"] + embedding_client = modules["embedding_client"] + tracked_db = modules["tracked_db"] + + stored: dict[str, int] = {} + + # Pass 1. The public write path is the only one that hands back rows, so it is how the + # premise ids are obtained -- and it does no dedup, so nothing is silently collapsed. + explicit = conclusions["explicit"] + premise_ids: list[str] = [] + async with tracked_db("sandbox.seed_explicit") as db: + await crud.get_or_create_collection( + db, workspace, observer=observer, observed=observed + ) + if explicit: + documents = await crud.create_observations( + db, + [ + schemas.ConclusionCreate( + content=item["content"], + observer_id=observer, + observed_id=observed, + # Explicit rows must carry a session: create_documents refuses + # session-less explicit rows on a session-purity invariant, and an + # explicit conclusion genuinely does come from a conversation. + session_id=session, + ) + for item in explicit + ], + workspace, + ) + if len(documents) != len(explicit): + raise InjectError( + f"{observer} -> {observed}: asked for {len(explicit)} explicit " + f"conclusions, stored {len(documents)}" + ) + premise_ids = [document.id for document in documents] + stored["explicit"] = len(documents) + + # Pass 2. Both derived levels cite pass-1 ids, so neither needs the other's ids back. + derived = { + level: conclusions[level] + for level in ("deductive", "inductive") + if conclusions[level] + } + if derived: + async with tracked_db("sandbox.seed_derived") as db: + message_created_at = await latest_message_timestamp( + db, models, workspace, session + ) + if message_created_at is None: + raise InjectError( + f"session {session!r} has no messages, so derived conclusions have no " + "conversation timestamp to carry. Seed messages before conclusions." + ) + + for level, items in derived.items(): + contents = [item["content"] for item in items] + embeddings = await embedding_client.simple_batch_embed( + contents, on_oversize="truncate" + ) + if len(embeddings) != len(contents): + raise InjectError( + f"{observer} -> {observed}: embedded {len(embeddings)} of " + f"{len(contents)} {level} conclusions" + ) + + payload: list[Any] = [] + for item, embedding in zip(items, embeddings, strict=True): + source_ids = [premise_ids[index] for index in item["premises"]] + metadata: dict[str, Any] = { + # Not derived from messages, but the representation reads this to + # render the conclusion's timestamp. + "message_ids": [], + "message_created_at": message_created_at, + "source_ids": source_ids, + PREMISE_FIELD[level]: [ + explicit[index]["content"] for index in item["premises"] + ], + } + if level == "inductive": + metadata["pattern_type"] = "tendency" + metadata["confidence"] = ( + "high" if len(source_ids) > 1 else "low" + ) + payload.append( + schemas.DocumentCreate( + content=item["content"], + # Derived conclusions belong to the dream, not to one session, + # which is how the Dreamer writes them. + session_name=None, + level=level, + times_derived=1, + metadata=schemas.DocumentMetadata(**metadata), + embedding=embedding, + source_ids=source_ids, + ) + ) + + result = await crud.create_documents( + db, + payload, + workspace, + observer=observer, + observed=observed, + # Semantic dedup would replace or reject a seeded row against whatever the + # deriver already wrote, making the fixture's counts depend on the provider. + deduplicate=False, + ) + assert_clean(result, len(items), f"{observer} -> {observed} {level}") + stored[level] = len(result.created_documents) + + # Outside the write session on purpose: create_documents commits as it goes, and the + # check should read the committed rows back on its own connection rather than through + # the identity map of the session that wrote them. + await verify_links(modules, workspace, observer, observed, premise_ids, derived) + + return stored + + +async def verify_links( + modules: dict[str, Any], + workspace: str, + observer: str, + observed: str, + premise_ids: list[str], + derived: dict[str, list[dict[str, Any]]], +) -> None: + """Confirm the premise links actually traverse, through Honcho's own read helpers. + + Nothing in the write path validates source_ids: a dangling id is stored happily and then + degrades quietly at read time to "referenced N premise IDs but none found in database", + while the conclusions endpoint does not expose source_ids at all. So a completely broken + reasoning tree looks exactly like a working one from outside. + + Both directions are read back from the stored rows rather than from what this script + intended, which is the only version of the check that can fail. Downward: every cited + premise must be reachable from its children, which is what catches a child written into a + different (observer, observed) collection. Upward: the source_ids those children actually + carry must all resolve to live rows. + """ + crud = modules["crud"] + tracked_db = modules["tracked_db"] + + cited = sorted( + { + premise_ids[index] + for items in derived.values() + for item in items + for index in item["premises"] + } + ) + if not cited: + return + + async with tracked_db("sandbox.verify_links") as db: + declared: set[str] = set() + for premise_id in cited: + children = await crud.get_child_observations( + db, workspace, premise_id, observer=observer, observed=observed + ) + if not children: + raise InjectError( + f"{observer} -> {observed}: premise {premise_id} has no reachable " + "children, so the reasoning tree does not traverse downward. Premise and " + "conclusion must share one (observer, observed) pair." + ) + for child in children: + declared.update(child.source_ids or []) + + resolved = await crud.get_documents_by_ids(db, workspace, sorted(declared)) + missing = declared - {document.id for document in resolved} + if missing: + raise InjectError( + f"{observer} -> {observed}: {len(missing)} stored premise id(s) resolve to " + "nothing, so the reasoning chain would read as empty. First: " + f"{sorted(missing)[0]}" + ) + + log(f"{observer} -> {observed}: {len(cited)} premise link(s) traverse both ways") + + +async def run(fixture: dict[str, Any]) -> int: + from src import crud, models, schemas + from src.cache.client import close_cache, init_cache + from src.db import engine + from src.dependencies import tracked_db + from src.embedding_client import embedding_client + + check_signatures(crud, schemas) + + workspace = fixture["workspace"] + session = fixture["session"] + peers = fixture["peers"] + + planned: list[tuple[str, str, dict[str, list[dict[str, Any]]]]] = [] + for spec in peers: + if not any(spec.get(level) for level in LEVELS): + continue + explicit_count = len(spec.get("explicit", [])) + conclusions = { + level: normalize(spec.get(level, []), level, spec["id"], explicit_count) + for level in LEVELS + } + if ( + any(conclusions[level] for level in ("deductive", "inductive")) + and not explicit_count + ): + raise InjectError( + f"peer {spec['id']!r} has derived conclusions but no explicit ones for their " + "premises to point at." + ) + for observer in resolve_observers(spec, peers): + planned.append((observer, spec["id"], conclusions)) + + if not planned: + log("fixture declares no conclusions - nothing to seed") + return 0 + + # cashews decorates the collection lookups, and an unconfigured backend raises rather than + # degrading, so the cache has to be up before the first crud call. + await init_cache() + try: + modules = { + "crud": crud, + "schemas": schemas, + "models": models, + "embedding_client": embedding_client, + "tracked_db": tracked_db, + } + for observer, observed, conclusions in planned: + stored = await inject_pair( + modules, workspace, session, observer, observed, conclusions + ) + summary = " ".join(f"{level}={stored.get(level, 0)}" for level in LEVELS) + log(f"{observer} -> {observed}: {summary}") + finally: + await close_cache() + # Without this the process can hang on exit holding pool connections, which inside + # `compose exec` looks like the seed itself wedging. + await engine.dispose() + + return 0 + + +def main() -> int: + raw = os.environ.get("SANDBOX_FIXTURE_JSON") + if not raw: + raise InjectError( + "SANDBOX_FIXTURE_JSON is unset. This script is run by sandbox.sh, which passes the " + "fixture in through the environment; it is not meant to be run by hand." + ) + return asyncio.run(run(json.loads(raw))) + + +if __name__ == "__main__": + try: + sys.exit(main()) + except InjectError as exc: + print(f"[conclusions] FAILED: {exc}", file=sys.stderr) + sys.exit(1) diff --git a/sandbox/sandbox.sh b/sandbox/sandbox.sh index 475efa2f..4f4e0b85 100755 --- a/sandbox/sandbox.sh +++ b/sandbox/sandbox.sh @@ -175,6 +175,26 @@ MSG )" } +seed_py() { + SANDBOX_BASE_URL="http://127.0.0.1:${SANDBOX_API_PORT:-18000}" \ + uv run --no-project --with-editable "$REPO/sdks/python" \ + python "$HERE/seed.py" "$1" +} + +# Conclusion levels and premise links are not reachable from the public API, so this +# part of the fixture is written by a script running inside the api container, which +# already is Honcho's venv with the api's settings and embedding client. The script +# arrives on stdin and the fixture in the environment, so nothing has to be mounted +# and nothing is left behind in the container. +# +# It decides for itself whether the fixture declares any conclusions, rather than +# being gated by a grep here that a comment mentioning a level name would fool. +inject_conclusions() { + compose exec -T \ + -e SANDBOX_FIXTURE_JSON="$(cat "$HERE/fixture.json")" \ + api /app/.venv/bin/python - < "$HERE/inject_conclusions.py" +} + # -------------------------------------------------------------------------- # Commands # -------------------------------------------------------------------------- @@ -240,9 +260,9 @@ cmd_seed() { compose up -d --wait --no-recreate >/dev/null say "seeding ($PROVIDER)" - SANDBOX_BASE_URL="http://127.0.0.1:${SANDBOX_API_PORT:-18000}" \ - uv run --no-project --with-editable "$REPO/sdks/python" \ - python "$HERE/seed.py" + seed_py seed + inject_conclusions + seed_py verify say "snapshotting to $TEMPLATE" stamp_fingerprint @@ -322,7 +342,8 @@ fingerprint() { alembic="$(compose exec -T database psql -tAX -U postgres -d "$DB" \ -c "SELECT version_num FROM alembic_version" 2>/dev/null | tr -d '[:space:]')" local files - files="$(cat "$HERE/fixture.json" "$HERE/seed.py" | shasum -a 256 | cut -c1-16)" + files="$(cat "$HERE/fixture.json" "$HERE/seed.py" "$HERE/inject_conclusions.py" \ + | shasum -a 256 | cut -c1-16)" echo "alembic=$alembic fixture=$files provider=$PROVIDER" } diff --git a/sandbox/seed.py b/sandbox/seed.py index 9ec544b3..59d19e14 100644 --- a/sandbox/seed.py +++ b/sandbox/seed.py @@ -3,13 +3,25 @@ Run through sandbox.sh, which supplies the base URL and owns the Docker and Postgres side. This script only talks to the API: - uv run --no-project --with-editable ./sdks/python python sandbox/seed.py + uv run --no-project --with-editable ./sdks/python python sandbox/seed.py verify -It is provider-agnostic. Under the mock provider the *text* of a conclusion is -synthetic and unrelated to the messages, so nothing here asserts on conclusion -content - only that derivation ran and produced something. What it does assert -strictly is the round trip that harness integrations actually get wrong: which -messages landed, on which peers, with which observation topology. +Two phases, because seeded conclusions are written between them by +inject_conclusions.py, which sandbox.sh runs inside the api container: + + seed.py seed peers, messages, derivation + seed.py verify the whole round trip, seeded conclusions included + +The split is what keeps the deriver honest. The check that proves derivation ran +at all is "some conclusion exists", and once conclusions are seeded that would +pass against a completely dead deriver - so it runs in the seed phase, before +anything is injected. + +It is provider-agnostic. Under the mock provider the *text* of a derived +conclusion is synthetic and unrelated to the messages, so nothing here asserts on +derived content. Seeded conclusions are the opposite: their text is committed, so +they are asserted exactly, in both modes. What is asserted strictly either way is +the round trip that harness integrations actually get wrong: which messages +landed, on which peers, with which observation topology. Exits non-zero on any failure. An empty sandbox that reports success is the exact failure mode this whole thing exists to prevent. @@ -43,6 +55,63 @@ def log(message: str) -> None: print(f"[seed] {message}", flush=True) +LEVELS = ("explicit", "deductive", "inductive") + + +def seeded_conclusions( + fixture: dict[str, Any], +) -> dict[tuple[str, str], dict[str, list[str]]]: + """What inject_conclusions.py should have written, keyed by (observer, observed). + + Mirrors the injector's resolution rules so verification is stated in terms of the + fixture rather than of whatever happens to be in the database. + """ + peers = fixture["peers"] + planned: dict[tuple[str, str], dict[str, list[str]]] = {} + for spec in peers: + by_level = { + level: [ + item if isinstance(item, str) else item["content"] + for item in spec.get(level, []) + ] + for level in LEVELS + } + if not any(by_level.values()): + continue + override = spec.get("observer") + observers = ( + [override] + if override + else [ + peer["id"] + for peer in peers + if peer.get("observe_others") and peer["id"] != spec["id"] + ] + ) + for observer in observers: + planned[(observer, spec["id"])] = by_level + return planned + + +def cited_premise_texts(spec: dict[str, Any], explicit: list[str]) -> set[str]: + """The explicit conclusions this peer's derived conclusions name as premises.""" + return { + explicit[index] + for level in ("deductive", "inductive") + for item in spec.get(level, []) + if isinstance(item, dict) + for index in item.get("premises", []) + } + + +def count_conclusions(peers: dict[str, Any]) -> int: + total = 0 + for observer in peers.values(): + for observed_id in peers: + total += len(list(observer.conclusions_of(observed_id).list(size=100))) + return total + + def drain(honcho: Honcho, what: str) -> None: """Block until the deriver queue is empty, or fail loudly. @@ -69,16 +138,25 @@ def drain(honcho: Honcho, what: str) -> None: def main() -> int: + phase = sys.argv[1] if len(sys.argv) > 1 else "seed" + if phase not in ("seed", "verify"): + raise SeedError(f"unknown phase {phase!r} (expected 'seed' or 'verify')") + fixture = json.loads(FIXTURE.read_text()) base_url = os.environ.get("SANDBOX_BASE_URL", "http://127.0.0.1:18000") workspace_id = fixture["workspace"] - log(f"seeding workspace {workspace_id!r} at {base_url}") honcho = Honcho(base_url=base_url, workspace_id=workspace_id, api_key="sandbox") - peers = {spec["id"]: honcho.peer(spec["id"]) for spec in fixture["peers"]} session = honcho.session(fixture["session"]) + if phase == "verify": + verify(honcho, session, peers, fixture) + log("verify complete") + return 0 + + log(f"seeding workspace {workspace_id!r} at {base_url}") + session.add_peers( [ ( @@ -112,7 +190,17 @@ def main() -> int: if fixture.get("dreams"): drain(honcho, "dream") - verify(honcho, session, peers, fixture) + # The one check that proves derivation happened, made here rather than in verify + # because seeded conclusions land afterwards and would satisfy it on their own. + derived = count_conclusions(peers) + if derived == 0: + raise SeedError( + "no conclusions were derived. The queue drained, so the deriver ran and " + "produced nothing - check the provider wiring " + "(LLM_OPENAI_BASE_URL / EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL) " + "and the deriver logs." + ) + log(f"verified {derived} derived conclusions present") log("seed complete") return 0 @@ -154,10 +242,10 @@ def verify( ) log("verified observation topology") - # Presence only. Under the mock provider, conclusion text is synthetic and - # says nothing about the messages, and the level mix is explicit-only because - # the Dreamer specialists write via tool calls the mock never emits. Asserting - # on either would pass in real mode and fail in mock mode. + # Derived conclusions get presence only. Under the mock provider their text is + # synthetic and says nothing about the messages, and the level mix is + # explicit-only because the Dreamer specialists write via tool calls the mock + # never emits. Asserting on either would pass in real mode and fail in mock mode. total = 0 for observer_id, observer in peers.items(): for observed_id in peers: @@ -168,13 +256,85 @@ def verify( if total == 0: raise SeedError( - "no conclusions were derived. The queue drained, so the deriver ran and " - "produced nothing - check the provider wiring " + "no conclusions at all. Both derivation and injection produced nothing - " + "check the provider wiring " "(LLM_OPENAI_BASE_URL / EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL) " "and the deriver logs." ) log(f"verified {total} conclusions present") + verify_seeded(peers, fixture, total) + + +def verify_seeded( + peers: dict[str, Any], + fixture: dict[str, Any], + total: int, +) -> None: + """Assert the seeded conclusions landed exactly, level by level. + + Seeded text is committed, so unlike derived conclusions it is asserted by content + in both modes - which is the whole reason the fixture carries these keys. + """ + planned = seeded_conclusions(fixture) + if not planned: + log("fixture declares no conclusions to seed") + return + + seeded_total = 0 + for (observer_id, observed_id), by_level in planned.items(): + stored = list(peers[observer_id].conclusions_of(observed_id).list(size=100)) + for level, expected in by_level.items(): + if not expected: + continue + actual = {c.content for c in stored if c.level == level} + missing = [content for content in expected if content not in actual] + if missing: + raise SeedError( + f"{observer_id} -> {observed_id}: {len(missing)} seeded {level} " + f"conclusion(s) missing, first is {missing[0]!r}. Honcho collapses " + "conclusions whose content matches something already stored, so " + "check for near-duplicate text in the fixture." + ) + seeded_total += len(expected) + summary = " ".join(f"{level}={len(by_level[level])}" for level in LEVELS) + log(f"verified seeded {observer_id} -> {observed_id}: {summary}") + + # Derived and seeded conclusions must both be present. The seed phase already + # proved derivation ran against an un-injected database; this catches the reverse + # error of an injection that somehow replaced the derived rows. + if total <= seeded_total: + raise SeedError( + f"found {total} conclusions but {seeded_total} were seeded, leaving none " + "derived. Injection should add to the deriver's output, not replace it." + ) + log( + f"verified {total - seeded_total} derived conclusions alongside {seeded_total} seeded" + ) + + # Premise text is what the working representation renders for a derived + # conclusion, and it is the only part of the reasoning tree an API client can + # see - schemas.Conclusion does not expose source_ids. + by_peer = {spec["id"]: spec for spec in fixture["peers"]} + for (observer_id, observed_id), by_level in planned.items(): + premise_texts = cited_premise_texts(by_peer[observed_id], by_level["explicit"]) + if not premise_texts: + continue + rendered = peers[observer_id].representation( + target=observed_id, max_conclusions=100 + ) + absent = [text for text in premise_texts if text not in rendered] + if absent: + raise SeedError( + f"{observer_id} -> {observed_id}: premise text missing from the " + f"representation, first is {absent[0]!r}. Premises render only for " + "deductive conclusions and sources only for inductive ones, so a " + "premise stored under the wrong metadata key renders as nothing." + ) + log( + f"verified premise text renders in {observer_id}'s representation of {observed_id}" + ) + if __name__ == "__main__": try: