RomanNum3ral
/
honcho
mirror of https://github.com/plastic-labs/honcho.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

feat(sandbox): seed conclusions at every reasoning level 

The fixture could only get memory by running the deriver, and derived memory is
assertable in exactly one mode. Under the mock provider a conclusion's text is
synthetic and its level is always explicit, because the Dreamer's specialists
write via tool calls the mock never emits -- so the README had to say: assert
that conclusions exist, never what they say. That rules out content assertions
in the only mode that is free and deterministic, which is most of what a harness
test wants.

fixture.json now takes optional explicit/deductive/inductive keys per peer. The
peer carrying them is the observed and the observer is inferred from
observe_others, so the standard shape resolves to assistant -> alice, the pair
the dream already uses. Items are a bare string or {content, premises}, where a
premise indexes the same peer's explicit list. With no keys the sandbox behaves
exactly as before.

Level and premise links are not reachable from the public API:
crud.create_observations hardcodes level="explicit" with no source_ids, and
neither the schema nor the SDKs have a field for either. Widening the API was
rejected -- it would let any client assert a conclusion is deductive with
premises it invented. So inject_conclusions.py runs inside the api container,
which already is Honcho's venv with the api's settings and a live embedding
client; the script arrives on stdin and the fixture in the environment, so
nothing is mounted and nothing is left in the container. The cost is calling
internals out of a pinned image, so it checks the signatures it depends on and
says "bump image.env and update this script together" rather than half-seeding.

Premise indices become real source_ids pointing at the rows from the explicit
pass, so the tree actually traverses in both directions; synthetic ids would
satisfy the schema and resolve to nothing. Two passes suffice because premises
only ever cite explicit conclusions.

Every count is asserted exactly, because each failure mode here reduces a count
without raising: exact-content dedup is always on and cannot be disabled,
semantic dedup replaces rows, per-item embedding failures drop rows, and the
session-purity invariant skips session-less explicit rows. Conclusions on a peer
nobody observes is an error rather than a silent no-op. And because a broken
reasoning tree is invisible from outside -- the API does not expose source_ids
and get_reasoning_chain is a Dialectic tool, not a route -- the seeder reads its
own links back before the snapshot: every cited premise must be reachable from
its children, and the source_ids those children carry must all resolve.

seed.py splits into seed and verify phases so injection cannot mask the deriver.
The check that proves derivation ran is "some conclusion exists", which seeded
rows would satisfy on their own, so it now runs before anything is injected.
Verify then asserts seeded content and level exactly, in both modes, plus that
premise text renders in the representation. The fingerprint hashes the new file
too, or editing it would restore a stale template.

Verified from clean volumes in mock mode: up in 41.5s seeding 4 explicit, 2
deductive and 1 inductive alongside 4 derived, with 3 premise links traversing
both ways and every declared premise resolving in SQL; reset restores it in
0.89s. Negative cases all fail loudly: an unobserved peer, an out-of-range
premise index, derived conclusions with no explicit to cite, near-duplicate
content collapsing under dedup, a dangling premise id, and a premise sought in
the wrong collection. Real mode is unverified -- it needs a key and injection
touches the provider only through the embedding client.
This commit is contained in:
Eugene Eisenstein 2026-09-03 16:45:00 -04:00
parent 03fd460b3f
commit 35e7b99846
5 changed files with 820 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

97
sandbox/README.md
View File

@ -3,8 +3,9 @@
A local Honcho you can wipe and rebuild in seconds, so harness testing stops depending on
whatever state your laptop happens to be in.
It comes up **already seeded** — a known workspace, peers, session, messages, and conclusions the
deriver has actually produced — and `reset` returns it to that exact state without re-deriving.
It comes up **already seeded** — a known workspace, peers, session, messages, conclusions the
deriver has actually produced, and committed conclusions at every reasoning level — and `reset`
returns it to that exact state without re-deriving.
```bash
sandbox/sandbox.sh up # start + seed (first run also builds/pulls)
@ -29,14 +30,19 @@ Both ways of resolving that ship here; `mock` is the default.
| **Seed time** | ~15s | ~6 min |
| **Reset time** | 0.86s | 1.29s |
| **Vector recall** | **Untestable** — see below | Testable |
| **What you get** | 4 conclusions, all `explicit`, synthetic text | 22 conclusions — 16 `explicit`, 4 `deductive`, 2 `inductive` |
| **Derived** | 4 conclusions, all `explicit`, synthetic text | 22 conclusions — 16 `explicit`, 4 `deductive`, 2 `inductive` |
| **Seeded** | 7 — 4 `explicit`, 2 `deductive`, 1 `inductive`, committed text | the same 7, identical bytes |
| **Use it for** | CI, harness smoke tests, anything that needs a repeatable answer | Recall quality, validating against real model output |
Those counts are from the committed fixture, measured on both paths. They are the sharpest
The derived counts are from the committed fixture, measured on both paths. They are the sharpest
illustration of the difference: mock gives you four copies of `[mock] mock dummy placeholder …`,
while real mode reasons its way from "my cat Marzipan is named after the pylon's colour" to the
deductive conclusion *"alice's cat is named after a feature of alice's pedestrian bridge project."*
Anything asserting on that kind of inference needs real mode.
Anything asserting on a model's *own* inference needs real mode.
The seeded row is why you may not need to. Seeded conclusions are written straight into the
collection rather than derived, so their text and level are identical in both modes — see
[Seeded conclusions](#seeded-conclusions).
```bash
sandbox/sandbox.sh up # mock
@ -54,10 +60,11 @@ exactly as far apart as two unrelated strings. Any recall assertion built on moc
lexical or full-text search. A vector-ranking assertion will fail there for reasons that have
nothing to do with the code you are testing — that is what `real` mode is for.
**Mock conclusions are synthetic.** The text is derived from the request, not from the meaning of
your messages, and the level is always `explicit` (the Dreamer's specialists write via tool calls,
which the mock deliberately never emits). Assert that conclusions *exist*; do not assert on what
they say or on the level mix, or your test will pass in one mode and fail in the other.
**Mock *derived* conclusions are synthetic.** The text comes from the request, not from the meaning
of your messages, and the level is always `explicit` (the Dreamer's specialists write via tool
calls, which the mock deliberately never emits). Assert that derived conclusions *exist*; do not
assert on what they say or on the level mix, or your test will pass in one mode and fail in the
other. Assert against the seeded conclusions instead — that is what they are for.
### Real mode
@ -109,6 +116,77 @@ The cost of a snapshot is that it can go stale. Three guards:
If you hit the staleness refusal: `sandbox/sandbox.sh seed`.
## Seeded conclusions
`fixture.json` can carry pre-made conclusions per peer, at any of the three reasoning levels.
All three keys are optional; with none of them the sandbox behaves exactly as it would without
this feature.
```json
{
"id": "alice",
"observe_me": true,
"observe_others": false,
"explicit": [
"alice is a structural engineer based in Rotterdam",
"alice has a cat called Marzipan, named after the colour of the bridge's pylon"
],
"deductive": [
{ "content": "alice named her cat after a feature of the bridge she is designing",
"premises": [0, 1] }
],
"inductive": ["alice optimises for low lifetime maintenance cost"]
}
```
**The peer carrying the keys is the one being observed.** Conclusions are keyed by an
`(observer, observed)` pair, and the observer is inferred: every fixture peer with
`observe_others` set, excluding the observed itself. For the standard harness shape that is
exactly `assistant -> alice` — the same pair the dream uses and the one a harness reads from. A
peer nobody else observes is an **error**, not a silent no-op; add an explicit `"observer"` to
that peer to override the inference, which is also how you reach self-representation
(`alice -> alice`).
**Items are a bare string, or `{content, premises}`.** Each premise is an index into the same
peer's `explicit` list. Premises are optional, but a derived conclusion without them is not
something the Dreamer would ever write, so the committed fixture always supplies them.
**Contents must be unique.** Honcho collapses a conclusion whose content matches something
already stored — case- and whitespace-insensitively — and that cannot be switched off. The
seeder asserts exact counts, so a near-duplicate fails the seed rather than quietly vanishing.
### What premises buy you, and what they don't
The premise indices become real `Document.source_ids` pointing at the actual rows from the
`explicit` pass, so the reasoning tree genuinely traverses in both directions. Two caveats on
where that is visible:
- **Premise text renders in the representation.** `peer.representation(target=…)` prints each
premise indented under its conclusion, which is the practical payoff and what `seed.py verify`
asserts.
- **Premise *links* are not on the API.** The conclusion response carries `level` but not
`source_ids`, and `get_reasoning_chain` is a Dialectic tool rather than a route. So a test can
only reach the links through the Dialectic, or through SQL. Because of that, a broken
reasoning tree is invisible from outside — so the seeder checks its own links before the
snapshot is taken, resolving every premise id and confirming each is reachable from its
children. A dangling or mis-filed premise fails the seed.
### Why this is not done over the API
The public create-conclusions endpoint always writes `level="explicit"` with no premises, and
neither the schema nor the SDKs have a field for either. The columns exist; only an in-process
caller can set them. So `sandbox.sh` runs `inject_conclusions.py` **inside the api container**,
which already is Honcho's venv with the api's settings and a live embedding client — the script
goes in on stdin and the fixture through the environment, so nothing is mounted and nothing is
left behind.
The cost is that the injector calls Honcho *internals*, which carry no stability contract, out
of the image pinned in `image.env`. It checks the signatures it depends on before writing and
fails with a "bump `image.env` and update this script together" message, rather than half-seeding
a database. Widening the public API to accept a level was the alternative and was deliberately
rejected: it would let any client assert a conclusion is `deductive` with premises it invented.
## What's in the fixture
`fixture.json` — two peers with **explicitly stated** observation topology (`alice` is observed and
@ -137,6 +215,7 @@ dreams is 8 hours — so the seed schedules one directly.
| `init.sql` | Creates the `honcho_sandbox` database. |
| `fixture.json` | The seeded conversation. |
| `seed.py` | Populates and verifies. Talks only to the API. |
| `inject_conclusions.py` | Writes the seeded conclusions. Runs inside the api container. |
| `real.env.example` | Template for real-mode credentials. |
Exactly one provider overlay is always composed on top of the base, so the choice is visible in the

31
sandbox/fixture.json
View File

@ -3,8 +3,37 @@
"session": "sandbox-session",
"_comment": "Peer topology is stated explicitly because it is the thing that silently breaks (DEV-2462 asserts on it). alice is observed and does not observe; the assistant observes and is not observed - the standard harness shape.",
"_conclusions_comment": "The explicit/deductive/inductive keys are optional seeded memory, written straight into the (observer, observed) collection rather than derived. They exist because derived conclusions are only assertable in real mode: the mock provider's conclusion text is synthetic and its level is always explicit. Seeded text is identical in both modes, so a harness test can assert on content and level for free. Premise indices point into the same peer's explicit list, and they are what make the conclusion's reasoning chain resolve - excadrill's equivalent fixtures use synthetic source ids that never resolve to a real document. A premise-less derived conclusion is allowed by the format but is not something the Dreamer would ever write, so the committed fixture does not use one.",
"peers": [
{ "id": "alice", "observe_me": true, "observe_others": false },
{
"id": "alice",
"observe_me": true,
"observe_others": false,
"explicit": [
"alice is a structural engineer based in Rotterdam",
"alice is designing a pedestrian bridge over the Nieuwe Maas spanning 240 metres on a single cable-stayed pylon",
"alice specified weathering steel for the bridge because its maintenance budget is close to zero",
"alice has a cat called Marzipan, named after the colour of the bridge's pylon"
],
"deductive": [
{
"content": "alice named her cat after a feature of the bridge she is designing",
"premises": [1, 3]
},
{
"content": "alice selects materials on lifetime maintenance cost rather than capital cost",
"premises": [2]
}
],
"inductive": [
{
"content": "alice lets the Nieuwe Maas bridge project shape decisions well outside her working hours",
"premises": [1, 3]
}
]
},
{ "id": "assistant", "observe_me": false, "observe_others": true }
],

502
sandbox/inject_conclusions.py Normal file
View File

@ -0,0 +1,502 @@
"""Seed conclusions at every reasoning level, from inside the api container.
Run by sandbox.sh, never from the host:
compose exec -T -e SANDBOX_FIXTURE_JSON="$(cat fixture.json)" api \
/app/.venv/bin/python - < sandbox/inject_conclusions.py
Level and premise links are not reachable from the public API. `crud.create_observations`
hardcodes `level="explicit"`, `source_ids=NULL`, `internal_metadata={}`, and `ConclusionCreate`
has no field for any of them. The columns exist on Document; only an in-process caller can set
them. So this script imports Honcho's own write helpers, which is why it runs in the container
rather than beside seed.py: the container already *is* Honcho's venv, with the api's settings
and a correctly wired embedding client.
That comes at a price worth stating. These are internal APIs with no stability contract, and
they are imported out of the *pinned* image, not the working tree — so a digest bump can move
them underneath us. `check_signatures` fails loudly on that rather than letting a broken seed
look like a working one.
Two passes, not three: premise indices point into the same peer's `explicit` list, so both
derived levels reference pass-1 ids and nothing has to reference a derived id.
Everything here is asserted exactly, because every silent-failure mode in this path reduces a
count without raising: exact-content dedup is always on and cannot be switched off, semantic
dedup replaces rows, per-item embedding failures drop rows, and the session-purity invariant
skips explicit rows with no session. A seed that quietly wrote nothing is the failure this
whole sandbox exists to prevent.
"""
from __future__ import annotations
import asyncio
import inspect
import json
import os
import sys
from typing import Any
LEVELS = ("explicit", "deductive", "inductive")
# Levels whose premise text renders under a different metadata key. The working representation
# prints DocumentMetadata.premises for deductive and .sources for inductive, and each is read
# only for its own level, so putting the text under the wrong one renders nothing.
PREMISE_FIELD = {"deductive": "premises", "inductive": "sources"}
class InjectError(RuntimeError):
"""Seeding conclusions did not reach a usable state."""
def log(message: str) -> None:
print(f"[conclusions] {message}", flush=True)
def check_signatures(crud: Any, schemas: Any) -> None:
"""Fail loudly if the image's internals moved.
Cheap insurance against the real hazard of importing unversioned internals out of a pinned
image: without this, a renamed keyword surfaces as a TypeError mid-seed, or worse, as a
seed that silently wrote fewer rows.
"""
expected = {
"create_documents": (
"documents",
"workspace_name",
"observer",
"observed",
"deduplicate",
),
"create_observations": ("observations", "workspace_name"),
"get_or_create_collection": ("workspace_name", "observer", "observed"),
"get_documents_by_ids": ("workspace_name", "document_ids"),
"get_child_observations": (
"workspace_name",
"parent_id",
"observer",
"observed",
),
}
for name, params in expected.items():
signature = inspect.signature(getattr(crud, name))
missing = [p for p in params if p not in signature.parameters]
if missing:
raise InjectError(
f"crud.{name} is missing expected parameters {missing} in this image. "
"The sandbox seeds conclusions through Honcho internals, which carry no "
"stability contract; bump sandbox/image.env and update this script together."
)
for model, fields in (
(
schemas.DocumentCreate,
("content", "level", "metadata", "embedding", "source_ids"),
),
(
schemas.DocumentMetadata,
("message_ids", "message_created_at", "premises", "sources"),
),
):
missing = [f for f in fields if f not in model.model_fields]
if missing:
raise InjectError(
f"{model.__name__} is missing expected fields {missing} in this image. "
"Bump sandbox/image.env and update this script together."
)
def normalize(
items: list[Any], level: str, peer_id: str, explicit_count: int
) -> list[dict[str, Any]]:
"""Accept either a bare string or {content, premises}, and validate premise indices.
An out-of-range index is rejected here rather than written as a dangling source id. Nothing
downstream validates source_ids -- a bad one resolves to "referenced N premise IDs but none
found in database" at read time, which is precisely the quiet wrongness to avoid.
"""
normalized: list[dict[str, Any]] = []
for position, item in enumerate(items):
if isinstance(item, str):
content, premises = item, []
elif isinstance(item, dict):
content = item.get("content")
premises = item.get("premises", [])
unknown = set(item) - {"content", "premises"}
if unknown:
raise InjectError(
f"{peer_id}.{level}[{position}] has unknown keys {sorted(unknown)}"
)
else:
raise InjectError(
f"{peer_id}.{level}[{position}] must be a string or an object, got {type(item).__name__}"
)
if not content or not content.strip():
raise InjectError(f"{peer_id}.{level}[{position}] has empty content")
if premises and level == "explicit":
raise InjectError(
f"{peer_id}.explicit[{position}] declares premises. Explicit conclusions come "
"straight from messages and are the premises other levels point at."
)
for index in premises:
if not isinstance(index, int) or not 0 <= index < explicit_count:
raise InjectError(
f"{peer_id}.{level}[{position}] premise {index!r} is not a valid index into "
f"{peer_id}.explicit (which has {explicit_count} entries)"
)
normalized.append({"content": content, "premises": premises})
return normalized
def resolve_observers(spec: dict[str, Any], peers: list[dict[str, Any]]) -> list[str]:
"""Who holds conclusions about this peer.
The peer carrying the keys is the observed. Absent an explicit override, the observers are
every peer configured to observe others -- which for the standard harness shape is exactly
the assistant. Resolving to nobody is an error: the conclusions would be written into no
collection at all and the seed would report success having stored nothing.
"""
observed = spec["id"]
override = spec.get("observer")
if override:
return [override]
observers = [
peer["id"]
for peer in peers
if peer.get("observe_others") and peer["id"] != observed
]
if not observers:
raise InjectError(
f"peer {observed!r} has seeded conclusions but no other peer observes it: "
"no fixture peer besides itself sets observe_others, and it declares no "
f'explicit "observer". The conclusions would be written nowhere. Either give '
f'{observed!r} an "observer", or set observe_others on the peer that should '
"hold them."
)
return observers
async def latest_message_timestamp(
db: Any, models: Any, workspace: str, session: str
) -> str | None:
"""The conversation's own clock, for the representation to render.
Derived conclusions are stamped with the last ingested message time rather than the seed's
wall-clock, so the representation shows when the conversation happened. Honcho's own dream
path back-dates the same way.
"""
from sqlalchemy import func, select
result = await db.execute(
select(func.max(models.Message.created_at)).where(
models.Message.workspace_name == workspace,
models.Message.session_name == session,
)
)
newest = result.scalar_one_or_none()
return newest.strftime("%Y-%m-%dT%H:%M:%SZ") if newest else None
def assert_clean(result: Any, requested: int, label: str) -> None:
"""No row may be dropped, deduped, or replaced.
create_documents reports these as counters rather than raising, so without this the seed
reports success while having written fewer conclusions than the fixture declares. Exact
content dedup cannot be disabled, so this fires on a reseed over existing content too --
which is correct: seed.py always starts from an empty database.
"""
created = len(result.created_documents)
counters = {
"exact duplicates within the batch": result.exact_dup_in_batch_count,
"exact duplicates already stored": result.exact_dup_existing_count,
"semantically rejected": result.semantic_dup_rejected_count,
"semantically replaced": result.semantic_dup_replaced_count,
}
dropped = {name: count for name, count in counters.items() if count}
if created != requested or dropped:
raise InjectError(
f"{label}: asked for {requested} conclusions, stored {created}"
+ (
f" ({', '.join(f'{n}: {c}' for n, c in dropped.items())})"
if dropped
else ""
)
+ ". Conclusion contents must be unique; near-identical text is collapsed by "
"Honcho's dedup before it reaches the database."
)
async def inject_pair(
modules: dict[str, Any],
workspace: str,
session: str,
observer: str,
observed: str,
conclusions: dict[str, list[dict[str, Any]]],
) -> dict[str, int]:
"""Seed one (observer, observed) collection. Returns per-level counts actually stored."""
crud = modules["crud"]
schemas = modules["schemas"]
models = modules["models"]
embedding_client = modules["embedding_client"]
tracked_db = modules["tracked_db"]
stored: dict[str, int] = {}
# Pass 1. The public write path is the only one that hands back rows, so it is how the
# premise ids are obtained -- and it does no dedup, so nothing is silently collapsed.
explicit = conclusions["explicit"]
premise_ids: list[str] = []
async with tracked_db("sandbox.seed_explicit") as db:
await crud.get_or_create_collection(
db, workspace, observer=observer, observed=observed
)
if explicit:
documents = await crud.create_observations(
db,
[
schemas.ConclusionCreate(
content=item["content"],
observer_id=observer,
observed_id=observed,
# Explicit rows must carry a session: create_documents refuses
# session-less explicit rows on a session-purity invariant, and an
# explicit conclusion genuinely does come from a conversation.
session_id=session,
)
for item in explicit
],
workspace,
)
if len(documents) != len(explicit):
raise InjectError(
f"{observer} -> {observed}: asked for {len(explicit)} explicit "
f"conclusions, stored {len(documents)}"
)
premise_ids = [document.id for document in documents]
stored["explicit"] = len(documents)
# Pass 2. Both derived levels cite pass-1 ids, so neither needs the other's ids back.
derived = {
level: conclusions[level]
for level in ("deductive", "inductive")
if conclusions[level]
}
if derived:
async with tracked_db("sandbox.seed_derived") as db:
message_created_at = await latest_message_timestamp(
db, models, workspace, session
)
if message_created_at is None:
raise InjectError(
f"session {session!r} has no messages, so derived conclusions have no "
"conversation timestamp to carry. Seed messages before conclusions."
)
for level, items in derived.items():
contents = [item["content"] for item in items]
embeddings = await embedding_client.simple_batch_embed(
contents, on_oversize="truncate"
)
if len(embeddings) != len(contents):
raise InjectError(
f"{observer} -> {observed}: embedded {len(embeddings)} of "
f"{len(contents)} {level} conclusions"
)
payload: list[Any] = []
for item, embedding in zip(items, embeddings, strict=True):
source_ids = [premise_ids[index] for index in item["premises"]]
metadata: dict[str, Any] = {
# Not derived from messages, but the representation reads this to
# render the conclusion's timestamp.
"message_ids": [],
"message_created_at": message_created_at,
"source_ids": source_ids,
PREMISE_FIELD[level]: [
explicit[index]["content"] for index in item["premises"]
],
}
if level == "inductive":
metadata["pattern_type"] = "tendency"
metadata["confidence"] = (
"high" if len(source_ids) > 1 else "low"
)
payload.append(
schemas.DocumentCreate(
content=item["content"],
# Derived conclusions belong to the dream, not to one session,
# which is how the Dreamer writes them.
session_name=None,
level=level,
times_derived=1,
metadata=schemas.DocumentMetadata(**metadata),
embedding=embedding,
source_ids=source_ids,
)
)
result = await crud.create_documents(
db,
payload,
workspace,
observer=observer,
observed=observed,
# Semantic dedup would replace or reject a seeded row against whatever the
# deriver already wrote, making the fixture's counts depend on the provider.
deduplicate=False,
)
assert_clean(result, len(items), f"{observer} -> {observed} {level}")
stored[level] = len(result.created_documents)
# Outside the write session on purpose: create_documents commits as it goes, and the
# check should read the committed rows back on its own connection rather than through
# the identity map of the session that wrote them.
await verify_links(modules, workspace, observer, observed, premise_ids, derived)
return stored
async def verify_links(
modules: dict[str, Any],
workspace: str,
observer: str,
observed: str,
premise_ids: list[str],
derived: dict[str, list[dict[str, Any]]],
) -> None:
"""Confirm the premise links actually traverse, through Honcho's own read helpers.
Nothing in the write path validates source_ids: a dangling id is stored happily and then
degrades quietly at read time to "referenced N premise IDs but none found in database",
while the conclusions endpoint does not expose source_ids at all. So a completely broken
reasoning tree looks exactly like a working one from outside.
Both directions are read back from the stored rows rather than from what this script
intended, which is the only version of the check that can fail. Downward: every cited
premise must be reachable from its children, which is what catches a child written into a
different (observer, observed) collection. Upward: the source_ids those children actually
carry must all resolve to live rows.
"""
crud = modules["crud"]
tracked_db = modules["tracked_db"]
cited = sorted(
{
premise_ids[index]
for items in derived.values()
for item in items
for index in item["premises"]
}
)
if not cited:
return
async with tracked_db("sandbox.verify_links") as db:
declared: set[str] = set()
for premise_id in cited:
children = await crud.get_child_observations(
db, workspace, premise_id, observer=observer, observed=observed
)
if not children:
raise InjectError(
f"{observer} -> {observed}: premise {premise_id} has no reachable "
"children, so the reasoning tree does not traverse downward. Premise and "
"conclusion must share one (observer, observed) pair."
)
for child in children:
declared.update(child.source_ids or [])
resolved = await crud.get_documents_by_ids(db, workspace, sorted(declared))
missing = declared - {document.id for document in resolved}
if missing:
raise InjectError(
f"{observer} -> {observed}: {len(missing)} stored premise id(s) resolve to "
"nothing, so the reasoning chain would read as empty. First: "
f"{sorted(missing)[0]}"
)
log(f"{observer} -> {observed}: {len(cited)} premise link(s) traverse both ways")
async def run(fixture: dict[str, Any]) -> int:
from src import crud, models, schemas
from src.cache.client import close_cache, init_cache
from src.db import engine
from src.dependencies import tracked_db
from src.embedding_client import embedding_client
check_signatures(crud, schemas)
workspace = fixture["workspace"]
session = fixture["session"]
peers = fixture["peers"]
planned: list[tuple[str, str, dict[str, list[dict[str, Any]]]]] = []
for spec in peers:
if not any(spec.get(level) for level in LEVELS):
continue
explicit_count = len(spec.get("explicit", []))
conclusions = {
level: normalize(spec.get(level, []), level, spec["id"], explicit_count)
for level in LEVELS
}
if (
any(conclusions[level] for level in ("deductive", "inductive"))
and not explicit_count
):
raise InjectError(
f"peer {spec['id']!r} has derived conclusions but no explicit ones for their "
"premises to point at."
)
for observer in resolve_observers(spec, peers):
planned.append((observer, spec["id"], conclusions))
if not planned:
log("fixture declares no conclusions - nothing to seed")
return 0
# cashews decorates the collection lookups, and an unconfigured backend raises rather than
# degrading, so the cache has to be up before the first crud call.
await init_cache()
try:
modules = {
"crud": crud,
"schemas": schemas,
"models": models,
"embedding_client": embedding_client,
"tracked_db": tracked_db,
}
for observer, observed, conclusions in planned:
stored = await inject_pair(
modules, workspace, session, observer, observed, conclusions
)
summary = " ".join(f"{level}={stored.get(level, 0)}" for level in LEVELS)
log(f"{observer} -> {observed}: {summary}")
finally:
await close_cache()
# Without this the process can hang on exit holding pool connections, which inside
# `compose exec` looks like the seed itself wedging.
await engine.dispose()
return 0
def main() -> int:
raw = os.environ.get("SANDBOX_FIXTURE_JSON")
if not raw:
raise InjectError(
"SANDBOX_FIXTURE_JSON is unset. This script is run by sandbox.sh, which passes the "
"fixture in through the environment; it is not meant to be run by hand."
)
return asyncio.run(run(json.loads(raw)))
if __name__ == "__main__":
try:
sys.exit(main())
except InjectError as exc:
print(f"[conclusions] FAILED: {exc}", file=sys.stderr)
sys.exit(1)

29
sandbox/sandbox.sh
View File

@ -175,6 +175,26 @@ MSG
)"
}
seed_py() {
SANDBOX_BASE_URL="http://127.0.0.1:${SANDBOX_API_PORT:-18000}" \
uv run --no-project --with-editable "$REPO/sdks/python" \
python "$HERE/seed.py" "$1"
}
# Conclusion levels and premise links are not reachable from the public API, so this
# part of the fixture is written by a script running inside the api container, which
# already is Honcho's venv with the api's settings and embedding client. The script
# arrives on stdin and the fixture in the environment, so nothing has to be mounted
# and nothing is left behind in the container.
#
# It decides for itself whether the fixture declares any conclusions, rather than
# being gated by a grep here that a comment mentioning a level name would fool.
inject_conclusions() {
compose exec -T \
-e SANDBOX_FIXTURE_JSON="$(cat "$HERE/fixture.json")" \
api /app/.venv/bin/python - < "$HERE/inject_conclusions.py"
}
# --------------------------------------------------------------------------
# Commands
# --------------------------------------------------------------------------
@ -240,9 +260,9 @@ cmd_seed() {
compose up -d --wait --no-recreate >/dev/null
say "seeding ($PROVIDER)"
SANDBOX_BASE_URL="http://127.0.0.1:${SANDBOX_API_PORT:-18000}" \
uv run --no-project --with-editable "$REPO/sdks/python" \
python "$HERE/seed.py"
seed_py seed
inject_conclusions
seed_py verify
say "snapshotting to $TEMPLATE"
stamp_fingerprint
@ -322,7 +342,8 @@ fingerprint() {
alembic="$(compose exec -T database psql -tAX -U postgres -d "$DB" \
-c "SELECT version_num FROM alembic_version" 2>/dev/null | tr -d '[:space:]')"
local files
files="$(cat "$HERE/fixture.json" "$HERE/seed.py" | shasum -a 256 | cut -c1-16)"
files="$(cat "$HERE/fixture.json" "$HERE/seed.py" "$HERE/inject_conclusions.py" \
| shasum -a 256 | cut -c1-16)"
echo "alembic=$alembic fixture=$files provider=$PROVIDER"
}

190
sandbox/seed.py
View File

@ -3,13 +3,25 @@
Run through sandbox.sh, which supplies the base URL and owns the Docker and
Postgres side. This script only talks to the API:
uv run --no-project --with-editable ./sdks/python python sandbox/seed.py
uv run --no-project --with-editable ./sdks/python python sandbox/seed.py verify
It is provider-agnostic. Under the mock provider the *text* of a conclusion is
synthetic and unrelated to the messages, so nothing here asserts on conclusion
content - only that derivation ran and produced something. What it does assert
strictly is the round trip that harness integrations actually get wrong: which
messages landed, on which peers, with which observation topology.
Two phases, because seeded conclusions are written between them by
inject_conclusions.py, which sandbox.sh runs inside the api container:
seed.py seed peers, messages, derivation
seed.py verify the whole round trip, seeded conclusions included
The split is what keeps the deriver honest. The check that proves derivation ran
at all is "some conclusion exists", and once conclusions are seeded that would
pass against a completely dead deriver - so it runs in the seed phase, before
anything is injected.
It is provider-agnostic. Under the mock provider the *text* of a derived
conclusion is synthetic and unrelated to the messages, so nothing here asserts on
derived content. Seeded conclusions are the opposite: their text is committed, so
they are asserted exactly, in both modes. What is asserted strictly either way is
the round trip that harness integrations actually get wrong: which messages
landed, on which peers, with which observation topology.
Exits non-zero on any failure. An empty sandbox that reports success is the exact
failure mode this whole thing exists to prevent.
@ -43,6 +55,63 @@ def log(message: str) -> None:
print(f"[seed] {message}", flush=True)
LEVELS = ("explicit", "deductive", "inductive")
def seeded_conclusions(
fixture: dict[str, Any],
) -> dict[tuple[str, str], dict[str, list[str]]]:
"""What inject_conclusions.py should have written, keyed by (observer, observed).
Mirrors the injector's resolution rules so verification is stated in terms of the
fixture rather than of whatever happens to be in the database.
"""
peers = fixture["peers"]
planned: dict[tuple[str, str], dict[str, list[str]]] = {}
for spec in peers:
by_level = {
level: [
item if isinstance(item, str) else item["content"]
for item in spec.get(level, [])
]
for level in LEVELS
}
if not any(by_level.values()):
continue
override = spec.get("observer")
observers = (
[override]
if override
else [
peer["id"]
for peer in peers
if peer.get("observe_others") and peer["id"] != spec["id"]
]
)
for observer in observers:
planned[(observer, spec["id"])] = by_level
return planned
def cited_premise_texts(spec: dict[str, Any], explicit: list[str]) -> set[str]:
"""The explicit conclusions this peer's derived conclusions name as premises."""
return {
explicit[index]
for level in ("deductive", "inductive")
for item in spec.get(level, [])
if isinstance(item, dict)
for index in item.get("premises", [])
}
def count_conclusions(peers: dict[str, Any]) -> int:
total = 0
for observer in peers.values():
for observed_id in peers:
total += len(list(observer.conclusions_of(observed_id).list(size=100)))
return total
def drain(honcho: Honcho, what: str) -> None:
"""Block until the deriver queue is empty, or fail loudly.
@ -69,16 +138,25 @@ def drain(honcho: Honcho, what: str) -> None:
def main() -> int:
phase = sys.argv[1] if len(sys.argv) > 1 else "seed"
if phase not in ("seed", "verify"):
raise SeedError(f"unknown phase {phase!r} (expected 'seed' or 'verify')")
fixture = json.loads(FIXTURE.read_text())
base_url = os.environ.get("SANDBOX_BASE_URL", "http://127.0.0.1:18000")
workspace_id = fixture["workspace"]
log(f"seeding workspace {workspace_id!r} at {base_url}")
honcho = Honcho(base_url=base_url, workspace_id=workspace_id, api_key="sandbox")
peers = {spec["id"]: honcho.peer(spec["id"]) for spec in fixture["peers"]}
session = honcho.session(fixture["session"])
if phase == "verify":
verify(honcho, session, peers, fixture)
log("verify complete")
return 0
log(f"seeding workspace {workspace_id!r} at {base_url}")
session.add_peers(
[
(
@ -112,7 +190,17 @@ def main() -> int:
if fixture.get("dreams"):
drain(honcho, "dream")
verify(honcho, session, peers, fixture)
# The one check that proves derivation happened, made here rather than in verify
# because seeded conclusions land afterwards and would satisfy it on their own.
derived = count_conclusions(peers)
if derived == 0:
raise SeedError(
"no conclusions were derived. The queue drained, so the deriver ran and "
"produced nothing - check the provider wiring "
"(LLM_OPENAI_BASE_URL / EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL) "
"and the deriver logs."
)
log(f"verified {derived} derived conclusions present")
log("seed complete")
return 0
@ -154,10 +242,10 @@ def verify(
)
log("verified observation topology")
# Presence only. Under the mock provider, conclusion text is synthetic and
# says nothing about the messages, and the level mix is explicit-only because
# the Dreamer specialists write via tool calls the mock never emits. Asserting
# on either would pass in real mode and fail in mock mode.
# Derived conclusions get presence only. Under the mock provider their text is
# synthetic and says nothing about the messages, and the level mix is
# explicit-only because the Dreamer specialists write via tool calls the mock
# never emits. Asserting on either would pass in real mode and fail in mock mode.
total = 0
for observer_id, observer in peers.items():
for observed_id in peers:
@ -168,13 +256,85 @@ def verify(
if total == 0:
raise SeedError(
"no conclusions were derived. The queue drained, so the deriver ran and "
"produced nothing - check the provider wiring "
"no conclusions at all. Both derivation and injection produced nothing - "
"check the provider wiring "
"(LLM_OPENAI_BASE_URL / EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL) "
"and the deriver logs."
)
log(f"verified {total} conclusions present")
verify_seeded(peers, fixture, total)
def verify_seeded(
peers: dict[str, Any],
fixture: dict[str, Any],
total: int,
) -> None:
"""Assert the seeded conclusions landed exactly, level by level.
Seeded text is committed, so unlike derived conclusions it is asserted by content
in both modes - which is the whole reason the fixture carries these keys.
"""
planned = seeded_conclusions(fixture)
if not planned:
log("fixture declares no conclusions to seed")
return
seeded_total = 0
for (observer_id, observed_id), by_level in planned.items():
stored = list(peers[observer_id].conclusions_of(observed_id).list(size=100))
for level, expected in by_level.items():
if not expected:
continue
actual = {c.content for c in stored if c.level == level}
missing = [content for content in expected if content not in actual]
if missing:
raise SeedError(
f"{observer_id} -> {observed_id}: {len(missing)} seeded {level} "
f"conclusion(s) missing, first is {missing[0]!r}. Honcho collapses "
"conclusions whose content matches something already stored, so "
"check for near-duplicate text in the fixture."
)
seeded_total += len(expected)
summary = " ".join(f"{level}={len(by_level[level])}" for level in LEVELS)
log(f"verified seeded {observer_id} -> {observed_id}: {summary}")
# Derived and seeded conclusions must both be present. The seed phase already
# proved derivation ran against an un-injected database; this catches the reverse
# error of an injection that somehow replaced the derived rows.
if total <= seeded_total:
raise SeedError(
f"found {total} conclusions but {seeded_total} were seeded, leaving none "
"derived. Injection should add to the deriver's output, not replace it."
)
log(
f"verified {total - seeded_total} derived conclusions alongside {seeded_total} seeded"
)
# Premise text is what the working representation renders for a derived
# conclusion, and it is the only part of the reasoning tree an API client can
# see - schemas.Conclusion does not expose source_ids.
by_peer = {spec["id"]: spec for spec in fixture["peers"]}
for (observer_id, observed_id), by_level in planned.items():
premise_texts = cited_premise_texts(by_peer[observed_id], by_level["explicit"])
if not premise_texts:
continue
rendered = peers[observer_id].representation(
target=observed_id, max_conclusions=100
)
absent = [text for text in premise_texts if text not in rendered]
if absent:
raise SeedError(
f"{observer_id} -> {observed_id}: premise text missing from the "
f"representation, first is {absent[0]!r}. Premises render only for "
"deductive conclusions and sources only for inductive ones, so a "
"premise stored under the wrong metadata key renders as nothing."
)
log(
f"verified premise text renders in {observer_id}'s representation of {observed_id}"
)
if __name__ == "__main__":
try: