RomanNum3ral
/
honcho
mirror of https://github.com/plastic-labs/honcho.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

test(unified): add deriver_no_memory_signal naturalistic suite 

L1 reproducer for the deriver_no_memory_signal failure mode: the
deriver writes observations that don't add signal about the user or
their work -- bare acknowledgments paraphrased as facts ("user said
yes"), shell-history fragments ("user executed cd ~/foo"), instruction-
event-records that look like durable user attributes but aren't.

Inputs: 8 sessions sampled from SALT-NLP/SWE-chat (ODC-BY,
https://arxiv.org/abs/2604.20779), persona-balanced across Expert
Nitpicker / Vague Requester / Mind Changer. Two sessions from the
underlying extract are excluded for data quality (one Japanese with
a heavily skewed user/assistant role split; one near-monologue);
documented in the fixture description.

Per session: ingest messages, wait for queue empty, run an llm_judge
asking whether the user's representation contains any Tier 1 bare-
speech-act observation, then save_artifact-dump the representation
for offline analysis. The suite uses continue_on_failure to aggregate
per-session pass/fail. Judge prompt is calibrated with annotated
example observations across three tiers (Tier 1 / Tier 2 / Signal) to
distinguish bare-event records from specifics-anchored observations.

Inputs file at tests/unified/test_cases/data/swe_chat_sample_persona_balanced.json
-- the committed dataset extract. Metadata embedded in the file
documents its provenance (SALT-NLP/SWE-chat, ODC-BY, selection
criteria, extraction timestamp).
This commit is contained in:
thrialectics 2026-05-11 13:16:52 -04:00
parent d7c7568ceb
commit a59d69a9da
2 changed files with 1408 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

985
tests/unified/test_cases/data/swe_chat_sample_persona_balanced.json Normal file
View File

File diff suppressed because one or more lines are too long

423
tests/unified/test_cases/deriver_no_memory_signal.json Normal file
View File

File diff suppressed because one or more lines are too long