* fix(tests): repair unsatisfiable unified-test assertions and surface traces
Five of the eight persistent `unified-tests` failures assert things the code
cannot produce. None are regressions.
Raise the queue-drain timeout to 600s on the three large longmem fixtures.
They ingest 484-550 messages across ~50 sessions, then wait on the 60s
`WaitAction` default; the deriver is still working normally when the timer
fires. Matches the sibling 550-message case that already passes.
Raise `max_tokens` to 2500 in the two config-summary fixtures. Context
allocates 40% of the limit to the summary, so the previous 400 gave a
160-token budget while `SUMMARY.MAX_TOKENS_SHORT` is 1000 — no conforming
summary could ever fit, and the query returned `summary=None` even though the
summary was created.
Drop `session_id` from the dream test's `get_representation` step. A bare
session id becomes a one-element allowlist, and an allowlist narrows levels to
`ALLOWLIST_SAFE_LEVELS` (`explicit`), so the deductive and inductive
observations the step asserts on are excluded by design. The unscoped
representation is where the dreamer's conclusions are actually served.
Delete `WaitAction.flush`. Flush is process-wide — the harness starts the
deriver with `DERIVER_FLUSH_ENABLED=true` — and there is no per-request flush,
so the field never had an effect despite being set in 47 places. `TestStep`
now forbids extra fields so a dead knob cannot silently accumulate again.
Presign the reasoning traces alongside `results.json` and report both to the
Discord webhook and a GitHub job summary. The traces hold the full prompts and
model outputs and were already uploaded, but only `results.json` was surfaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): record why a unified test failed, not just that it did
`results.json` carried only name, status and duration, so a red run said
which test failed and nothing about why. The reason existed solely in the job
log, where the secrets action's masking can render it unreadable — diagnosing
a failure meant re-reading GHA logs that had digits redacted out of them.
`execute` now returns the `StepFailure` that stopped the test (step index,
step type, and the exception message) instead of a bare bool. Assertion
failures already raised useful text, including the LLM judge's own reasoning;
that text now reaches `results.json`, the console output, the job summary and
the Discord message rather than being discarded at the call site.
`results` moves from a `(status, duration)` tuple to a `TestOutcome` with
named fields so the failure can ride along.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): keep the Discord report inside the webhook size limit
The failure reasons added to the Discord message pushed it past Discord's
2000-character content limit, and the webhook answered 400 — run
33779689337 sent no notification at all. Six LLM-judge verdicts run to
~2760 characters; capping the count at ten did nothing because the length
was never the count.
Reasons are now clipped per line for Discord only; the job summary, the
console and results.json keep them whole. `send_discord_message` also clamps
the assembled content, so an over-long report loses its tail rather than the
entire notification.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): keep the Discord report short and link to the Actions run
The Discord message restated every failure, which pushed it past Discord's
2000-character limit and returned a 400 — run 33779689337 sent no
notification at all.
The report is now the headline, the results link, an Actions run link, and
the traces S3 key. Per-test failure reasons stay in the job summary that the
Actions link points at, along with both presigned URLs, so nothing is lost by
not repeating them in chat.
Restating failures was not the only size risk. A presigned URL carries an
OIDC session token and can run past a thousand characters by itself, so two
of them exceeded the limit unaided — which is why the traces go in as their
S3 key, the `aws s3 cp` path, at ~90 characters instead of ~1500.
`clamp_lines` drops whole lines rather than characters, since half a
presigned URL is useless and renders as broken markdown, and drops the
longest line first so an overlong URL cannot evict the short Actions link
that leads to everything else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(api): say when a session summary is dropped for budget
`get_context` allocates 40% of the token limit to the summary, but that limit
is what remains *after* the peer representation and peer card are subtracted,
not the `tokens` the caller asked for. When nothing fits, the caller receives
`summary: null` — indistinguishable from a session that has no summary — and
the only trace was a debug line in a different module.
`_select_summary_for_context` now logs at info when summaries exist and none
was chosen, with the budget and the sizes that missed it.
The two `config_summary_control` fixtures go to 4000. Measured against CI run
33779689337, their 12 messages produce 12 explicit observations costing ~1176
tokens, so the original `max_tokens: 400` left a budget of -776: no summary of
any size could have been served, and the earlier reading of this failure — a
160-token budget against a 388-token summary — had the mechanism wrong. 2500
was also short, leaving 529 against a `SUMMARY.MAX_TOKENS_SHORT` of 1000; 3676
is the minimum that guarantees a conforming summary fits.
Tests cover the budget arithmetic at each of those limits, the new log line,
and that a stored summary is served through the route with and without an
observer — the retrieval path itself was never at fault and had no coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(api): report a dropped summary on the path get_context actually takes
The previous commit added this log to `_select_summary_for_context`, which
only runs when `get_context` is given a `peer_target`. The unified
`config_summary` fixtures set `observer_peer_id`, but the runner does not
forward it, so those requests take `summarizer.get_session_context` instead —
where the same outcome was reported at debug and stayed invisible.
That also retracts the representation-budget explanation for those fixtures.
Nothing is subtracted from the limit on this path: the summary gets 40% of the
requested tokens outright, so at `max_tokens: 4000` a 99-token summary has a
1600-token budget and fits comfortably. The reason it is still absent is not
the budget, and the log now says so on the right path.
Tests cover both paths, and record that the fixtures exercise the one without
a representation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): drop the ignored observer from the config_summary fixtures
`observer_peer_id` has no effect on a `get_context` step — the runner does not
forward it — so it read as scoping a request that was never scoped. The step
description now records that these are unscoped reads and what naming an
observer would change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
9677f3d80c