* fix(tests): repair unsatisfiable unified-test assertions and surface traces
Five of the eight persistent `unified-tests` failures assert things the code
cannot produce. None are regressions.
Raise the queue-drain timeout to 600s on the three large longmem fixtures.
They ingest 484-550 messages across ~50 sessions, then wait on the 60s
`WaitAction` default; the deriver is still working normally when the timer
fires. Matches the sibling 550-message case that already passes.
Raise `max_tokens` to 2500 in the two config-summary fixtures. Context
allocates 40% of the limit to the summary, so the previous 400 gave a
160-token budget while `SUMMARY.MAX_TOKENS_SHORT` is 1000 — no conforming
summary could ever fit, and the query returned `summary=None` even though the
summary was created.
Drop `session_id` from the dream test's `get_representation` step. A bare
session id becomes a one-element allowlist, and an allowlist narrows levels to
`ALLOWLIST_SAFE_LEVELS` (`explicit`), so the deductive and inductive
observations the step asserts on are excluded by design. The unscoped
representation is where the dreamer's conclusions are actually served.
Delete `WaitAction.flush`. Flush is process-wide — the harness starts the
deriver with `DERIVER_FLUSH_ENABLED=true` — and there is no per-request flush,
so the field never had an effect despite being set in 47 places. `TestStep`
now forbids extra fields so a dead knob cannot silently accumulate again.
Presign the reasoning traces alongside `results.json` and report both to the
Discord webhook and a GitHub job summary. The traces hold the full prompts and
model outputs and were already uploaded, but only `results.json` was surfaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): record why a unified test failed, not just that it did
`results.json` carried only name, status and duration, so a red run said
which test failed and nothing about why. The reason existed solely in the job
log, where the secrets action's masking can render it unreadable — diagnosing
a failure meant re-reading GHA logs that had digits redacted out of them.
`execute` now returns the `StepFailure` that stopped the test (step index,
step type, and the exception message) instead of a bare bool. Assertion
failures already raised useful text, including the LLM judge's own reasoning;
that text now reaches `results.json`, the console output, the job summary and
the Discord message rather than being discarded at the call site.
`results` moves from a `(status, duration)` tuple to a `TestOutcome` with
named fields so the failure can ride along.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): keep the Discord report inside the webhook size limit
The failure reasons added to the Discord message pushed it past Discord's
2000-character content limit, and the webhook answered 400 — run
33779689337 sent no notification at all. Six LLM-judge verdicts run to
~2760 characters; capping the count at ten did nothing because the length
was never the count.
Reasons are now clipped per line for Discord only; the job summary, the
console and results.json keep them whole. `send_discord_message` also clamps
the assembled content, so an over-long report loses its tail rather than the
entire notification.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): keep the Discord report short and link to the Actions run
The Discord message restated every failure, which pushed it past Discord's
2000-character limit and returned a 400 — run 33779689337 sent no
notification at all.
The report is now the headline, the results link, an Actions run link, and
the traces S3 key. Per-test failure reasons stay in the job summary that the
Actions link points at, along with both presigned URLs, so nothing is lost by
not repeating them in chat.
Restating failures was not the only size risk. A presigned URL carries an
OIDC session token and can run past a thousand characters by itself, so two
of them exceeded the limit unaided — which is why the traces go in as their
S3 key, the `aws s3 cp` path, at ~90 characters instead of ~1500.
`clamp_lines` drops whole lines rather than characters, since half a
presigned URL is useless and renders as broken markdown, and drops the
longest line first so an overlong URL cannot evict the short Actions link
that leads to everything else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(api): say when a session summary is dropped for budget
`get_context` allocates 40% of the token limit to the summary, but that limit
is what remains *after* the peer representation and peer card are subtracted,
not the `tokens` the caller asked for. When nothing fits, the caller receives
`summary: null` — indistinguishable from a session that has no summary — and
the only trace was a debug line in a different module.
`_select_summary_for_context` now logs at info when summaries exist and none
was chosen, with the budget and the sizes that missed it.
The two `config_summary_control` fixtures go to 4000. Measured against CI run
33779689337, their 12 messages produce 12 explicit observations costing ~1176
tokens, so the original `max_tokens: 400` left a budget of -776: no summary of
any size could have been served, and the earlier reading of this failure — a
160-token budget against a 388-token summary — had the mechanism wrong. 2500
was also short, leaving 529 against a `SUMMARY.MAX_TOKENS_SHORT` of 1000; 3676
is the minimum that guarantees a conforming summary fits.
Tests cover the budget arithmetic at each of those limits, the new log line,
and that a stored summary is served through the route with and without an
observer — the retrieval path itself was never at fault and had no coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(api): report a dropped summary on the path get_context actually takes
The previous commit added this log to `_select_summary_for_context`, which
only runs when `get_context` is given a `peer_target`. The unified
`config_summary` fixtures set `observer_peer_id`, but the runner does not
forward it, so those requests take `summarizer.get_session_context` instead —
where the same outcome was reported at debug and stayed invisible.
That also retracts the representation-budget explanation for those fixtures.
Nothing is subtracted from the limit on this path: the summary gets 40% of the
requested tokens outright, so at `max_tokens: 4000` a 99-token summary has a
1600-token budget and fits comfortably. The reason it is still absent is not
the budget, and the log now says so on the right path.
Tests cover both paths, and record that the fixtures exercise the one without
a representation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): drop the ignored observer from the config_summary fixtures
`observer_peer_id` has no effect on a `get_context` step — the runner does not
forward it — so it read as scoping a request that was never scoped. The step
description now records that these are unscoped reads and what naming an
observer would change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Structured outputs for dialectic
* cleanup
* rename json_schema_to_pydantic to clarify it's not a general schema converter
* clean up schema DoS guards
* simplification and cleanup of schema conversion
* chore: ruff and pyproject toml
* chore: basedpyright cleanup in test
* fix: some needed unrelated test failures
* test(schema_conversion-and-anthropic-backend): expand test coverage
include table tests
* fix(llm): support combined tool calling and structured output across backends
- OpenAI: parse() 500s on non-strict function tools; route tool-carrying
structured requests through create() with an explicit json_schema
response_format (mirrors the streaming path)
- Anthropic: skip the '{' JSON prefill when tools are present so tool_use
blocks stay reachable; make the schema instruction conditional and rely
on parse + repair
- Gemini: native response_schema + function calling is rejected before
Gemini 3; with tools present, inject a schema instruction into the final
turn instead and rely on parse + repair
- All backends: tool-call turns carry no consumable content, so skip
structured-output parsing on them
Extracted from the dialectic structured-output branch (DEV-1652) so the
transport layer can land independently.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): exercise combined tools + structured output per provider
Two-turn live flow per backend: a forced tool-call turn (structured
parsing must be skipped) followed by a replay turn that must return a
schema-conforming answer with tools still attached. Asserts the
provider-specific request shaping: no parse() for OpenAI (500s on
non-strict tools), no '{' prefill for Anthropic, no native
response_schema for Gemini.
Verified against live OpenAI (gpt-4.1, gpt-5, gpt-5.4, gpt-5.4-mini)
and Gemini (gemini-2.5-flash).
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(unified): dialectic chat with response_format schema under tool use
Adds response_format pass-through to the unified runner's chat query and
a test case that forces the dialectic tool loop (reasoning off + global
enumeration question) while requiring a schema-conforming JSON answer —
end-to-end coverage of the combined tools + structured output transport
path on whichever provider each level is configured with.
Verified locally against a full harness run (json_match assertions pass;
the llm_judge assertion additionally runs in CI where the Anthropic key
is available).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: some needed unrelated test failures
* ci: add label-triggered live LLM test workflow
Adding the run-live-llm label to a PR (or workflow_dispatch) runs
tests/live_llm/ against real provider APIs — the only place the
--live-llm suite runs in CI. Reuses the unified-tests environment and
its Secrets Manager staging-dotenv resolution for provider keys; runs
on ubuntu-latest (no Fly runner, no Docker — the suite only touches the
LLM backends). Pins LIVE_LLM_ANTHROPIC_45_PLUS_MODELS=claude-sonnet-4-5
since the Anthropic family has no default models and would otherwise
silently collect empty.
Opt-in by design: live model behavior is variable, so this is a signal,
not a required check.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: run live LLM tests on main pushes touching the transport
Mirrors unified-tests' push trigger, scoped to paths that can affect
the live suite (src/llm/, config, the tests, deps, and the workflow
itself) so provider API calls aren't spent on unrelated changes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: disable auth in live LLM test environment
The staging dotenv sets AUTH_USE_AUTH=true without a usable JWT secret,
and src/config.py validates the pair at import time — the same reason
unified-tests overrides it. This suite never runs the API server.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(live_llm): fix gpt-5.4 reasoning_effort and gemini replay-turn flake
- test_live_openai: gpt-5.4 dropped 'minimal' from the reasoning_effort
vocabulary, so the gpt5 caching test 400'd — and the OpenAI backend's
BadRequestError terminal swallowed it into an empty CompletionResult.
Pick the effort per model generation.
- test_live_tools_structured_output: use tool_choice='auto' on the
replay turn, matching the production dialectic loop (which never
forces 'none') — NONE mode is what provoked gemini-2.5-flash's empty
candidates. Drop the temperature pin so retries actually resample,
and treat a repeat tool call as a retryable attempt.
Verified live: full suite green, gemini 4/4 consecutive passes.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: fail live LLM run when no staging secret was loaded
If the latest-tag fetch fails and no second tag exists, the fallback
step is skipped rather than failed, and the job would proceed without
provider keys — every test then skips via require_provider_key and the
run goes green. Guard on both fetch outcomes so that path fails loudly.
DEV-2035
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(live-llm-tests-GHA): remove extra comments
* feat(structured-output): enable non-recursive schema references
* docs(structured-outputs): clean up new doc
* test(structured-output): fix caching refs memory leak, add tests
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Eugene Eisenstein