CI slices failed the offload tests with 0.5s witness timeouts: on a
loaded shared runner the event loop thread can take >0.5s to get
scheduled even when NOT blocked, making the probe report a false
positive. A genuinely blocked loop can never set the progress event at
any timeout (the witness coroutine can't run at all), so 5s only
absorbs scheduler flake without weakening the invariant. Mutation
re-verified: reverting the offload still fails all 4 tests.
The sweep-path test parametrizes over _runner/_adapter, which live on
TestGatewayRedeliverySweep; main later added
TestUnconnectedPlatformKeepsItsBudget at the cherry-pick anchor point and
the test landed in that class, where the helpers don't exist
(AttributeError x2). Placement-only move.
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
The delivery ledger durably records a final response before the send so a
crash between finalize and platform ACK can redeliver it on the next boot.
attempts is that redelivery budget, capped at MAX_ATTEMPTS=3.
sweep_recoverable() claims every dead-owner row and increments attempts
before the caller knows whether it can send. self.adapters only holds a
platform after its connect() succeeded, so when the platform failed to
connect this boot _redeliver_pending_obligations() hits its "adapter is
None" branch and continues WITHOUT sending — but the attempt is already
spent. Three such boots and the row abandons, having never been sent once.
That is the loss the ledger exists to prevent, and the trigger correlates
with the crash that created the obligation: the network trouble that killed
the send tends to still be there on the next boot. Worse, the message stays
lost — once abandoned it is never retried even after the platform recovers.
Reproduced against the real runner with an unconnected adapter:
boot 1: claimed=1 state='attempting' attempts=1 (0 sends attempted)
boot 2: claimed=1 state='attempting' attempts=2 (0 sends attempted)
boot 3: claimed=1 state='attempting' attempts=3 (0 sends attempted)
boot 4: claimed=0 state='abandoned' attempts=3 (0 sends attempted)
Let the caller declare which platforms it can send on, and skip claiming
rows for the others. attempts then only ever buys a real send. Rows for a
platform that never returns are still bounded by the stale cutoff, so
nothing accumulates. The parameter is keyword-only and optional — omitting
it keeps the previous claim-everything behaviour for other callers.
A final response generated but not confirmed-delivered was the one
artifact the gateway could lose without a trace: crash or planned
restart between finalize and platform ACK dropped it silently, and the
resume path re-ran the whole turn at full cost (#58818 P1, #41696,
#63695's gateway half).
gateway/delivery_ledger.py records each outbound final response in
state.db (same conventions as the async-delegation ledger: WAL, owner
pid + process-start-time liveness, bounded retention):
pending -> attempting -> delivered | failed
startup sweep on dead-owner rows -> redeliver | abandoned
Contract (the lessons from the closed delivery-outbox attempt #61790):
- obligation recorded BEFORE the first send attempt; cleared only on
SendResult.success (destination acceptance, #51184)
- ambiguity is labeled, never silently retried: rows that were mid-send
when the process died redeliver with a visible '♻️ Recovered reply —
may be a duplicate' prefix (honest at-least-once)
- stable ids from session_key + inbound message id + content, so
distinct threads/topics can never collide
- poison rows bounded: 3 attempts / 24h freshness -> abandoned; claim
atomically re-stamps ownership so racing sweeps can't double-claim
- redelivery clears resume_pending for the session so the resume path
never re-runs a turn whose answer the ledger already holds
- best-effort everywhere: ledger failure can never block or delay a send
- slash-command/ephemeral/empty responses are not recorded; cron and
proactive delivery stay on DeliveryRouter (separate subsystem)
Config: gateway.delivery_ledger (default on; no version bump needed).
Validation: 30 ledger+producer tests; 352 blast-radius gateway tests
green; cross-process E2E (record in process A, kill it mid-send, claim
+ marker + redeliver in a fresh process B against the same state.db).
kshitijk4poor
ibaldr89
Teknium
Drexuxux