CI caught the file hanging AFTER '6 passed in 4.32s' until the runner's
300s SIGKILL. Two defects, same class the PR fixes:
1. The executor-refused (interpreter shutdown) fallback ran the native
call UNBOUNDED on the calling thread — a wedged pipeline would block
process exit forever. Now runs on a bounded daemon exit-thread with
the same timeout/abandon semantics as the executor lane.
2. The wedge tests left daemon workers parked on Event.wait() and live
sessions registered on the atexit shutdown hook; exit re-ran the
wedged pops (bounded, 10s each) and the per-file runner timed out.
Autouse teardown now releases every wedge and drains each runtime.
Canonical runner: 4.4s (was 300s file-timeout kill). Bare pytest was a
false green for this class — it exits before atexit replay cost shows.
The NeMo Relay native binding's scope.pop/push are synchronous and
unbounded ('returns after the scope is closed successfully'). When the
native pipeline cannot make progress, the session coordinator's turn and
session finalization block forever inside run_conversation: delegated
children finish their turns but never return, and delegation batches die
on the stall watchdog. Proven live 2026-08-10 on the staging fleet — a
falsification probe (plugin disabled, identical config) completed the
same delegation batch that wedged with the plugin active.
Bound every scope lifecycle operation that gates turn/session completion
(session push, turn push, turn pop, logical-LLM pops, session pop,
subscriber flush) by running the native call on a shared
DaemonThreadPoolExecutor and honoring a 10s result timeout. On breach a
TimeoutError propagates into each call site's existing exception
handling — warn, retain the unclosed-prefix diagnostics, continue — so
the worst case is one lost span, never a blocked agent. timeout=None
preserves byte-identical synchronous behavior for all other callers, and
interpreter-shutdown paths fall back to the synchronous call so the
atexit flush still exports.
Observability must never block the product.
Victor Kyriazakos