notify_session_compacted closed the old session scope immediately on a
legacy rotating compaction. A compaction can complete while a turn is
still live on the old session; closing then pops the session scope under
the live turn scope, violating the stack's LIFO order — the exact
invariant the rest of the segmentation feature protects.
Now: when the old session has an active turn, set close_pending instead;
that turn's end_turn consumes the flag after its own turn scope pops and
it unregisters from the active-turn table. Sabotage-verified: the new
test fails without the fix.
Continuous gateway sessions keep the Relay session scope open for days;
close-driven export means the session root span and out-of-turn marks
never export until /new or idle-end, and a crash loses the open segment
entirely.
Opt-in segmentation (both defaults OFF => scope lifecycle byte-identical
to today):
gateway.telemetry.session_segments.on_compaction: false
gateway.telemetry.session_segments.max_turns: 0
Rotation closes the current session scope and pushes the next segment
(same session_id attribute, plus hermes.session.segment=N and
segment_reason=compaction|max_turns) ONLY at a turn boundary in
begin_turn — never mid-turn (scope stack is LIFO). Compaction completion
just flags rotate_pending (observer semantics, nothing on the compaction
critical path); legacy rotating compaction closes the orphaned old
session scope so its segment exports. Both native calls ride the
existing bounded scope-op executor: a wedged rotation costs one segment
span, never the agent. Segment bookkeeping advances even on native
failure so a degraded rotation cannot retry every turn.
CI caught the file hanging AFTER '6 passed in 4.32s' until the runner's
300s SIGKILL. Two defects, same class the PR fixes:
1. The executor-refused (interpreter shutdown) fallback ran the native
call UNBOUNDED on the calling thread — a wedged pipeline would block
process exit forever. Now runs on a bounded daemon exit-thread with
the same timeout/abandon semantics as the executor lane.
2. The wedge tests left daemon workers parked on Event.wait() and live
sessions registered on the atexit shutdown hook; exit re-ran the
wedged pops (bounded, 10s each) and the per-file runner timed out.
Autouse teardown now releases every wedge and drains each runtime.
Canonical runner: 4.4s (was 300s file-timeout kill). Bare pytest was a
false green for this class — it exits before atexit replay cost shows.
The NeMo Relay native binding's scope.pop/push are synchronous and
unbounded ('returns after the scope is closed successfully'). When the
native pipeline cannot make progress, the session coordinator's turn and
session finalization block forever inside run_conversation: delegated
children finish their turns but never return, and delegation batches die
on the stall watchdog. Proven live 2026-08-10 on the staging fleet — a
falsification probe (plugin disabled, identical config) completed the
same delegation batch that wedged with the plugin active.
Bound every scope lifecycle operation that gates turn/session completion
(session push, turn push, turn pop, logical-LLM pops, session pop,
subscriber flush) by running the native call on a shared
DaemonThreadPoolExecutor and honoring a 10s result timeout. On breach a
TimeoutError propagates into each call site's existing exception
handling — warn, retain the unclosed-prefix diagnostics, continue — so
the worst case is one lost span, never a blocked agent. timeout=None
preserves byte-identical synchronous behavior for all other callers, and
interpreter-shutdown paths fall back to the synchronous call so the
atexit flush still exports.
Observability must never block the product.
The native Relay pipeline binds its Futures to the event loop that
entered run_in_session_async. While a managed tool callback executes,
that loop is blocked until the callback returns — so a nested managed
relay call made from inside the callback (vision_analyze's auxiliary
LLM call on a worker-thread loop) awaits a Future that can never
complete: 'RuntimeError: Future attached to a different loop', or a
deadlock, plus 'Event loop is closed' at shutdown when the orphaned
future completes late. (#77244)
Fix: managed_callback_guard, a ContextVar depth marker set around every
Hermes callback the relay adapters hand to the native pipeline
(relay_tools.execute invoke, relay_llm execute/execute_async invoke,
ManagedLlmStream run_callback). resolve_execution_context returns the
no-relay triple while the marker is set, so nested calls run unmanaged.
The marker propagates through contextvars.copy_context() into the
worker threads tools use for their internal async work.
Top-level turn LLM calls and tool wraps stay fully managed — verified
live: vision_analyze works under active shared metrics while the main
turn still records managed llm.execute events.
Alternative fixes considered and rejected: removing
retain_managed_execution (kills the shared-metrics managed pipeline)
and gating on main-thread identity (managed tool wraps legitimately
run on the run_agent thread, so that gate disables relay everywhere).
Teknium
Victor Kyriazakos
Bryan Bednarski
Alex Fournier
Jeffrey Quesnelle