806c2b1fdc
Merge updated client resource metrics into active-install metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-04 15:10:41 -07:00
44897dd6f0
Merge origin/main into feat/hermes-relay-client-dimensions
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-04 11:19:43 -07:00
5943bab1ec
Merge branch 'main' into feat/hermes-relay-model-metrics
2026-08-04 12:07:51 -04:00
80c7ccf4a6
fix(relay): gate skipped task completion
...
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
2026-08-04 09:45:46 -06:00
e1caa611bf
fix(relay): preserve skipped turn context
...
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
2026-08-03 13:43:17 -06:00
2e65b0c604
test(relay): enforce LIFO in overlap regression
...
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
2026-08-03 13:43:17 -06:00
a2a08fe147
fix(relay): gate skipped turn metrics
...
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
2026-08-03 13:43:17 -06:00
9a9b670e29
fix(relay): avoid concurrent turn scope corruption
...
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
2026-08-03 13:43:16 -06:00
07ac9ca687
Merge updated client resource metrics into active-install metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
# Conflicts:
# hermes_cli/observability/schemas/hermes.shared_metrics.v1.schema.json
# hermes_cli/observability/shared_metrics_contract.py
# hermes_cli/observability/shared_metrics_subscriber.py
# scripts/smoke_nemo_relay_shared_metrics.py
# tests/hermes_cli/test_relay_shared_metrics.py
# tests/hermes_cli/test_relay_shared_metrics_runtime.py
2026-07-31 07:43:11 -07:00
3d5fcab70f
Merge updated tool metrics into skill metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
# Conflicts:
# hermes_cli/observability/schemas/hermes.shared_metrics.v1.schema.json
# scripts/smoke_nemo_relay_shared_metrics.py
# tests/agent/test_skill_commands.py
# tests/hermes_cli/test_relay_shared_metrics.py
# tests/hermes_cli/test_relay_shared_metrics_runtime.py
# tests/tools/test_skill_manager_tool.py
# tests/tools/test_skill_usage.py
# tests/tools/test_skills_tool.py
2026-07-31 07:30:30 -07:00
aef76ba398
Merge updated model metrics into tool metrics
...
# Conflicts:
# hermes_cli/observability/shared_metrics_contract.py
# hermes_cli/observability/shared_metrics_subscriber.py
# scripts/smoke_nemo_relay_shared_metrics.py
# tests/hermes_cli/test_plugins.py
# tests/hermes_cli/test_relay_shared_metrics.py
# tests/hermes_cli/test_relay_shared_metrics_runtime.py
# tests/run_agent/test_run_agent.py
# tests/test_model_tools.py
# tests/tools/test_approval.py
2026-07-31 07:18:02 -07:00
c0369f0891
Merge remote-tracking branch 'origin/main' into feat/hermes-relay-model-metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-30 11:12:51 -07:00
dfb8c1bd4d
fix(observability): preserve shared metrics compatibility
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 14:09:47 -07:00
39975613b1
test: prune wave 2 + speed fixes — 28,106 → 19,757 test functions, suite wall 315s → 294s
...
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
2026-07-29 13:39:40 -07:00
6b81590c55
test: prune low-value tests suite-wide (wave 1) — 46,820 → 28,106 test functions
...
Systematic prune per AGENTS.md test policy, one pass over every major
test tree (gateway, hermes_cli, tools, agent, run_agent, plugins, cli,
cron, tui_gateway, honcho/openviking, root-level):
- DELETE: source-reading tests (read_text/getsource on prod files),
change-detector tests (exact catalog counts, model-name snapshots,
config version literals), mock-echo tests (assert a mock returns what
it was told), assertion-free/trivial tests, near-duplicate
parametrizations (boundaries + one representative kept), async/sync
twin duplicates, cosmetic within-file variations.
- KEEP (mandatory): security/redaction/approval guards, message-role
alternation invariants, prompt-caching/deterministic-call-id
invariants, issue-number regression tests (deduped), E2E tests.
- 6 test files deleted outright (script-style/no-assert or fully
redundant); conftest.py, fakes/, fixtures/ untouched.
- tests/acp/conftest.py added: autouse fixture stubs the live
models.dev/GitHub/Copilot/Anthropic inventory fetches that ACP server
tests performed on every session create — test_server.py 147s → 3.4s,
and the tests are now genuinely hermetic.
- Sleep-based slowness shrunk where safe (codex_ttfb_watchdog,
compression_concurrent_fork, etc.); no wall-clock assertion tightened.
Verification: full hermetic suite via scripts/run_tests.sh —
2439 files, 31,130 tests passed, 0 failed, 0 flaky retries, 315s wall
(baseline: 583s wall, 13,564s subprocess CPU).
2026-07-29 13:10:23 -07:00
46bae75042
feat(observability): add Relay active install metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 12:20:34 -07:00
f1fd678e44
feat(observability): add Relay skill metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 12:20:34 -07:00
8502e464a8
fix(observability): harden tool lifecycle metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 12:20:17 -07:00
4ad78a98fb
fix(observability): derive tool metrics from runtime metadata
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-29 11:49:45 -07:00
44d5a2df5a
test: point relay-metrics mocks at read_raw_config_readonly
...
enabled() now reads via read_raw_config_readonly(); the 7 monkeypatch/
patch sites in test_relay_shared_metrics_runtime.py that stubbed
hermes_cli.config.read_raw_config no longer intercepted the read,
failing 18 tests on CI slice 7/8. Repro'd locally, retargeted the
mocks; 147 passed + 2 skipped across both relay metrics files.
2026-07-29 11:33:41 -07:00
8b0c3da8c0
feat(observability): aggregate bounded tool metrics
2026-07-29 11:25:55 -07:00
dc4714b1e0
feat(observability): report model and provider usage
2026-07-29 11:24:13 -07:00
1c582c4c4a
fix(observability): gate export on subscriber flush
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 12:08:10 -07:00
2aa34c5484
fix(observability): export shared metrics after tasks
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 11:06:14 -07:00
3b55419cb1
fix(relay): close logical calls in stack order
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-28 10:19:53 -07:00
14bed44c8c
Reapply "feat(observability): integrate NeMo Relay runtime and shared metrics"
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 21:10:51 -07:00
841a5a744a
Revert "feat(observability): integrate NeMo Relay runtime and shared metrics"
2026-07-27 22:28:08 -04:00
8699ff58ed
fix(relay): close child sessions inside turn ownership
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:20:05 -07:00
bc597571fe
perf(relay): bypass inactive execution adapters
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:14:50 -07:00
502e4d63a2
perf(relay): avoid disabled mark initialization
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-23 14:31:12 -07:00
43d994986e
fix(telemetry): keep consent profile-owned
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-23 14:04:07 -07:00
4788994bd2
fix(relay): preserve managed callback context
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-23 13:53:59 -07:00
ea171c947a
perf(relay): bypass inactive tool interception
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-23 13:38:49 -07:00
591ba267d2
fix(relay): release session lock before callbacks
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-23 13:33:41 -07:00
a6fbff1d7e
fix(relay): retain active child sessions
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-23 13:22:50 -07:00
9baa8cc96c
test(observability): exercise shared metrics with native Relay
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 09:00:50 -04:00
55b7903cf7
fix(observability): harden Relay terminal lifecycle metrics
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:59:42 -04:00
4dedaa4237
refactor(runtime): consolidate Relay lifecycle ownership
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:59:06 -04:00
9bc521b138
refactor(runtime): manage Relay LLM tool and subagent execution
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:57:38 -04:00
056e7df0e0
fix(observability): harden Relay metrics isolation and aggregation
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:55:07 -04:00
64faff6768
refactor(observability): move Relay runtime ownership into core
...
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:54:49 -04:00
Alex Fournier
Jeffrey Quesnelle
Bryan Bednarski
Teknium
Jeffrey Quesnelle