Commit Graph

81 Commits

Author SHA1 Message Date
Teknium aac74be2f1 fix(approval): classify CLI/TUI approval timeouts separately from explicit denials
When an approval prompt expired without a response, every CLI-side path
collapsed the timeout into the same 'deny' choice as an explicit user
refusal, so the agent was told the user denied the action when the user
simply never answered. The gateway wait already distinguished the two
('timed out without user response... Silence is not consent.'); this
brings the CLI/TUI/ACP surfaces to parity.

- prompt_dangerous_approval(): input()-path expiry now returns a distinct
  'timeout' choice (still fail-closed).
- cli.py _approval_callback + hermes_cli/callbacks.py approval_callback:
  deadline expiry returns 'timeout' instead of 'deny'.
- check_all_command_guards / _run_approval_gate CLI tails: 'timeout' maps
  to outcome='timeout' with a 'timed out without user response... Silence
  is not consent.' BLOCKED message (matching the gateway wording);
  explicit deny keeps outcome='denied' and gains user_consent=False for
  shape parity.
- computer_use: 'timeout' verdict threads through the CLI adapter and
  yields a 'prompt timed out — the user did not respond' error instead of
  'denied by user'.
- ACP permissions bridge: FutureTimeout returns 'timeout' (other failures
  still 'deny'); elicitation maps 'timeout' to 'cancel' like the gateway's
  unresolved outcome; codex wire mapping documents deny/timeout→decline.
- write_approval already treats unknown choices as 'stage, not drop', so
  a timeout now stages the memory write instead of silently refusing it.

Every timeout path remains fail-closed — the action never runs; only the
classification reported to the agent changes.
2026-08-02 20:21:59 -07:00
Teknium 8714040954 fix(computer_use): resolve gateway session-key namespace in permission-mode lookup
Follow-up to the #68246 salvage. The backend permission-mode resolution
only checked the DB session_id the tool path passes, but gateway /yolo
keys approval bypass off the gateway session_key (contextvar). Consult
both namespaces so /yolo works on messaging platforms, not just CLI/TUI.
Adds a regression test driving the real approval contextvar + yolo
toggle path E2E.
2026-07-29 12:19:37 -07:00
Francesco Bonacci c268397752 feat(computer_use): align cua-driver 0.10 permission modes 2026-07-29 12:19:37 -07:00
Francesco Bonacci 847e401b74 feat(computer_use): align cua-driver 0.9 contracts
Salvaged from PR #67807 by @f-trycua onto current main.

- Foreground gate: discover delivery_mode support from the live tools/list
  inputSchema.properties (fail closed), not the never-shipped
  input.delivery_mode capability token
- bring_to_front: standalone strict-schema MCP tool (inject_session=False),
  separate approval scope, requires foreground
- Verdict precedence: confirmed > unverifiable (verify before retry) >
  suspected_noop/refusal (escalate); surfaced as explicit verdict field
- Typed cua_browser_* route inside computer_use (browser_route.py) with
  exact-binding, adapter-injected session, snapshot-scoped refs
- Per-Hermes-session backend isolation + release_computer_use_session seam
  wired into AIAgent.close()
- Recorded 0.9 tools/list fixture replaces fabricated capability tokens
2026-07-29 12:19:37 -07:00
teknium1 5b751dc0ad chore: remove unused imports and dead locals (ruff F401/F841 sweep)
Cleans F401 unused imports and F841 dead local assignments across
root *.py, agent/, hermes_cli/, tools/, gateway/, cron/, tui_gateway/
(tests/, plugins/, skills/ excluded).

Intentionally KEPT (false positives / test-patch surfaces):
- agent/transports/__init__.py package re-exports
- cli.py browser_connect re-exports (DEFAULT_BROWSER_CDP_URL area,
  used by tests/cli/test_cli_browser_connect.py)
- hermes_cli/main.py _prompt_auth_credentials_choice /
  _model_flow_bedrock_api_key (accessed via main_mod attr in tests)
- gateway/run.py aliased replay_cleanup + whatsapp_identity re-exports
  and _PORT_BINDING_PLATFORM_VALUES (test-referenced)
- hermes_cli/web_server.py get_running_pid (tests monkeypatch it) and
  _OAUTH_TOKEN_URL availability probe
- hermes_cli/config.py get_process_hermes_home re-export (noqa'd F811
  chain) and yaml availability-probe import
- hermes_cli/nous_subscription.py managed_nous_tools_enabled
  (tests patch hermes_cli.nous_subscription.managed_nous_tools_enabled)
- try/except ImportError availability probes (env_loader, tts_tool,
  mcp_tool, web_server anthropic OAuth block)
- tools/web_tools.py noqa F401 re-exports
- hermes_cli/setup_whatsapp_cloud.py:263 'proceed' skipped: possible
  missing-guard bug, flagged for separate review
- unused function parameters (signature changes out of scope)

Side-effect RHS calls preserved where only the binding was dead
(e.g. web_server proc = _spawn_hermes_action -> bare call).
2026-07-29 11:53:39 -07:00
Teknium 16720dc45b fix(computer-use): hide the --no-overlay help probe console too
Follow-up to the #62821 salvage: the _cua_driver_supports_no_overlay
--help probe is another Windows-reachable spawn; give it the same
windows_hide_flags() treatment. Also adjust the status test to stub
_resolve_driver_cmd (permissions.py resolves via that helper, not
shutil.which).
2026-07-29 11:35:33 -07:00
motoblurr 7bc9956660 fix(computer-use): normalize Windows manifest paths in WSL
A Windows-installed cua-driver can return an absolute
``C:\Users\...\cua-driver.exe`` mcp_invocation.command to a Hermes
process running inside WSL. POSIX spawning can't use the raw Windows
string even though the binary is reachable through DrvFS. Translate
``<drive>:\...`` to ``/mnt/<drive>/...`` in _resolve_mcp_invocation
(before the path-separator check, since backslash is not a separator
on POSIX), only when actually running under WSL.

Salvaged from #63532 by @motoblurr (original commit carried a
placeholder 'Hermes Agent <hermes@local>' identity; re-authored).
Fixes #63938 premise.
2026-07-29 11:35:33 -07:00
ZundamonnoVRChatkaisetu 50e27abdd3 fix(computer-use): hide Windows cua-driver subprocess consoles
Apply windows_hide_flags() (CREATE_NO_WINDOW; 0 on POSIX) at the
Windows-reachable cua-driver subprocess boundaries: manifest probe,
update checker, CLI fallback transport, doctor health-report spawn,
and the permissions/status runner. Prevents OpenConsole/Windows
Terminal windows flashing into the foreground when spawned from
GUI-backed Gateway/Desktop processes.

The env-probe half of the original PR was already implemented on main
and is not re-applied here.

Salvaged from #62821 by @ZundamonnoVRChatkaisetu (original commits
carried a placeholder 'Claude Code Enterprise' identity; re-authored
to the contributor's GitHub identity).
2026-07-29 11:35:33 -07:00
monerostar b9215f5bc9 fix(computer_use): surface CLI --version when health_report version lies 2026-07-29 11:09:08 -07:00
camaleonidas c6db7b0f4f fix(computer-use): fallback doctor when cua-driver health_report is unclassified
cua-driver 0.10.0 marks health_report as risk.class=unclassified and denies
the MCP call with isError. Hermes doctor previously treated the bare
{exit_code:1} structuredContent as a real report and printed
"cua-driver ? on ? — ?" with exit 1.

Detect isError / non-schema payloads, raise HealthReportUnavailable, and
compose a schema_version=1 report from working probes (check_permissions,
list_apps, CLI --version/doctor). Prefer real health_report when present.

Tests cover unclassified denial, schema preference, and overall mapping.
2026-07-29 11:09:08 -07:00
Andrew Chen 0411869503 fix(computer-use): block destructive key combos in hyphen notation
`_canon_key_combo` (the `_BLOCKED_KEY_COMBOS` gate in
`handle_computer_use`) split key strings on `+` only, but the cua-driver
backend's `_parse_key_combo` splits on both `+` and `-`. So a model could
issue `{"action":"key","keys":"ctrl-alt-delete"}` (or `alt-f4`,
`cmd-shift-q`): the gate saw a single unknown token and let it through
while the backend executed the real destructive shortcut.

Split the gate on both `+` and `-` so it canonicalizes combos the same
way the backend does. Non-destructive hyphen combos (`cmd-c`) and the
literal `-` zoom key (`cmd+-`) are unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-29 11:09:08 -07:00
Koho Zheng f2a4ca9637 fix(computer-use): normalize cua-driver result envelopes
cua-driver 0.7.x can return list_windows/list_apps payloads under
structuredContent.windows, data.windows, data._legacy_windows, or
top-level windows/_legacy_windows (direct CLI responses). The wrapper
only read structuredContent.windows, so discovery came back empty
(capture 0x0, list_apps []) while raw cua-driver calls worked.

- add _windows_from_tool_result(): walks the known envelope shapes in
  priority order, skipping empty higher-priority envelopes
- route _load_windows() (MCP + CLI re-fetch paths) through the helper,
  covering capture() and focus_app()
- harden _ingest_windows(): skip non-dict members, normalize untrusted
  app_name/title/z_index fields
- list_apps(): prefer structuredContent.apps, fall through populated
  data/top-level envelopes, derive unique apps from window-shaped
  payloads via _apps_from_windows(), keep the text-line fallback last
- tests for every envelope shape, precedence, malformed records, and
  app derivation

Salvaged from #63037 (Reaper-Legion), which itself preserved the
original implementation from #57961 (kohoj); #73007 (umi008)
independently proposed the same normalization later.

Fixes #57905

Co-authored-by: Reaper <248977840+Reaper-Forge@users.noreply.github.com>
Co-authored-by: Ulises Millan Guerrero <ulises.millanguerrero@gmail.com>
2026-07-29 11:08:58 -07:00
Atakan 56bda4529b fix(computer_use): revive ended cua-driver sessions once 2026-07-28 14:24:30 -07:00
teknium1 920facdc4c fix(update): never blind-reinstall cua-driver during hermes update
'Refreshing cua-driver (Computer Use)...' could hang for minutes on
Windows: when the driver's native check-update verb returned an
indeterminate result (old driver without the verb, offline, GitHub
rate-limited, or the probe timing out), install_cua_driver(upgrade=True)
fell through to the full upstream installer — a silent, output-captured
run with a 660s ceiling, plus install.ps1's 600s concurrency-lock wait
on Windows on top. Every 'hermes update' paid that cost.

Two changes:

- install_cua_driver() grows require_confirmed_update: with it set, an
  indeterminate check keeps the installed version and returns fast,
  printing the force path (hermes computer-use install --upgrade).
  'hermes update' passes it; the explicit --upgrade CLI keeps the old
  fall-through so a force refresh still works when the check can't
  answer.
- cua_driver_update_check() default timeout is now 25s on Windows
  (8s unchanged on POSIX): first-spawn of the exe under Defender /
  SmartScreen routinely exceeds 8s, and a false timeout is exactly the
  indeterminate result that used to trigger the multi-minute reinstall.
2026-07-26 13:47:35 -07:00
Brooklyn Nicholson 4e49af94be fix(computer_use): stop the cua-driver child on exit
CuaDriverBackend caches a long-lived cua-driver subprocess for the life of
the Hermes process, and stop() was never called from anywhere — the driver
outlived the session that spawned it. #69903 stopped the orphan from pegging
a core by disabling the cursor overlay, but left the process behind; this is
item 3 of #28152 ("Hermes does not keep the driver alive after tool
completion").

Register an atexit hook, mirroring browser_tool's
atexit.register(_emergency_cleanup_all_sessions). atexit only, no signal
handlers, for the prompt_toolkit reason documented there. reset_backend_for_tests
now reuses the same teardown instead of repeating it.
2026-07-26 03:38:23 -05:00
teknium1 a5147331ea fix: repair sweep fallout — duplicate encoding kwargs, non-subprocess call sites, kwarg-snapshot tests
- Strip the salvaged commit's inline encoding kwargs where main had since
  gained its own (process_registry, local env, cua doctor, gateway,
  commands, gateway_windows — the latter keeps its locale-aware
  _schtasks_encoding() from #38186)
- Revert encoding kwargs mistakenly applied to non-subprocess APIs
  (exa get_contents, tempfile.mkstemp in webhook.py)
- Guard the ddgs worker Popen (new on main since #55339)
- Update two kwarg-snapshot test assertions for the new kwargs
2026-07-24 11:45:57 -07:00
teknium1 d4b867cf9f fix(windows): sweep remaining unguarded text-mode subprocess sites codebase-wide
AST-driven pass over every subprocess.run/Popen/check_output/check_call/call
with text=True (or universal_newlines=True) and no explicit encoding=:
append encoding='utf-8', errors='replace' at the kwarg site. 136 call
sites across 28 files (cli.py, hermes_cli/main.py, tools_config.py,
environments, computer_use, gateway, scripts, skills helpers, agent/*).

Together with the salvaged #55339/#60741 commits this closes out issue
#53428's bug class; the salvaged #60751 linter rule in
check-windows-footguns.py now enforces it repo-wide (verified: 807 files
scanned, zero findings).
2026-07-24 11:45:57 -07:00
Stoltemberg c89481db5e fix: add explicit UTF-8 encoding to all subprocess text=True calls (#53428)
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.

Fixes #53428 (master tracker for Windows GBK locale crash).

Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
2026-07-24 11:45:57 -07:00
Dickson Neoh 08298dabbd fix(computer-use): handle Linux cua window metadata
Treat cua-driver's Linux `is_on_screen: null` as unknown instead of
off-screen, and skip GNOME Shell desktop/backdrop helper windows
(ding "Desktop Icons", @!x,y;BDHF) when selecting the default capture
target — they are targetable X11 windows but capture as empty.

Reconciled with the _NET_ACTIVE_WINDOW fallback from #58030: helper
windows are filtered out of the candidate pool first, then the tied
z-order active-window probe runs on the remaining real app windows.
Also falls back to the requested app name for _last_app when Linux
windows carry no app_name.

Salvaged from #54173 by @dnth.
2026-07-23 08:11:58 -07:00
Brooklyn Nicholson cc1765dce2 Merge remote-tracking branch 'origin/main' into bb/computer-use-perf
# Conflicts:
#	hermes_cli/config.py
#	tools/computer_use/cua_backend.py
2026-07-23 01:39:21 -05:00
Brooklyn Nicholson 69b97a97f7 Merge remote-tracking branch 'origin/main' into bb/salvage-53841-no-overlay
# Conflicts:
#	tools/computer_use/cua_backend.py
2026-07-23 01:26:21 -05:00
Brooklyn Nicholson cdc123ec2f fix(computer_use): only disable agent cursor after session handshake
Guard the post-start set_agent_cursor_enabled on _session._started so
call_tool cannot re-enter session.start() (matches the start_session
lifecycle guard).
2026-07-23 01:10:21 -05:00
Brooklyn Nicholson 12ad13ddca perf(computer_use): cap capture size and cache vision routing
Cut steady-state Computer Use latency without changing default behavior
or waiting on cua-driver:

- Cap screenshots via set_config(max_image_dimension) on session start
  (config: computer_use.max_image_dimension, default 1456)
- Cache aux-vision routing per (provider, model) so captures skip
  repeated load_config()
- Add computer_use.capture_after_mode (default som) so users can opt
  follow-ups down to ax (elements only) for speed
2026-07-23 00:52:43 -05:00
Brooklyn Nicholson f957fe3760 fix(computer_use): default --no-overlay on macOS for idle CPU
Auto-detect now disables the cursor overlay on darwin as well as
headless/WSL2 Linux. After start_session, also call
set_agent_cursor_enabled(false) when the policy is on so older drivers
without --no-overlay still tear the overlay down.

Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
2026-07-23 00:45:26 -05:00
David Metcalfe 3d84689714 fix(computer_use): address sweeper feedback on --no-overlay subprocess + manifest probe
The hermes-sweeper review #4701565902 (2026-07-15) flagged two
consistency issues in `_cua_driver_supports_no_overlay` and one
additive-config concern:

1. `cua-backend.py:260` — the `cua-driver --help` support probe
   inherited the full parent environment. cua-driver is a third-party
   binary; every other spawn site in this file (manifest probe at
   `:214`, MCP spawn at `:697`, install probe at `:997`) uses
   `_sanitize_subprocess_env(cua_driver_child_env())`. The `--help`
   probe should match. This was a low-impact leak (only help output
   exits), but inconsistency is the wrong default for a third-party
   subprocess.

2. `cua_backend.py:238` — when the manifest returned a `command`
   different from the input `driver_cmd` parameter (e.g. a relocated
   executable at `/opt/relocated/cua-driver` while the system binary
   is at `/usr/bin/cua-driver`), the support probe ran against
   `_CUA_DRIVER_CMD` (the default) instead of the manifest-discovered
   `command`. Two failure modes:
   - The wrapper binary supports `--no-overlay` but the system binary
     doesn't → probe returns False → overlay kept despite capability.
   - The system binary supports `--no-overlay` but the wrapper doesn't
     → probe returns True → MCP spawn crashes on the unknown flag.

3. The original commit bumped `_config_version` 31→32 for an additive
   default (`computer_use.no_overlay: None`). AGENTS.md specifies that
   additive defaults in existing sections are handled by deep merge
   and should NOT trigger a version bump. After cherry-picking onto
   current `origin/main` (which is already at 33), the bump is
   effectively dropped — resolved to main's 33.

Changes:

- Add `env=_sanitize_subprocess_env(cua_driver_child_env())` to the
  `--help` subprocess (with the same import + rationale comment as
  the manifest probe).
- Pass `driver_cmd=command` (or `driver_cmd=driver_cmd` for the
  fallback path) into `_mcp_args_with_overlay_flag`, so the support
  probe runs against the binary that will actually be launched.

Tests (3 new):

- `test_help_probe_passes_sanitized_env` — verifies `subprocess.run`
  is called with an `env=` kwarg.
- `test_manifest_command_drives_support_probe` — verifies the probe
  runs against the manifest command when it differs from the input
  driver_cmd.
- `test_fallback_uses_input_driver_cmd_for_support_probe` — verifies
  the fallback path (no command in manifest) uses the input
  driver_cmd.
- `test_probe_distinguishes_support_between_binaries` — sanity check
  that the lru_cache key on `driver_cmd` prevents cross-binary
  cache leakage.

File-revert negative test confirmed all three of the new
"manifest/probe" tests are load-bearing: with the pre-fix code, they
fail (probe runs against the default binary instead of the resolved
one); with the fix, they pass. 20/20 tests in
`tests/computer_use/test_cua_no_overlay.py` green.
`TestMcpInvocationResolution` (8/8) still green.

Refs: sweeper review #4701565902
2026-07-23 00:44:43 -05:00
David Metcalfe f7a6c7a6e5 fix(computer_use): add explicit encoding to /proc/version open() 2026-07-23 00:44:43 -05:00
David Metcalfe 8d4f7a0002 fix(computer_use): refine auto-detect to headless/WSL2 only, add driver version probe
Address review feedback from cross-vendor review (Flash + GPT-OSS):

1. Auto-detect now checks for headless Linux (no DISPLAY), WSL2
   (/proc/version contains 'microsoft'), instead of all Linux.
   Desktop Linux with a compositor keeps the overlay.

2. Add _cua_driver_supports_no_overlay() that probes cua-driver --help
   to check if the flag is supported. Older drivers (< 0.6.x) reject
   unknown flags, so passing --no-overlay would crash the MCP spawn.

3. Update tests to cover headless vs desktop Linux, WSL2 detection,
   version probe, and the unsupported-driver fallback path.
2026-07-23 00:44:43 -05:00
David Metcalfe 8ceada6e30 fix(computer_use): pass --no-overlay to cua-driver on Linux/WSL2 to prevent idle CPU
cua-driver's cursor overlay rendering loop can consume CPU indefinitely
when idle (#28152, #47032). On Linux/WSL2, the overlay serves no visual
purpose and the rendering path is the primary source of idle CPU usage.

Add computer_use.no_overlay config option (default: auto-detect) that
passes --no-overlay to cua-driver when enabled. Auto-detection disables
the overlay on Linux (covers WSL2, headless, containers) where it has no
benefit, and keeps it enabled on macOS/Windows where it is visually
useful.

Refs: #28152, #47032
2026-07-23 00:44:43 -05:00
Tianqing Yun 2f9d88caee fix: resolve cua-driver across computer-use surfaces 2026-07-23 00:42:46 -05:00
Tianqing Yun 030822b68d fix: resolve cua-driver from user-local install paths 2026-07-23 00:42:46 -05:00
HexLab98 9881328651 fix(computer-use): skip X11 active-window probe for exact targets
Limit the _NET_ACTIVE_WINDOW xprop fallback to unqualified default
captures so exact pid/window_id targeting does not pay up to a 2s
subprocess probe on Linux/X11.
2026-07-22 21:15:18 -07:00
HexLab98 f320f3e5d7 fix(computer-use): prefer X11 active window when z_index ties
When Linux/X11 reports the same z_index for every on-screen window,
prefer _NET_ACTIVE_WINDOW via xprop instead of list order. Keep the
higher-z-index-is-frontmost contract when ordering is informative.
2026-07-22 21:15:18 -07:00
Teknium 7a43ab042f
fix(computer_use): reconnect a dead cua-driver session instead of hanging (#67138)
Bug 1 of #55048: when the MCP connection dropped (driver crash / restart),
_lifecycle_coro exited but left _started=True, so the next list_apps/capture
passed _require_started() and then operated on a None session — hanging
forever instead of reconnecting.

- _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead
  session is re-enterable (idempotent no-op on the normal stop() path; atomic
  bool write, safe from the bridge-loop thread without the lock stop() holds).
- call_tool() re-enters start() when the session isn't active, rebuilding it
  before the call. The start_session/end_session handshake (driven by start()/
  stop() themselves) is exempted so bootstrap doesn't recurse.

Tests: two cases in test_computer_use_delivery_ladder.py — finally resets
_started, and call_tool restarts a dead session exactly once. Full
computer_use suite green (233).

Refs #55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the
delivery_mode work in #67123.
2026-07-18 15:07:04 -07:00
Teknium 9d6d772837
feat(computer_use): follow cua-driver's verify → escalate ladder (#67123)
Hermes' computer_use wrapper dropped cua-driver's structured action verdicts,
exposed no delivery_mode, and injected background-only guidance — so the agent
reported unverified no-ops as success and concluded cua-driver 'cannot drive'
Electron/Chromium surfaces (observed live on tldraw offline). Fixes #67052.

Phase A — preserve the result contract:
- ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode
- CuaDriverBackend._action() reads structuredContent (was data-only); a helper
  normalizes it, additive and None-safe on old drivers
- _text_response surfaces the fields additively (ok stays transport-only)

Phase B — bounded, model-reachable foreground:
- delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher,
  ABC, and all input methods
- foreground is capability-gated (input.delivery_mode); old drivers get a
  structured foreground_unsupported refusal, never a silent background downgrade
- no automatic/hidden foreground retry — the model selects it from the signal

Phase C — guidance + isolation:
- system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from
  background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven
  by returned effect/escalation, not predicted from the app being Electron
- foreground approval scoped by (action, delivery_mode): a background approval
  never silently authorizes foreground
- approval state keyed per session_id so concurrent gateway runs don't leak unlocks

Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/
unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating +
foreground_unsupported, and session-scoped foreground approval. Existing 265
computer_use tests still green.

Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click
returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground
request returned code='foreground_unsupported' — correct on a driver that predates
the input.delivery_mode capability.
2026-07-18 13:59:35 -07:00
kshitij 2b6897f982
fix(computer-use): target Linux app windows reliably (#63725)
* fix(computer-use): target Linux app windows reliably

Resolve app filters through the canonical cua-driver MCP app metadata and join running app PIDs back to windows. Preserve an exact selected window across capture_after, support direct capture by pid/window_id, and send the active window ID for coordinate pointer actions on Linux.

Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>

* fix(computer-use): address review on PR #63725

Address three review comments from @f-trycua:

1. type_text, press_key, and hotkey now carry _active_window_id and
   fail closed when it is missing. Previously they sent only the PID,
   so CUA Driver fell back to the first window for that PID — input
   could reach the wrong window in multi-window apps.

2. Coordinate scroll x/y are now capability-gated behind
   input.scroll.coordinates. CUA Driver 0.7.1 Linux schema rejects
   x/y on scroll; omitting them when the driver doesn't advertise
   support avoids the schema rejection while still routing via
   window_id.

3. Windows are sorted by z_index descending (higher = front, per CUA
   Driver semantics) instead of ascending. Null z_index (Wayland) is
   coerced to 0 in _ingest_windows so it doesn't crash the sort and
   sorts to the back instead of being selected as the capture target.

---------

Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
2026-07-16 00:34:20 +05:30
Teknium 0823230545
fix(computer-use): sanitize env on the 4 remaining cua-driver spawn sites (#59165)
PR #58889 fixed the CLI-fallback transport; review of that fix found the
same leak class at four sibling spawn sites of the third-party cua-driver
binary:

- _resolve_mcp_invocation (cua-driver manifest): no env= at all — full
  parent environment inherited
- cua_driver_update_check (check-update --json): telemetry env but no
  secret sanitization
- doctor._drive_health_report (<binary> mcp Popen): telemetry env only
- permissions._run (every macOS/Linux permission probe): telemetry env only

All now route through _sanitize_subprocess_env(cua_driver_child_env()),
matching the sanctioned MCP spawn and the #53503/#55709/#58889 strip-by-
default policy for non-terminal spawns. Sanitization degrades gracefully
(falls back to the telemetry env) so doctor/permission probes never break
on an import error.

4 regression tests covering each site.
2026-07-05 14:48:20 -07:00
srojk34 f10851e3fd fix(computer-use): sanitize subprocess env in cua-driver CLI fallback transport
_CuaDriverSession._call_tool_via_cli() (the EAGAIN/silent-empty MCP
fallback transport) invokes `cua-driver call <tool> <json>` via
subprocess.run() with no env= argument, so the third-party cua-driver
binary inherits the full, unsanitized parent environment. The primary
MCP spawn site (_lifecycle_coro) already applies
_sanitize_subprocess_env(cua_driver_child_env()) before opening the
stdio client, per the same policy #53503/#55709 established for other
subprocess spawn points — this fallback path, added alongside the
EAGAIN/silent-empty-capture hardening, missed it.

Fix: apply the same env=_sanitize_subprocess_env(cua_driver_child_env())
to the subprocess.run() call in _call_tool_via_cli(), mirroring the
sanctioned spawn site exactly (telemetry policy applied first, then
Hermes-managed secrets filtered).
2026-07-05 13:57:20 -07:00
Adrian Lastra 519ec7b3b3 fix(computer_use): parse (label) and = "value" AX element label forms
The SOM/AX element list dropped labels for two extremely common cua-driver
render forms, leaving the model unable to target elements by name:
  - [79] AXButton (Dark)              -> parenthesised label
  - [4]  AXStaticText = "Wi-Fi"       -> = "value" form
  - [92] AXPopUpButton = "Automatic"  -> = "value" form
The old regex only matched quoted "label" and id=Label, so System Settings
buttons/text/popups all surfaced with empty labels. That's why selecting the
macOS Appearance 'Dark' button by element index required guessing — the
labels weren't available to aim with.

Fix: extend _ELEMENT_LINE_RE to capture all four label forms (= "value",
"quoted", (parenthesised), id=Label), skipping a pure-digit (N) order number
in favour of the id= label. Verified live against System Settings: the
Appearance buttons now surface as Auto/Light/Dark.

Adds a regression test covering all label forms. Full suite: 84 passed.
2026-07-05 05:36:31 -07:00
Adrian Lastra 13b75e73ff fix(computer_use): re-fetch via CLI when MCP returns silent-empty captures
The first fix handled the EAGAIN McpError path. But the persistent MCP
session (long-running gateway/desktop worker) has a second failure mode:
list_windows or get_window_state 'succeed' over MCP yet return a
degenerate/empty payload (no windows, or no screenshot + blank tree)
WITHOUT raising — typically when the bridge reconnected mid-call and
dropped the heavy response. That surfaced to the model as a silent 0x0
capture with no error and no fallback firing (0.00s empty return).

Fix: detect empty results in capture() and re-fetch over the CLI
transport before giving up:
  - empty list_windows -> CLI re-fetch the window list
  - empty get_window_state (som/ax) -> CLI re-fetch the AX tree + screenshot
  - empty screenshot (vision) -> CLI re-fetch get_window_state for the PNG

Adds 2 regression tests. Full suite: 83 passed.
2026-07-05 05:36:31 -07:00
Adrian Lastra 7af9abd174 fix(computer_use): fall back to CLI transport when cua-driver MCP bridge hits EAGAIN
The cua-driver MCP stdio bridge intermittently (and on some machines
persistently) fails to forward heavier calls like get_window_state to
the daemon with POSIX EAGAIN — 'daemon transport error forwarding
get_window_state: Resource temporarily unavailable (os error 35)'.
The wrapper surfaced this as an empty 0x0 capture, so computer_use
returned blank screenshots even though the display, permissions, and
the daemon were all healthy (the direct 'cua-driver call' CLI path
worked fine throughout).

Fix: when the MCP path raises the transient/transport error, fall back
to the 'cua-driver call' subprocess transport, which talks to the
daemon over a different socket. The CLI fallback routes get_window_state
screenshots to a temp file via screenshot_out_file (tiny JSON response
instead of a multi-MB base64 blob that congests the socket), reads the
PNG back, retries with backoff, and remaps the JSON into the same
{data, images, structuredContent, isError} shape the MCP path produces
so capture()/_action() are transport-agnostic.

Adds _is_transient_daemon_error() classifier and 3 regression tests.
Verified live: captures that returned 0x0 now return full
1567x905 screenshots with the AX element tree.
2026-07-05 05:36:31 -07:00
Teknium de4310c8f6
fix(computer-use): report the wedged startup phase in the session ready-timeout error (#58801)
The 'never reached ready' error (issue #57025) was undiagnosable — doctor
and MCP test pass while the wrapper times out, with no hint where startup
stalled. Track a phase marker through _lifecycle_coro (binary-check →
manifest-discovery → mcp-initialize → capability-discovery → ready) and
include it in the timeout RuntimeError plus a pointer to doctor and the
agent.log phase timings.

Complements the 15s→30s bump + success-path phase timing log from #58760.
2026-07-05 05:36:09 -07:00
liuhao1024 d537d29a6f fix(computer-use): increase cua-driver session startup timeout from 15s to 30s
On Windows, the cua-driver MCP session initialization can exceed the 15s
timeout due to manifest subprocess discovery + MCP transport setup.
This makes the computer_use tool permanently unavailable even though
hermes computer-use doctor and hermes mcp test both pass.

- Increase _ready_event.wait timeout from 15s to 30s
- Add startup timing instrumentation (manifest + mcp_init durations)
- Log timing at INFO level for diagnosability

Fixes #57025
2026-07-05 03:15:49 -07:00
Marxb85 fab6d4d1b7 Fix computer-use crash on X11 windows with null PID
On X11 a window's PID comes from the optional _NET_WM_PID property, so
cua-driver's list_windows legitimately returns pid: null for windows that
don't set it (desktop root, panels, override-redirect popups). capture()
and focus_app() coerced every entry via int(w["pid"]) inside a list
comprehension, so a single null-pid window raised TypeError and aborted
the whole enumeration before any screenshot — capture was impossible on
any X11 desktop with even one such window.

Route both ingestion sites through a new _ingest_windows() helper that
skips entries lacking a usable pid/window_id (uncapturable anyway) and
coerces the rest.

Adds tests/tools/test_computer_use_null_pid_windows.py covering the
helper's filtering/coercion and an end-to-end capture() regression that
reproduced the crash.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:37:33 -07:00
Teknium d3d621f7c3
revert(windows): roll back terminal-popup PRs #53791 #53810 #53829 (#53853)
* Revert "fix(windows): capture is not a no-window boundary; route flashing spawns through chokepoint (#53829)"

This reverts commit 2ecca1e7d3.

* Revert "fix(windows): stop terminal-window popups from background spawns (#53810)"

This reverts commit 5db1430af9.

* Revert "fix(windows): stop subprocess console-window popups + add CI guard (#53791)"

This reverts commit ef17cd204d.
2026-06-27 15:59:00 -07:00
Teknium ef17cd204d
fix(windows): stop subprocess console-window popups + add CI guard (#53791)
* fix(windows): stop subprocess console-window popups + add CI guard

The single biggest source of Windows 'terminal popup' bug reports was bare
subprocess.run/Popen calls spawning a console window. The compat helpers
(windows_hide_flags / windows_detach_popen_kwargs) already existed but the
footgun checker had no rule to stop new bare calls from reintroducing the flash.

- scripts/check-windows-footguns.py: new AST-based rule flagging subprocess
  calls that can create a new console — output-redirection-aware (capture/
  redirect/check_output exempt) and POSIX-only-program-aware (launchctl/
  systemctl/brew/etc. exempt). Comprehensive on real popups, no annotation
  burden on calls that can't flash.
- Swept all genuine window-spawning sites through windows_hide_flags()/
  windows_detach_popen_kwargs(); marked intentionally-visible launches
  (editor/terminal/foreground re-exec) with '# windows-footgun: ok'.
- tests/scripts/test_windows_footgun_subprocess_rule.py: behavior-contract
  tests + full-repo cleanliness invariant.
- CONTRIBUTING.md: documents the rule + the helper pattern.

* test: accept creationflags kwarg in psutil_android fake_subprocess_run

The Windows no-window sweep added creationflags=windows_hide_flags() to
install_psutil_android.py's subprocess.run call; the test's fake stub had a
fixed (cmd) signature and raised TypeError on the new kwarg.
2026-06-27 13:03:51 -07:00
Brooklyn Nicholson 833710d33e Merge remote-tracking branch 'origin/main' into pr-50994
# Conflicts:
#	tools/computer_use/cua_backend.py
2026-06-22 18:48:07 -05:00
brooklyn! 760fd9513e
Merge pull request #51078 from NousResearch/bb/fix-vision-capture
fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
2026-06-22 18:37:18 -05:00
Brooklyn Nicholson 3c1058e2e9 fix(computer-use): set stdin=DEVNULL on cua-driver subprocess calls
The subprocess-stdin guard (TUI gateway fd-inheritance protection) flagged
the `permissions grant` call. None of the cua-driver probes/grant read
stdin, so DEVNULL is correct; apply it to the shared `_run` helper and the
grant call.
2026-06-22 17:59:18 -05:00
Brooklyn Nicholson 2dfcead683 feat(computer-use): make the preflight cross-platform (win/linux)
The card was macOS-only. cua-driver also runs on Windows and Linux, so
fold `cua-driver doctor` (cross-platform binary/health probes) into a
single OS-aware `ready` signal:

- macOS: ready == both TCC grants; keeps the permission rows + grant flow.
- Windows/Linux: no TCC toggles, so ready == driver health, with a
  per-OS note (SmartScreen/UIAccess on Windows; X11/XWayland on Linux).

`computer_use_status()` replaces the macOS-only `permissions_status()` and
surfaces `platform`, `ready`, `can_grant`, and the doctor `checks` (non-ok
ones render as warnings). CLI `permissions status`, the REST endpoint, and
the desktop card all key off the one payload. Grant stays macOS-only (400
elsewhere — nothing to grant).
2026-06-22 17:48:43 -05:00
Brooklyn Nicholson 807b696295 fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
Vision mode called a `screenshot` MCP tool that cua-driver dropped in
0.5.x (full-window PNG capture was folded into `get_window_state`). The
driver replied "Unknown tool: screenshot", so `images` came back empty,
`png_b64` stayed None, and capture returned a 0x0 result with no image
on every call. `som`/`ax` were unaffected because they already use
`get_window_state`, which masked the regression.

Route vision by capability:
- driver advertises `screenshot` (older builds) -> use it (no AX walk)
- otherwise -> call `get_window_state` but discard the AX tree/elements,
  returning only the PNG so vision stays free of element noise
- capabilities not yet discovered -> try `screenshot`, fall back to
  `get_window_state` on an empty image, so the path self-heals

Add `_image_from_tool_result` to pull the PNG from either an MCP image
content-part or `structuredContent.screenshot_png_b64`, and use it on
the som path too so the image won't silently drop on driver builds that
deliver it via structuredContent instead of a content part.

Verified live (vision: 1568x954, 0 elements; som: image + 527 elements)
and with unit coverage of all four routing cases.
2026-06-22 17:41:42 -05:00