Ben Barclay eddfecd2ce fix(vision): cap vision_analyze fan-out concurrency process-wide ... A single agent turn can fan out N vision_analyze calls at once — the classic trigger is "analyze every frame of this video", where ffmpeg explodes a clip into dozens of frames and the model calls vision_analyze on each. Every call does a CPU-heavy base64-encode/resize burst AND holds a long-lived LLM stream open. The tool executor runs concurrent tool calls on a per-session ThreadPoolExecutor (_MAX_TOOL_WORKERS=8), and multiple agent sessions share one process (the dashboard runs the agent in-process), so there was no global ceiling. In prod (June 2026) a video-frame fan-out pinned a worker thread at ~100% CPU and starved the shared asyncio event loop that also serves the dashboard's /api/status liveness probe, flapping the instance to UNHEALTHY even though nothing had crashed. Add a process-global threading.BoundedSemaphore that bounds how many vision analyses run concurrently across the whole process, held across the entire analysis (image load + encode + LLM call) in the single _handle_vision_analyze chokepoint (covers both the native fast path and the legacy aux-LLM path). It is a threading semaphore, NOT asyncio: each vision call is dispatched through model_tools._run_async on a per-thread event loop, so an asyncio primitive bound to one loop cannot coordinate across them. The acquire is offloaded via run_in_executor so waiting for a slot never blocks the calling loop. Default: min(host CPUs, 4), floored at 1 — respect the host's concurrency, or lower. Override via auxiliary.vision.max_concurrency (config.yaml) or HERMES_VISION_MAX_CONCURRENCY (env). Values < 1 are ignored so the cap can never be disabled into an unbounded fan-out. Tests: bounded-fan-out regression guard + a control proving it would fail without the cap; resolver tests for host-cpu default, ceiling clamp, low-cpu host, env override, and sub-1 rejection. Pre-existing handler tests updated for the now-async _handle_vision_analyze. Verified via the real registry.dispatch -> _run_async per-thread-loop path (16 concurrent calls, peak bounded to cap).		2026-06-29 01:27:10 -07:00
..
__init__.py	…
conftest.py	…
test_accretion_caps.py	…
test_ansi_strip.py	…
test_approval.py	fix(approvals): warn and default to manual on unknown approvals.mode	2026-06-28 19:04:18 -07:00
test_approval_heartbeat.py	…
test_approval_interrupt.py	…
test_approval_plugin_hooks.py	…
test_async_delegation.py	…
test_base_environment.py	fix(environments): use $BASHPID for atomic snapshot temp + harden failure path	2026-06-28 02:08:57 -07:00
test_blueprints.py	…
test_browser_camofox.py	…
test_browser_camofox_auth.py	fix(browser): send Authorization header in Camofox HTTP calls when CAMOFOX_API_KEY is set	2026-06-29 01:26:24 -07:00
test_browser_camofox_ensure_tab.py	fix(tools): send listItemId instead of sessionKey in Camofox tab creation	2026-06-29 01:26:24 -07:00
test_browser_camofox_persistence.py	test(camofox): accept headers= kwarg in persistence test mocks	2026-06-29 01:26:24 -07:00
test_browser_camofox_state.py	…
test_browser_camofox_timeout.py	fix(tools): read browser.command_timeout in Camofox HTTP client	2026-06-29 01:26:24 -07:00
test_browser_cdp_override.py	…
test_browser_cdp_tool.py	…
test_browser_chromium_autoinstall.py	feat(browser): auto-install Chromium binary on local cold-start failure	2026-06-28 12:25:15 -05:00
test_browser_chromium_check.py	fix(windows): cover remaining console-flash spawn legs (#54417)	2026-06-28 13:49:08 -07:00
test_browser_cleanup.py	…
test_browser_cloud_fallback.py	…
test_browser_cloud_provider_cache.py	…
test_browser_console.py	…
test_browser_content_none_guard.py	…
test_browser_eval_ssrf.py	fix(browser): close eval return-value SSRF bypass (sibling of #44731)	2026-06-28 02:42:01 -07:00
test_browser_eval_supervisor_path.py	…
test_browser_get_images_ssrf.py	fix(browser): extend private-network guard to browser_get_images	2026-06-28 14:25:10 -07:00
test_browser_hardening.py	revert(windows): roll back terminal-popup PRs #53791 #53810 #53829 (#53853)	2026-06-27 15:59:00 -07:00
test_browser_homebrew_paths.py	fix(windows): cover remaining console-flash spawn legs (#54417)	2026-06-28 13:49:08 -07:00
test_browser_hybrid_routing.py	…
test_browser_lightpanda.py	…
test_browser_open_timeout.py	test(browser): cover open timeout diagnostics and failed navigate title	2026-06-28 12:14:21 -05:00
test_browser_orphan_reaper.py	…
test_browser_secret_exfil.py	…
test_browser_snapshot_ssrf.py	fix(browser): allow local sidecar sessions to bypass SSRF guard	2026-06-28 02:42:01 -07:00
test_browser_ssrf_local.py	…
test_browser_supervisor.py	…
test_browser_supervisor_healthcheck.py	…
test_browser_type_redaction.py	…
test_budget_config.py	…
test_checkpoint_manager.py	…
test_clarify_gateway.py	fix: accept typed clarify choice replies	2026-06-28 04:13:19 -07:00
test_clarify_tool.py	…
test_clipboard.py	…
test_code_execution.py	…
test_code_execution_modes.py	…
test_code_execution_windows_env.py	…
test_command_guards.py	…
test_computer_use.py	…
test_computer_use_capture_routing.py	…
test_computer_use_vision_routing.py	…
test_config_null_guard.py	…
test_container_cwd_sanitize.py	fix(file-tools): sanitize host/relative cwd override before it reaches container sandbox (#54447) (#54616)	2026-06-29 15:32:20 +10:00
test_credential_files.py	fix(hermes-home): only honour legacy dir layout when it has content	2026-06-27 04:57:15 +05:30
test_credential_pool_env_fallback.py	…
test_cron_approval_mode.py	…
test_cron_prompt_injection.py	…
test_cronjob_run_immediate.py	…
test_cronjob_tools.py	…
test_cross_profile_guard.py	…
test_daytona_environment.py	fix(daytona): quote single-upload mkdir parent path (#54440)	2026-06-28 14:33:03 -07:00
test_debug_helpers.py	…
test_delegate.py	test(delegate): cover stale parent base_url inheritance for subagents	2026-06-27 03:59:36 +05:30
test_delegate_composite_toolsets.py	…
test_delegate_subagent_timeout_diagnostic.py	…
test_delegate_toolset_scope.py	…
test_discord_tool.py	…
test_docker_cgroup_limits.py	fix(docker): gate resource limit flags on cgroup controller availability (#54516)	2026-06-29 11:01:08 +10:00
test_docker_config_migrate.py	…
test_docker_environment.py	fix(docker): gate resource limit flags on cgroup controller availability (#54516)	2026-06-29 11:01:08 +10:00
test_docker_find.py	…
test_docker_orphan_reaper_integration.py	…
test_dockerfile_immutable_install.py	refactor(ci): faster docker builds via --link and chmod removal	2026-06-26 19:15:18 -07:00
test_dockerfile_node_modules_perms.py	…
test_dockerfile_pid1_reaping.py	…
test_env_passthrough.py	fix(env-passthrough): fail closed when provider blocklist import fails	2026-06-28 02:05:43 -07:00
test_env_probe.py	…
test_execute_code_approval_cluster.py	…
test_feishu_tools.py	…
test_file_operations.py	…
test_file_operations_edge_cases.py	…
test_file_ops_cwd_tracking.py	…
test_file_read_guards.py	…
test_file_staleness.py	…
test_file_state_registry.py	…
test_file_sync.py	…
test_file_sync_back.py	…
test_file_sync_perf.py	…
test_file_tools.py	fix(file-ops): make preserved cwd reachable at write-time resolution (#26211)	2026-06-27 19:29:06 -07:00
test_file_tools_container_config.py	…
test_file_tools_cwd_resolution.py	fix(file-ops): keep worktree isolation when restoring preserved cwd (#26211)	2026-06-27 19:29:06 -07:00
test_file_tools_live.py	…
test_file_tools_tilde_profile.py	…
test_file_write_safety.py	fix: use os.pathsep, add tests, update tips for multi-root support	2026-06-27 04:01:12 +05:30
test_find_shell.py	…
test_force_dangerous_override.py	…
test_fuzzy_match.py	…
test_gateway_cwd_contract.py	…
test_hardline_blocklist.py	…
test_heartbeat_stale_thresholds.py	…
test_hermes_subprocess_env.py	fix(security): centralize credential-safe subprocess env (#29157)	2026-06-27 20:45:31 -07:00
test_hidden_dir_filter.py	…
test_homeassistant_tool.py	…
test_image_generation.py	…
test_image_generation_artifacts.py	…
test_image_generation_env.py	…
test_image_generation_image_to_image.py	…
test_image_generation_plugin_dispatch.py	…
test_init_session_cwd_respect.py	…
test_interrupt.py	…
test_kanban_redaction.py	…
test_kanban_tools.py	…
test_lazy_deps.py	…
test_lazy_deps_durable_target.py	…
test_line_ending_preservation.py	…
test_llm_content_none_guard.py	…
test_local_background_child_hang.py	…
test_local_env_blocklist.py	fix(terminal): strip VIRTUAL_ENV/CONDA_PREFIX from terminal subprocess env	2026-06-28 01:04:20 +05:30
test_local_env_cwd_recovery.py	…
test_local_env_windows_msys.py	…
test_local_interrupt_cleanup.py	…
test_local_shell_init.py	…
test_local_tempdir.py	…
test_managed_browserbase_and_modal.py	…
test_managed_media_gateways.py	…
test_managed_modal_environment.py	…
test_managed_tool_gateway.py	…
test_mcp_cancelled_error_propagation.py	…
test_mcp_capability_gating.py	…
test_mcp_circuit_breaker.py	fix(mcp): never permanently wedge the circuit breaker on a dead transport (#53599)	2026-06-27 04:39:54 -07:00
test_mcp_client_cert.py	…
test_mcp_dynamic_discovery.py	…
test_mcp_elicitation.py	…
test_mcp_empty_error_message.py	…
test_mcp_image_content.py	…
test_mcp_invalid_url.py	…
test_mcp_loop_profile_override.py	…
test_mcp_oauth.py	fix(mcp): suppress interactive OAuth stdin prompts during background discovery (#35927)	2026-06-27 04:59:23 +05:30
test_mcp_oauth_bidirectional.py	…
test_mcp_oauth_cold_load_expiry.py	…
test_mcp_oauth_integration.py	…
test_mcp_oauth_manager.py	…
test_mcp_oauth_metadata.py	…
test_mcp_preflight_content_type.py	fix(mcp): skip preflight content-type probe for OAuth servers	2026-06-28 04:47:39 -07:00
test_mcp_probe.py	…
test_mcp_reconnect_signal.py	…
test_mcp_sse_transport.py	…
test_mcp_stability.py	…
test_mcp_structured_content.py	…
test_mcp_tool.py	…
test_mcp_tool_401_handling.py	…
test_mcp_tool_issue_948.py	…
test_mcp_tool_session_expired.py	…
test_mcp_utility_capability_gating.py	…
test_memory_tool.py	…
test_memory_tool_import_fallback.py	…
test_memory_tool_schema.py	…
test_microsoft_graph_auth.py	…
test_microsoft_graph_client.py	…
test_modal_bulk_upload.py	…
test_modal_sandbox_fixes.py	fix(terminal): require approval for host-bound Docker commands (#54483)	2026-06-29 11:35:41 +10:00
test_modal_snapshot_isolation.py	…
test_notify_on_complete.py	…
test_osv_check.py	…
test_parse_env_var.py	…
test_patch_failure_tracking.py	…
test_patch_parser.py	…
test_pr_6656_regressions.py	…
test_process_registry.py	feat(desktop): live agent terminals + agent-driven tab close	2026-06-28 21:15:14 -05:00
test_read_extract.py	…
test_read_loop_detection.py	…
test_refresh_agent_mcp_tools.py	…
test_registry.py	…
test_resolve_path.py	…
test_schema_sanitizer.py	…
test_search_budget_truncation.py	…
test_search_error_guard.py	…
test_search_hidden_dirs.py	…
test_send_message_missing_platforms.py	…
test_send_message_react.py	…
test_send_message_target_parse.py	…
test_send_message_telegram_proxy.py	…
test_send_message_tool.py	test+harden: modernize salvaged Matrix path for current plugin layout	2026-06-28 12:48:08 +05:30
test_session_search.py	fix(session_search): demote cron below interactive sessions in discover ranking (#53597)	2026-06-27 04:41:22 -07:00
test_shared_container_task_id.py	…
test_signal_media.py	…
test_singularity_preflight.py	…
test_skill_env_passthrough.py	…
test_skill_improvements.py	…
test_skill_manager_tool.py	fix(curator): fail closed on unverified skill deletes during consolidation (#53935)	2026-06-27 20:45:57 -07:00
test_skill_provenance.py	…
test_skill_size_limits.py	…
test_skill_usage.py	…
test_skill_view_path_check.py	…
test_skill_view_traversal.py	…
test_skills_ast_audit.py	…
test_skills_guard.py	…
test_skills_hub.py	fix(skills): replace string prefix check with strict path containment	2026-06-28 21:14:01 -07:00
test_skills_hub_browse_sh.py	…
test_skills_hub_clawhub.py	…
test_skills_list_modified_diff.py	…
test_skills_sync.py	fix(skills): skip shadowing when external_dirs provides the skill	2026-06-27 21:07:53 -07:00
test_skills_tool.py	…
test_slash_confirm.py	…
test_smart_approval_injection.py	…
test_spotify_client.py	…
test_ssh_bulk_upload.py	…
test_ssh_environment.py	…
test_stage2_hook_seed_one_symlinks.py	…
test_stage2_hook_symlink_chown.py	…
test_subprocess_stdin_guard.py	…
test_symlink_prefix_confusion.py	…
test_sync_back_backends.py	…
test_terminal_compound_background.py	…
test_terminal_config_env_sync.py	…
test_terminal_exit_semantics.py	…
test_terminal_foreground_timeout_cap.py	…
test_terminal_none_command_guard.py	…
test_terminal_output_transform_hook.py	fix(redact): stop DB-connstr redaction from corrupting code output (#33801) (#54061)	2026-06-28 01:15:39 -07:00
test_terminal_requirements.py	…
test_terminal_task_cwd.py	fix(terminal): require approval for host-bound Docker commands (#54483)	2026-06-29 11:35:41 +10:00
test_terminal_timeout_output.py	…
test_terminal_tool.py	…
test_terminal_tool_pty_fallback.py	…
test_terminal_tool_requirements.py	fix(tools): suppress transient check_fn flakes so subagents keep file/terminal tools	2026-06-27 19:29:00 -07:00
test_threaded_process_handle.py	…
test_threat_patterns.py	…
test_tirith_security.py	…
test_todo_tool.py	…
test_todo_tool_type_coercion.py	…
test_tool_backend_helpers.py	…
test_tool_output_limits.py	…
test_tool_result_storage.py	…
test_tool_search.py	…
test_transcription.py	…
test_transcription_command_providers.py	…
test_transcription_dotenv_fallback.py	…
test_transcription_plugin_dispatch.py	…
test_transcription_tools.py	…
test_tts_command_providers.py	…
test_tts_dotenv_fallback.py	…
test_tts_gemini.py	…
test_tts_kittentts.py	…
test_tts_max_text_length.py	…
test_tts_mistral.py	…
test_tts_opus_routing.py	…
test_tts_path_traversal.py	…
test_tts_piper.py	…
test_tts_plugin_dispatch.py	…
test_tts_speed.py	…
test_tts_xai_speech_tags.py	…
test_url_safety.py	…
test_video_analyze.py	…
test_video_generation_dispatch.py	…
test_video_generation_dynamic_schema.py	…
test_video_generation_tool_surface_matrix.py	…
test_vision_native_fast_path.py	…
test_vision_tools.py	fix(vision): cap vision_analyze fan-out concurrency process-wide	2026-06-29 01:27:10 -07:00
test_voice_cli_integration.py	…
test_voice_mode.py	…
test_watch_patterns.py	…
test_web_providers.py	…
test_web_providers_brave_free.py	…
test_web_providers_ddgs.py	…
test_web_providers_searxng.py	…
test_web_providers_xai.py	…
test_web_tools_config.py	…
test_web_tools_tavily.py	…
test_website_policy.py	…
test_whatsapp_send_message_media.py	feat(send_message): native WhatsApp media delivery via Baileys bridge (#53598)	2026-06-27 04:40:05 -07:00
test_windows_compat.py	fix(tools): use start_new_session instead of preexec_fn to prevent SIGSEGV in multi-threaded processes	2026-06-27 03:08:41 +05:30
test_windows_native_support.py	revert(windows): roll back terminal-popup PRs #53791 #53810 #53829 (#53853)	2026-06-27 15:59:00 -07:00
test_write_approval.py	…
test_write_deny.py	…
test_x_search_tool.py	…
test_yolo_mode.py	…
test_zombie_process_cleanup.py	…