hermes-agent/tools
Teknium ee8cbfdc03
feat(web_extract): truncate-and-store instead of LLM summarization (#54843)
* feat(web_extract): truncate-and-store instead of LLM summarization

web_extract no longer runs an auxiliary LLM over scraped pages. The extract
backends (Firecrawl/Tavily/Exa/Parallel) already return clean, boilerplate-
stripped markdown, so we return it directly: pages within a char budget
(default 15000, web.extract_char_limit) come back whole; larger pages get a
head+tail window plus an explicit footer giving the stored full-text path and
the read_file call to page through the omitted middle. The full clean text is
written to cache/web (mounted read-only into remote backends like the other
cache dirs), so nothing is lost.

Inline base64 images are converted to [IMAGE: alt] placeholders (token bombs
dropped) while real http(s) image URLs are preserved as links so the agent can
still web_extract/vision_analyze them.

Removes process_content_with_llm + the chunked summarizer + check_auxiliary_model
+ _resolve_web_extract_auxiliary. context_references._default_url_fetcher is
updated to the truncate path and its stale data.documents shape read is fixed
to results (it was silently returning empty).

Live before/after eval (firecrawl, 4 URLs): 11.7x faster overall (176.6s ->
15.1s); 10-60x on large pages. Quality identical; findability 4/4 (answer
recoverable from stored full text on every truncated page). web_search is
unchanged.

No own scraper added; no changes to web_search.

* fix(web_extract): add char_limit to execute_code web_extract stub

The new web_extract char_limit param must appear in the code_execution_tool
_TOOL_STUBS signature (and doc line) or test_stubs_cover_all_schema_params
fails — the stub schema must cover every real schema param.
2026-06-29 10:00:49 -07:00
..
computer_use
…
environments
…
neutts_samples
…
__init__.py
…
ansi_strip.py
…
approval.py
…
async_delegation.py
…
binary_extensions.py
…
blueprints.py
…
browser_camofox.py fix(camofox): auto-recover from stale tab 404 on navigate 2026-06-29 01:26:24 -07:00
browser_camofox_state.py
…
browser_cdp_tool.py
…
browser_dialog_tool.py
…
browser_supervisor.py
…
browser_tool.py fix(security): redact browser CDP endpoint logs 2026-06-29 04:25:26 -07:00
budget_config.py
…
checkpoint_manager.py
…
clarify_gateway.py
…
clarify_tool.py
…
close_terminal_tool.py
…
code_execution_tool.py feat(web_extract): truncate-and-store instead of LLM summarization (#54843) 2026-06-29 10:00:49 -07:00
computer_use_tool.py
…
credential_files.py feat(web_extract): truncate-and-store instead of LLM summarization (#54843) 2026-06-29 10:00:49 -07:00
cronjob_tools.py
…
debug_helpers.py
…
delegate_tool.py
…
discord_tool.py
…
env_passthrough.py
…
env_probe.py
…
fal_common.py
…
feishu_doc_tool.py
…
feishu_drive_tool.py
…
file_operations.py
…
file_state.py
…
file_tools.py fix(file-tools): sanitize host/relative cwd override before it reaches container sandbox (#54447) (#54616) 2026-06-29 15:32:20 +10:00
fuzzy_match.py
…
homeassistant_tool.py
…
image_generation_tool.py
…
interrupt.py
…
kanban_tools.py
…
lazy_deps.py fix(memory): lazy-install supermemory + mem0 SDKs like honcho/hindsight 2026-06-29 00:25:36 -07:00
managed_tool_gateway.py
…
mcp_oauth.py
…
mcp_oauth_manager.py
…
mcp_tool.py
…
memory_tool.py
…
microsoft_graph_auth.py
…
microsoft_graph_client.py
…
neutts_synth.py
…
openrouter_client.py
…
osv_check.py
…
patch_parser.py
…
path_security.py
…
process_registry.py
…
project_tools.py
…
read_extract.py
…
read_terminal_tool.py
…
registry.py
…
schema_sanitizer.py
…
send_message_tool.py
…
session_search_tool.py
…
skill_manager_tool.py
…
skill_provenance.py
…
skill_usage.py
…
skills_ast_audit.py
…
skills_guard.py
…
skills_hub.py
…
skills_sync.py
…
skills_tool.py
…
slash_confirm.py
…
terminal_tool.py
…
thread_context.py
…
threat_patterns.py
…
tirith_security.py
…
todo_tool.py
…
tool_backend_helpers.py
…
tool_output_limits.py
…
tool_result_storage.py
…
tool_search.py
…
transcription_tools.py
…
tts_tool.py
…
url_safety.py
…
video_generation_tool.py
…
vision_tools.py fix(vision): narrow the fan-out cap to the CPU encode burst only 2026-06-29 01:27:10 -07:00
voice_mode.py
…
web_tools.py feat(web_extract): truncate-and-store instead of LLM summarization (#54843) 2026-06-29 10:00:49 -07:00
website_policy.py
…
write_approval.py
…
x_search_tool.py
…
xai_http.py
…
yuanbao_tools.py
…