RomanNum3ral
/
hermes-agent
mirror of https://github.com/NousResearch/hermes-agent.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
hermes-agent/plugins/observability/langfuse
History
kshitij ace830134e fix: reuse redact_sensitive_text, fix leaky abstraction, fix test data 
Follow-up fixes from /hermes-pr-review + /simplify-code on PR #83437:

1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True)
   — the plugin's 11-pattern list was a strict subset of the 50+ patterns in
   agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens,
   HuggingFace tokens, DB connection strings, and Telegram bot tokens would
   all leak through the plugin's list but are caught by the existing redactor.
   Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py.

2. Remove dead 'not isinstance(client, object)' check in on_session_finalize —
   always False for any Python value.

3. Fix MoAClient.last_reference_metrics() to call the public
   self.chat.completions.last_reference_metrics() instead of reaching into
   the private _last_reference_metrics attribute via getattr.

4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass
   pre_coerced=input_messages to _messages_for_langfuse_input to avoid
   double-coercion + double _capture_content serialization per API request.

5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py
   for consistency with the other HERMES_LANGFUSE_* env vars.

6. Fix test_sanitized_mode_redacts_secrets test data — the old samples
   ('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too
   short to match the regex thresholds and never actually tested redaction.
   Updated to realistic-length secrets and changed assertions to check that
   the output differs from input (redact_sensitive_text masks rather than
   inserting the literal string 'REDACTED').
 		2026-08-13 23:10:16 +05:30
..
README.md
…
__init__.py
…
plugin.yaml
…

README.md

Langfuse Observability Plugin

This plugin ships bundled with Hermes but is opt-in — it only loads when you explicitly enable it.

Enable

Pick one:

# Interactive: walks you through credentials + SDK install + enable
hermes tools  # → Langfuse Observability

# Manual
pip install langfuse
hermes plugins enable observability/langfuse

Required credentials

Set these in ~/.hermes/.env (or via hermes tools):

HERMES_LANGFUSE_PUBLIC_KEY=pk-lf-...
HERMES_LANGFUSE_SECRET_KEY=sk-lf-...
HERMES_LANGFUSE_BASE_URL=https://cloud.langfuse.com   # or your self-hosted URL

Without the SDK or credentials the hooks no-op silently — the plugin fails open.

Verify

hermes plugins list                 # observability/langfuse should show "enabled"
hermes chat -q "hello"              # then check Langfuse for a "Hermes turn" trace

Generation observations include the Hermes system prompt when the provider uses a separate system param (Anthropic Messages API). Open an LLM call child span to inspect role: system (truncated via HERMES_LANGFUSE_MAX_CHARS).

Optional tuning

HERMES_LANGFUSE_ENV=production       # environment tag
HERMES_LANGFUSE_RELEASE=v1.0.0       # release tag
HERMES_LANGFUSE_SAMPLE_RATE=0.5      # sample 50% of traces
HERMES_LANGFUSE_MAX_CHARS=12000      # max chars per field (default: 12000)
HERMES_LANGFUSE_CAPTURE=sanitized    # content capture mode (see below)
HERMES_LANGFUSE_DEBUG=true           # verbose plugin logging

Capture modes

HERMES_LANGFUSE_CAPTURE controls how much content (prompts, responses, tool arguments/results) is exported. Structural metadata — IDs, roles, tool names, token usage, cost, timing — is always captured in every mode.

mode behavior
metadata No content. Each content field is replaced by a shape/size stub ({"omitted": true, "type": "text", "chars": N}).
sanitized (default) Content is exported after secret-pattern redaction (API keys, tokens, JWTs, private keys, password=-style assignments) and truncation. Redaction runs before truncation.
full Raw content, truncated only. Explicit opt-in — traces will contain whatever passed through the conversation, including injected memory and file contents.

The active mode is recorded on every trace as metadata.capture_mode.

Note: sanitized is pattern-based defense in depth, not a DLP guarantee. For personal sessions or shared Langfuse projects, prefer metadata.

Error + shutdown coverage

  • Failed model requests (api_request_error hook) close their generation with level=ERROR, status code, retry counters, and a capture-mode-scrubbed error message. Non-retryable failures also finish the turn trace.
  • Session end/finalize closes any still-open traces for that session and flushes queued events, so interrupted or tool-only turns don't dangle.

Disable

hermes plugins disable observability/langfuse