9.9 KiB
9.9 KiB
External Logging Integration Hooks
Interfaces for external logging systems.
Related: LOGGING_PATTERNS.md | AGENT_GUARDRAILS.md
Overview
This document defines hooks and interfaces for integrating agent logs with external logging systems. These patterns prepare the codebase for future integration with centralized logging, monitoring, and alerting platforms.
Integration Architecture
Hook Points
AGENT OPERATION
│
├── [HOOK: pre_operation] ────→ Log operation start
│
├── Execute operation
│
├── [HOOK: post_operation] ───→ Log operation result
│
└── [HOOK: on_error] ─────────→ Log error details
Data Flow Diagram
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Agent │ ───→ │ Log Hook │ ───→ │ External │
│ Operation │ │ Interface │ │ System │
└─────────────┘ └─────────────┘ └─────────────┘
│
├──→ Webhook (HTTP)
├──→ File (local/remote)
├──→ Queue (message bus)
└──→ API (direct call)
Standard Hook Interface
Required Interface Methods
interface LoggingHook {
// Initialize the hook with configuration
initialize(config: HookConfig): Promise<void>;
// Send log entries to external system
send(entries: LogEntry[]): Promise<SendResult>;
// Flush any buffered entries
flush(): Promise<void>;
// Clean shutdown
shutdown(): Promise<void>;
}
interface HookConfig {
endpoint?: string; // URL or path
auth?: AuthConfig; // Authentication
batch_size?: number; // Entries per send
timeout_ms?: number; // Send timeout
retry_count?: number; // Retry attempts
}
interface SendResult {
success: boolean;
entries_sent: number;
errors?: string[];
}
Configuration Schema
{
"logging_hook": {
"type": "webhook | file | queue | api",
"enabled": true,
"config": {
"endpoint": "https://logs.example.com/ingest",
"auth": {
"type": "bearer",
"token_env": "LOG_API_TOKEN"
},
"batch_size": 100,
"timeout_ms": 5000,
"retry_count": 3
}
}
}
Supported Integration Types
Webhook Integration
Send logs via HTTP POST to external endpoint.
WEBHOOK FLOW:
1. Buffer log entries locally
2. When batch_size reached or flush called:
POST /ingest
Content-Type: application/json
Authorization: Bearer <token>
Body: { "logs": [...entries] }
3. Handle response
4. Retry on failure
Configuration Example:
{
"type": "webhook",
"config": {
"endpoint": "https://logs.example.com/api/v1/ingest",
"auth": {
"type": "bearer",
"token_env": "LOGGING_API_TOKEN"
},
"headers": {
"X-Source": "agent-guardrails"
},
"batch_size": 50,
"timeout_ms": 10000
}
}
File-Based Integration
Write logs to file for pickup by external system.
FILE FLOW:
1. Buffer log entries
2. When batch_size reached or flush called:
- Write JSON Lines to file
- Rotate files by size/time
3. External system picks up files
Configuration Example:
{
"type": "file",
"config": {
"path": "/var/log/agent/{date}.jsonl",
"rotation": {
"max_size_mb": 100,
"max_age_days": 7
},
"format": "jsonl"
}
}
Queue-Based Integration
Send logs to message queue for async processing.
QUEUE FLOW:
1. Buffer log entries
2. When batch_size reached:
- Publish to queue topic
3. Consumer processes messages
Configuration Example:
{
"type": "queue",
"config": {
"provider": "redis | rabbitmq | sqs",
"connection": {
"host": "localhost",
"port": 6379
},
"topic": "agent-logs",
"batch_size": 100
}
}
Direct API Integration
Call logging API directly (e.g., Datadog, Splunk).
API FLOW:
1. Format entries for specific API
2. Call API endpoint
3. Handle API-specific responses
Configuration Example:
{
"type": "api",
"config": {
"provider": "datadog",
"api_key_env": "DD_API_KEY",
"site": "datadoghq.com",
"service": "agent-guardrails",
"source": "ai-agent"
}
}
Configuration Templates
Minimal Configuration (File)
{
"logging_hook": {
"type": "file",
"enabled": true,
"config": {
"path": "./logs/agent.jsonl"
}
}
}
Production Configuration (Webhook)
{
"logging_hook": {
"type": "webhook",
"enabled": true,
"config": {
"endpoint": "${LOG_ENDPOINT}",
"auth": {
"type": "bearer",
"token_env": "LOG_TOKEN"
},
"batch_size": 100,
"timeout_ms": 5000,
"retry_count": 3,
"retry_delay_ms": 1000
}
}
}
Placeholder Implementations
Stub Hook Template
For development/testing without external system:
class StubLoggingHook implements LoggingHook {
private buffer: LogEntry[] = [];
async initialize(config: HookConfig): Promise<void> {
console.log("StubLoggingHook initialized");
}
async send(entries: LogEntry[]): Promise<SendResult> {
this.buffer.push(...entries);
console.log(`[STUB] Received ${entries.length} log entries`);
return { success: true, entries_sent: entries.length };
}
async flush(): Promise<void> {
console.log(`[STUB] Flushing ${this.buffer.length} entries`);
this.buffer = [];
}
async shutdown(): Promise<void> {
await this.flush();
console.log("[STUB] Shutdown complete");
}
// For testing: get buffered entries
getBuffer(): LogEntry[] {
return [...this.buffer];
}
}
Console Hook (Development)
class ConsoleLoggingHook implements LoggingHook {
async initialize(config: HookConfig): Promise<void> {}
async send(entries: LogEntry[]): Promise<SendResult> {
entries.forEach(entry => {
const line = `[${entry.timestamp}] ${entry.level} [${entry.category}] ${entry.action} ${entry.target} → ${entry.result}`;
console.log(line);
});
return { success: true, entries_sent: entries.length };
}
async flush(): Promise<void> {}
async shutdown(): Promise<void> {}
}
Migration Path
From Local to External
MIGRATION STEPS:
1. PHASE 1: Local logging
- Implement LOGGING_PATTERNS
- Use StubLoggingHook or ConsoleLoggingHook
- Verify log format and content
2. PHASE 2: File-based external
- Configure FileLoggingHook
- External system picks up files
- Verify ingestion
3. PHASE 3: Direct integration
- Configure WebhookLoggingHook or APILoggingHook
- Remove file intermediate
- Full real-time logging
Gradual Rollout
1. Enable hook in development
2. Test with limited traffic
3. Monitor for errors
4. Gradually increase batch_size
5. Enable in production
Error Handling
External System Failures
ON SEND FAILURE:
1. Retry with exponential backoff
- Attempt 1: immediate
- Attempt 2: wait 1s
- Attempt 3: wait 2s
- Attempt 4: wait 4s
2. If all retries fail:
- Log error locally
- Continue operation (don't block agent)
- Buffer entries for next attempt
3. Circuit breaker (optional):
- After N consecutive failures, stop trying
- Retry after cooldown period
Retry Patterns
async function sendWithRetry(
entries: LogEntry[],
maxRetries: number = 3
): Promise<SendResult> {
for (let i = 0; i < maxRetries; i++) {
try {
return await hook.send(entries);
} catch (error) {
if (i === maxRetries - 1) throw error;
await sleep(Math.pow(2, i) * 1000);
}
}
}
Security Considerations
Credential Management
RULES:
- NEVER hardcode API keys or tokens
- Use environment variables
- Use secret management (GitHub Secrets, Vault)
- Rotate credentials regularly
Data Sanitization
BEFORE SENDING EXTERNALLY:
[ ] Remove passwords and tokens
[ ] Mask PII (email, phone, etc.)
[ ] Truncate large payloads
[ ] Remove file contents (log metadata only)
Quick Reference
+------------------------------------------------------------------+
| LOGGING INTEGRATION QUICK REFERENCE |
+------------------------------------------------------------------+
| HOOK INTERFACE: |
| initialize(config) → Setup connection |
| send(entries) → Send log batch |
| flush() → Send buffered entries |
| shutdown() → Clean disconnect |
+------------------------------------------------------------------+
| INTEGRATION TYPES: |
| webhook → HTTP POST to endpoint |
| file → Write to file for pickup |
| queue → Publish to message queue |
| api → Direct provider API |
+------------------------------------------------------------------+
| ERROR HANDLING: |
| - Retry with exponential backoff |
| - Don't block agent on failures |
| - Log errors locally |
+------------------------------------------------------------------+
Authored by: TheArchitectit Document Owner: Project Maintainers Last Updated: 2026-01-14 Line Count: ~300