768 lines
15 KiB
Markdown
768 lines
15 KiB
Markdown
# Troubleshooting Guide
|
|
|
|
> Common issues, solutions, and recovery procedures for Agent Guardrails Template
|
|
|
|
**Version:** 1.0
|
|
**Last Updated:** 2026-02-15
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Quick Diagnostics](#quick-diagnostics)
|
|
2. [Common Errors](#common-errors)
|
|
3. [Debug Mode](#debug-mode)
|
|
4. [Log Analysis](#log-analysis)
|
|
5. [Performance Issues](#performance-issues)
|
|
6. [Recovery Procedures](#recovery-procedures)
|
|
7. [Getting Help](#getting-help)
|
|
|
|
---
|
|
|
|
## Quick Diagnostics
|
|
|
|
Run this checklist to quickly identify common issues:
|
|
|
|
```bash
|
|
# 1. Verify installation
|
|
curl -s http://localhost:8094/mcp/v1/health | jq .
|
|
|
|
# 2. Check configuration
|
|
ls -la .teams/
|
|
ls -la .guardrails/
|
|
|
|
# 3. Validate tools
|
|
python scripts/team_manager.py --validate
|
|
|
|
# 4. Test MCP connection
|
|
curl -s -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{"jsonrpc":"2.0","method":"tools/list"}' | jq .
|
|
```
|
|
|
|
**Expected Results:**
|
|
- Health endpoint returns `{"status": "ok"}`
|
|
- `.teams/` and `.guardrails/` directories exist
|
|
- Team manager script exits with code 0
|
|
- Tools list returns available guardrail tools
|
|
|
|
---
|
|
|
|
## Common Errors
|
|
|
|
### MCP Server Connection Failed
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: Connection refused (localhost:8094)
|
|
Error: Could not connect to MCP server
|
|
```
|
|
|
|
**Causes:**
|
|
- MCP server not running
|
|
- Wrong port configuration
|
|
- Firewall blocking connection
|
|
|
|
**Solutions:**
|
|
|
|
1. **Start the MCP server:**
|
|
```bash
|
|
python mcp_server.py
|
|
# or
|
|
./start-mcp-server.sh
|
|
```
|
|
|
|
2. **Verify port configuration:**
|
|
```bash
|
|
# Check if server is listening
|
|
netstat -tlnp | grep 8094
|
|
# or
|
|
lsof -i :8094
|
|
```
|
|
|
|
3. **Check firewall rules:**
|
|
```bash
|
|
# For Linux
|
|
sudo ufw allow 8094/tcp
|
|
# For macOS
|
|
sudo pfctl -e
|
|
```
|
|
|
|
---
|
|
|
|
### Team Initialization Failed
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: TEAM-001: Team not found
|
|
Error: Failed to initialize project
|
|
```
|
|
|
|
**Causes:**
|
|
- Project name contains invalid characters
|
|
- `.teams/` directory does not exist
|
|
- Permission issues
|
|
|
|
**Solutions:**
|
|
|
|
1. **Verify project name:**
|
|
```bash
|
|
# Valid: my-project, project_123, team-alpha
|
|
# Invalid: my project, project;rm -rf /
|
|
```
|
|
|
|
2. **Create required directories:**
|
|
```bash
|
|
mkdir -p .teams/
|
|
chmod 755 .teams/
|
|
```
|
|
|
|
3. **Check permissions:**
|
|
```bash
|
|
ls -la .teams/
|
|
# Should be writable by current user
|
|
```
|
|
|
|
---
|
|
|
|
### Team Size Violation (TEAM-007)
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: TEAM-005: Team size violation
|
|
Team 7 has 8 members (maximum is 6)
|
|
```
|
|
|
|
**Causes:**
|
|
- Too many members assigned to a team
|
|
- Batch assignment exceeded limits
|
|
|
|
**Solutions:**
|
|
|
|
1. **Check current team sizes:**
|
|
```bash
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{
|
|
"jsonrpc":"2.0",
|
|
"method":"tools/call",
|
|
"params":{
|
|
"name":"guardrail_team_size_validate",
|
|
"arguments":{"project_name":"my-project"}
|
|
}
|
|
}'
|
|
```
|
|
|
|
2. **Remove excess members:**
|
|
```bash
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{
|
|
"jsonrpc":"2.0",
|
|
"method":"tools/call",
|
|
"params":{
|
|
"name":"guardrail_team_unassign",
|
|
"arguments":{
|
|
"project_name":"my-project",
|
|
"team_id":7,
|
|
"role_name":"Extra Role"
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
3. **Rebalance across teams:**
|
|
- Move members to teams with fewer than 4 members
|
|
- Split large teams into multiple smaller teams
|
|
|
|
---
|
|
|
|
### Phase Gate Check Failed
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: Phase gate requirements not met
|
|
Missing deliverables: Architecture Decision Records
|
|
```
|
|
|
|
**Causes:**
|
|
- Required deliverables not complete
|
|
- Required teams not assigned
|
|
- Approval not obtained
|
|
|
|
**Solutions:**
|
|
|
|
1. **List gate requirements:**
|
|
```bash
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{
|
|
"jsonrpc":"2.0",
|
|
"method":"tools/call",
|
|
"params":{
|
|
"name":"guardrail_phase_gate_check",
|
|
"arguments":{
|
|
"project_name":"my-project",
|
|
"from_phase":1,
|
|
"to_phase":2
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
2. **Complete missing deliverables:**
|
|
- See [Phase Gates](#phase-gate-requirements) section
|
|
- Submit required documents
|
|
- Obtain approvals
|
|
|
|
3. **Verify team assignments:**
|
|
```bash
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{
|
|
"jsonrpc":"2.0",
|
|
"method":"tools/call",
|
|
"params":{
|
|
"name":"guardrail_team_status",
|
|
"arguments":{
|
|
"project_name":"my-project",
|
|
"phase":"Phase 1"
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
---
|
|
|
|
### Role Already Assigned
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: TEAM-004: Person already assigned
|
|
Role 'Technical Lead' already has 'Alice Johnson' assigned
|
|
```
|
|
|
|
**Causes:**
|
|
- Attempted to assign to an occupied role
|
|
- Duplicate assignment in batch script
|
|
|
|
**Solutions:**
|
|
|
|
1. **Unassign current person:**
|
|
```bash
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{
|
|
"jsonrpc":"2.0",
|
|
"method":"tools/call",
|
|
"params":{
|
|
"name":"guardrail_team_unassign",
|
|
"arguments":{
|
|
"project_name":"my-project",
|
|
"team_id":7,
|
|
"role_name":"Technical Lead"
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
2. **Assign new person:**
|
|
```bash
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{
|
|
"jsonrpc":"2.0",
|
|
"method":"tools/call",
|
|
"params":{
|
|
"name":"guardrail_team_assign",
|
|
"arguments":{
|
|
"project_name":"my-project",
|
|
"team_id":7,
|
|
"role_name":"Technical Lead",
|
|
"person":"New Lead Name"
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
---
|
|
|
|
### API Key Authentication Failed
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: AUTH-001: Authentication required
|
|
Error: AUTH-002: Invalid API key
|
|
```
|
|
|
|
**Causes:**
|
|
- Missing Authorization header
|
|
- Expired or revoked API key
|
|
- Incorrect API key format
|
|
|
|
**Solutions:**
|
|
|
|
1. **Verify header format:**
|
|
```bash
|
|
curl -H "Authorization: Bearer YOUR_API_KEY" ...
|
|
```
|
|
|
|
2. **Generate new API key:**
|
|
```bash
|
|
python scripts/generate_api_key.py
|
|
```
|
|
|
|
3. **Check key permissions:**
|
|
```bash
|
|
python scripts/verify_api_key.py YOUR_API_KEY
|
|
```
|
|
|
|
---
|
|
|
|
## Debug Mode
|
|
|
|
Enable debug mode to get detailed logging for troubleshooting.
|
|
|
|
### Enable Debug Logging
|
|
|
|
**Option 1: Environment Variable**
|
|
```bash
|
|
export MCP_DEBUG=1
|
|
export MCP_LOG_LEVEL=debug
|
|
python mcp_server.py
|
|
```
|
|
|
|
**Option 2: Configuration File**
|
|
```json
|
|
// .mcp/config.json
|
|
{
|
|
"logging": {
|
|
"level": "debug",
|
|
"file": ".mcp/mcp.log",
|
|
"console": true
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option 3: Command Line Flag**
|
|
```bash
|
|
python mcp_server.py --debug --log-file .mcp/debug.log
|
|
```
|
|
|
|
### Debug Output Examples
|
|
|
|
**Normal Operation:**
|
|
```
|
|
[DEBUG] Received request: tools/call
|
|
[DEBUG] Tool: guardrail_team_list
|
|
[DEBUG] Parameters: {"project_name": "my-project"}
|
|
[DEBUG] Execution time: 45ms
|
|
[INFO] Response sent successfully
|
|
```
|
|
|
|
**Error Condition:**
|
|
```
|
|
[DEBUG] Received request: tools/call
|
|
[DEBUG] Tool: guardrail_team_assign
|
|
[DEBUG] Parameters: {"team_id": 99, ...}
|
|
[ERROR] Validation failed: Invalid team_id
|
|
[ERROR] Error code: TEAM-002
|
|
[DEBUG] Stack trace:
|
|
File "scripts/team_manager.py", line 45, in validate_team
|
|
raise InvalidTeamError(f"Team {team_id} not found")
|
|
[INFO] Error response sent
|
|
```
|
|
|
|
---
|
|
|
|
## Log Analysis
|
|
|
|
### Log File Locations
|
|
|
|
| Component | Log File | Description |
|
|
|-----------|----------|-------------|
|
|
| MCP Server | `.mcp/mcp.log` | Main server logs |
|
|
| Team Manager | `.mcp/team_manager.log` | Team operations |
|
|
| Validation | `.mcp/validation.log` | Guardrail checks |
|
|
| Audit | `.mcp/audit.log` | Security events |
|
|
|
|
### Log Format
|
|
|
|
```
|
|
[TIMESTAMP] [LEVEL] [COMPONENT] Message
|
|
```
|
|
|
|
**Example:**
|
|
```
|
|
2026-02-15 14:32:15 [INFO] [MCP] Server started on port 8094
|
|
2026-02-15 14:32:18 [DEBUG] [TEAM] Validating team assignment
|
|
2026-02-15 14:32:18 [ERROR] [VALID] TEAM-002: Invalid team ID
|
|
```
|
|
|
|
### Common Log Patterns
|
|
|
|
**Startup Issues:**
|
|
```bash
|
|
# Check for port binding errors
|
|
grep "Address already in use" .mcp/mcp.log
|
|
|
|
# Check for permission denied
|
|
grep "Permission denied" .mcp/mcp.log
|
|
|
|
# Check for missing files
|
|
grep "No such file" .mcp/mcp.log
|
|
```
|
|
|
|
**Authentication Issues:**
|
|
```bash
|
|
# Find failed authentication attempts
|
|
grep "AUTH-" .mcp/audit.log
|
|
|
|
# List unauthorized access attempts
|
|
grep "403\|Unauthorized" .mcp/audit.log
|
|
```
|
|
|
|
**Performance Issues:**
|
|
```bash
|
|
# Find slow requests
|
|
grep "slow\|timeout" .mcp/mcp.log
|
|
|
|
# High response times
|
|
awk '/Execution time/ && $NF > 1000 {print}' .mcp/mcp.log
|
|
```
|
|
|
|
### Log Rotation
|
|
|
|
Configure automatic log rotation to prevent disk space issues:
|
|
|
|
```bash
|
|
# Add to crontab
|
|
crontab -e
|
|
|
|
# Rotate logs daily at midnight
|
|
0 0 * * * /usr/sbin/logrotate /etc/logrotate.d/mcp
|
|
```
|
|
|
|
**logrotate configuration:**
|
|
```
|
|
# /etc/logrotate.d/mcp
|
|
/mnt/ollama/git/agent-guardrails-template/.mcp/*.log {
|
|
daily
|
|
rotate 7
|
|
compress
|
|
delaycompress
|
|
missingok
|
|
notifempty
|
|
create 644 user user
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Performance Issues
|
|
|
|
### Slow Team Operations
|
|
|
|
**Symptoms:**
|
|
- Team list takes > 5 seconds
|
|
- Batch assignments timeout
|
|
- Phase gate checks are slow
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check response times
|
|
time curl -s -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '...team_list...'
|
|
|
|
# Monitor server resources
|
|
htop
|
|
iostat -x 1
|
|
```
|
|
|
|
**Solutions:**
|
|
|
|
1. **Enable caching:**
|
|
```json
|
|
{
|
|
"cache": {
|
|
"enabled": true,
|
|
"ttl": 30
|
|
}
|
|
}
|
|
```
|
|
|
|
2. **Optimize batch operations:**
|
|
```bash
|
|
# Use parallel processing
|
|
python scripts/batch_execute.py --parallel 8
|
|
```
|
|
|
|
3. **Increase timeouts:**
|
|
```bash
|
|
curl --max-time 30 ...
|
|
```
|
|
|
|
---
|
|
|
|
### High Memory Usage
|
|
|
|
**Symptoms:**
|
|
- MCP server using > 500MB RAM
|
|
- System swapping
|
|
- Out of memory errors
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check memory usage
|
|
ps aux | grep mcp_server
|
|
free -h
|
|
|
|
# Monitor over time
|
|
watch -n 1 'ps -o pid,rss,cmd -p $(pgrep -f mcp_server)'
|
|
```
|
|
|
|
**Solutions:**
|
|
|
|
1. **Limit cache size:**
|
|
```json
|
|
{
|
|
"cache": {
|
|
"max_size": 100,
|
|
"ttl": 30
|
|
}
|
|
}
|
|
```
|
|
|
|
2. **Restart server periodically:**
|
|
```bash
|
|
# Add to cron
|
|
0 */6 * * * systemctl restart mcp-server
|
|
```
|
|
|
|
3. **Profile memory usage:**
|
|
```bash
|
|
python -m memory_profiler mcp_server.py
|
|
```
|
|
|
|
---
|
|
|
|
### Rate Limiting
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: RATE-001: Rate limit exceeded
|
|
Retry after 60 seconds
|
|
```
|
|
|
|
**Solutions:**
|
|
|
|
1. **Implement backoff:**
|
|
```python
|
|
import time
|
|
import random
|
|
|
|
def with_backoff(func, max_retries=5):
|
|
for i in range(max_retries):
|
|
try:
|
|
return func()
|
|
except RateLimitError:
|
|
wait = (2 ** i) + random.random()
|
|
time.sleep(wait)
|
|
raise MaxRetriesExceeded()
|
|
```
|
|
|
|
2. **Use batch endpoints:**
|
|
```bash
|
|
# Instead of multiple single calls
|
|
python scripts/batch_execute.py --file operations.json
|
|
```
|
|
|
|
3. **Request limit increase:**
|
|
Contact support to increase rate limits for your use case.
|
|
|
|
---
|
|
|
|
## Recovery Procedures
|
|
|
|
### Restore from Backup
|
|
|
|
**Prerequisites:**
|
|
- Backup files in `.teams/backups/`
|
|
- Valid project configuration
|
|
|
|
**Steps:**
|
|
```bash
|
|
# 1. Stop MCP server
|
|
pkill -f mcp_server
|
|
|
|
# 2. Backup current state
|
|
cp -r .teams/ .teams/emergency-backup-$(date +%Y%m%d)
|
|
|
|
# 3. Restore from backup
|
|
cp .teams/backups/project-backup-20260214.json .teams/my-project.json
|
|
|
|
# 4. Restart server
|
|
python mcp_server.py
|
|
|
|
# 5. Verify restoration
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"guardrail_team_list","arguments":{"project_name":"my-project"}}}'
|
|
```
|
|
|
|
---
|
|
|
|
### Reset Project State
|
|
|
|
**Warning:** This will remove all team assignments and reset to initial state.
|
|
|
|
```bash
|
|
# 1. Archive current state
|
|
mv .teams/my-project.json .teams/my-project-$(date +%Y%m%d).json.bak
|
|
|
|
# 2. Re-initialize project
|
|
curl -X POST http://localhost:8094/mcp/v1/message \
|
|
-d '{
|
|
"jsonrpc":"2.0",
|
|
"method":"tools/call",
|
|
"params":{
|
|
"name":"guardrail_team_init",
|
|
"arguments":{"project_name":"my-project"}
|
|
}
|
|
}'
|
|
|
|
# 3. Re-assign team members from backup reference
|
|
```
|
|
|
|
---
|
|
|
|
### Repair Corrupted Configuration
|
|
|
|
**Symptoms:**
|
|
```
|
|
Error: Invalid JSON in team configuration
|
|
Error: PROJ-002: Project configuration missing
|
|
```
|
|
|
|
**Steps:**
|
|
```bash
|
|
# 1. Validate JSON syntax
|
|
python -m json.tool .teams/my-project.json > /dev/null
|
|
|
|
# 2. If invalid, try to recover
|
|
python scripts/repair_config.py .teams/my-project.json
|
|
|
|
# 3. If recovery fails, restore from backup
|
|
cp .teams/backups/my-project.json .teams/my-project.json
|
|
```
|
|
|
|
---
|
|
|
|
### Emergency Rollback
|
|
|
|
Use when critical errors occur during batch operations:
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# emergency_rollback.sh
|
|
|
|
PROJECT_NAME="$1"
|
|
BACKUP_FILE=".teams/backups/${PROJECT_NAME}-pre-batch.json"
|
|
|
|
if [ ! -f "$BACKUP_FILE" ]; then
|
|
echo "No backup found for $PROJECT_NAME"
|
|
exit 1
|
|
fi
|
|
|
|
echo "Rolling back $PROJECT_NAME..."
|
|
cp "$BACKUP_FILE" ".teams/${PROJECT_NAME}.json"
|
|
echo "Rollback complete."
|
|
|
|
echo "Verifying..."
|
|
curl -s -X POST http://localhost:8094/mcp/v1/message \
|
|
-d "{\"jsonrpc\":\"2.0\",\"method\":\"tools/call\",\"params\":{\"name\":\"guardrail_team_list\",\"arguments\":{\"project_name\":\"$PROJECT_NAME\"}}}"
|
|
```
|
|
|
|
---
|
|
|
|
## Getting Help
|
|
|
|
### Self-Service Resources
|
|
|
|
1. **Documentation:**
|
|
- [TEAM_TOOLS.md](./TEAM_TOOLS.md) - Tool reference
|
|
- [AGENT_GUARDRAILS.md](./AGENT_GUARDRAILS.md) - Safety protocols
|
|
- [TEAM_STRUCTURE.md](./TEAM_STRUCTURE.md) - Team definitions
|
|
|
|
2. **Error Code Lookup:**
|
|
- See TEAM_TOOLS.md Error Handling section
|
|
- Search logs for error codes
|
|
|
|
3. **Community:**
|
|
- GitHub Issues: Report bugs and feature requests
|
|
- Discussions: Ask questions, share solutions
|
|
|
|
### Support Channels
|
|
|
|
| Issue Type | Channel | Response Time |
|
|
|------------|---------|---------------|
|
|
| Critical outage | Email: oncall@example.com | 15 minutes |
|
|
| Security issue | security@example.com | 4 hours |
|
|
| Feature request | GitHub Issues | 2-3 days |
|
|
| General question | GitHub Discussions | 1-2 days |
|
|
|
|
### Required Information
|
|
|
|
When reporting issues, include:
|
|
|
|
1. **Error message:** Full text or screenshot
|
|
2. **Log snippets:** Relevant sections from `.mcp/mcp.log`
|
|
3. **Steps to reproduce:** Minimal example
|
|
4. **Environment:**
|
|
```bash
|
|
python --version
|
|
uname -a
|
|
git log --oneline -1
|
|
```
|
|
5. **Configuration:** (sanitized)
|
|
```bash
|
|
cat .mcp/config.json | grep -v password
|
|
```
|
|
|
|
### Diagnostic Script
|
|
|
|
Run this script to gather diagnostic information:
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# diagnose.sh - Collect diagnostic information
|
|
|
|
echo "=== Agent Guardrails Diagnostics ==="
|
|
echo "Date: $(date)"
|
|
echo ""
|
|
|
|
echo "=== System Information ==="
|
|
uname -a
|
|
echo ""
|
|
|
|
echo "=== Python Version ==="
|
|
python --version
|
|
echo ""
|
|
|
|
echo "=== MCP Server Status ==="
|
|
pgrep -f mcp_server || echo "MCP server not running"
|
|
echo ""
|
|
|
|
echo "=== Recent Log Entries ==="
|
|
tail -50 .mcp/mcp.log 2>/dev/null || echo "No log file found"
|
|
echo ""
|
|
|
|
echo "=== Configuration ==="
|
|
ls -la .teams/ 2>/dev/null || echo "No .teams directory"
|
|
ls -la .guardrails/ 2>/dev/null || echo "No .guardrails directory"
|
|
echo ""
|
|
|
|
echo "=== Health Check ==="
|
|
curl -s http://localhost:8094/mcp/v1/health 2>/dev/null || echo "Health check failed"
|
|
echo ""
|
|
|
|
echo "=== Diagnostics Complete ==="
|
|
```
|
|
|
|
---
|
|
|
|
**Last Updated:** 2026-02-15
|
|
**Version:** 1.0
|