161 lines
9.8 KiB
Markdown
161 lines
9.8 KiB
Markdown
# AngelusVigil: AI Threat Detection
|
|
|
|
## What This Is
|
|
|
|
AngelusVigil is a real-time threat detection engine that analyzes nginx web server logs using a 3-model ML ensemble (autoencoder + random forest + isolation forest) to classify HTTP traffic as benign or malicious. It deploys as a dockerized sidecar alongside any nginx-based infrastructure with zero code changes required. The system processes logs through a 4-stage async pipeline, extracts 35-dimensional feature vectors, scores traffic using both rule-based and ML detection, and surfaces alerts through a React dashboard with live WebSocket updates.
|
|
|
|
## Why This Matters
|
|
|
|
Web application firewalls (WAFs) catch known attack patterns, but they miss novel attacks and struggle with behavioral anomalies. In the 2017 Equifax breach (CVE-2017-5638), attackers exploited an Apache Struts vulnerability that signature-based tools missed for months. The 2021 Log4Shell vulnerability (CVE-2021-44228) spread across millions of servers before WAF rules were updated. The Capital One breach in 2019 involved a misconfigured WAF that an insider exploited to exfiltrate 100 million customer records.
|
|
|
|
ML-based detection fills these gaps by learning what "normal" looks like for your specific traffic and flagging deviations. This project builds that system from scratch.
|
|
|
|
**Real world scenarios where this applies:**
|
|
- A SaaS platform running nginx as a reverse proxy wants to detect SQL injection, path traversal, and credential stuffing without buying a commercial WAF
|
|
- A DevSecOps team needs real time alerting when attack patterns spike against their API endpoints, with enough context (GeoIP, feature vectors, matched rules) to triage quickly
|
|
- A security team wants to combine signature based detection (day 0 coverage) with ML models that improve over time as they label true/false positives from production traffic
|
|
|
|
## What You'll Learn
|
|
|
|
This project teaches you how ML-powered threat detection works at the infrastructure level. By building it yourself, you'll understand:
|
|
|
|
**Security Concepts:**
|
|
- Anomaly detection using autoencoders trained exclusively on normal traffic, so anything the model can't reconstruct well is suspicious
|
|
- Ensemble learning where multiple models vote on whether a request is malicious, reducing false positives from any single model
|
|
- Cold-start detection using ModSecurity CRS-inspired rules that provide immediate coverage before ML models are trained
|
|
- Feature engineering for HTTP traffic, turning raw log lines into 35-dimensional numeric vectors that capture request structure, behavioral patterns, and temporal signals
|
|
|
|
**Technical Skills:**
|
|
- Building async data pipelines with backpressure using `asyncio.Queue` and poison-pill shutdown propagation
|
|
- Training and exporting PyTorch and scikit-learn models to ONNX format for fast CPU inference
|
|
- Implementing sliding window aggregation with Redis sorted sets for per-IP behavioral features
|
|
- Writing a Watchdog-based file tailer that handles nginx log rotation without missing lines
|
|
|
|
**Tools and Techniques:**
|
|
- ONNX Runtime for cross-framework model serving (27% faster than native PyTorch inference)
|
|
- MLflow experiment tracking for model versioning and metric logging
|
|
- Redis pub/sub for real-time WebSocket relay across multiple backend workers
|
|
- Docker Compose orchestration of a 5-service production stack
|
|
|
|
## Prerequisites
|
|
|
|
Before starting, you should understand:
|
|
|
|
**Required knowledge:**
|
|
- Python async/await (you'll work with `asyncio.Queue`, `asyncio.Task`, and async context managers throughout the pipeline)
|
|
- Basic machine learning concepts (what training/inference means, what loss functions do, the difference between supervised and unsupervised learning)
|
|
- HTTP fundamentals (request methods, status codes, query strings, headers, how nginx access logs are structured)
|
|
- Docker and Docker Compose (the entire system runs as containers)
|
|
|
|
**Tools you'll need:**
|
|
- Docker and Docker Compose v2
|
|
- Python 3.13+ (the backend uses modern type syntax like `dict[str, float]` and `X | None`)
|
|
- Node.js 20+ and pnpm (for the React frontend)
|
|
- just (command runner, like make but better)
|
|
- uv (Python package manager)
|
|
|
|
**Helpful but not required:**
|
|
- Experience with FastAPI or any async Python web framework
|
|
- Familiarity with PyTorch or scikit-learn
|
|
- Understanding of Redis data structures (sorted sets, pub/sub)
|
|
|
|
## Quick Start
|
|
|
|
Get the project running locally:
|
|
|
|
```bash
|
|
cd PROJECTS/advanced/ai-threat-detection
|
|
|
|
# Install just if you don't have it
|
|
# https://github.com/casey/just
|
|
|
|
# One-time setup: install deps and create .env
|
|
just setup
|
|
|
|
# Edit .env with your values (at minimum set POSTGRES_PASSWORD and API_KEY)
|
|
# GEOIP credentials are optional for local dev
|
|
|
|
# Start the dev stack (postgres, redis, backend with hot-reload, frontend)
|
|
just dev-up
|
|
|
|
# In another terminal, start the dev log generator to simulate traffic
|
|
just devlog-up
|
|
|
|
# Generate some mixed traffic (normal + attacks)
|
|
just devlog-simulate mixed 200
|
|
```
|
|
|
|
Expected output: Open `http://localhost:5173` in your browser. You should see the dashboard with stat cards showing detected threats, a severity distribution bar, a live alert feed receiving WebSocket events, and ranked lists of top attacker IPs and most targeted paths.
|
|
|
|
The backend starts in rules-only mode. To enable ML detection, train the models:
|
|
|
|
```bash
|
|
just vigil-train
|
|
```
|
|
|
|
After training completes (about 2 minutes with synthetic data), restart the backend and it will load the ONNX models and switch to hybrid detection mode.
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
ai-threat-detection/
|
|
├── backend/
|
|
│ ├── app/
|
|
│ │ ├── api/ # FastAPI route handlers (health, threats, stats, models, ws)
|
|
│ │ ├── core/
|
|
│ │ │ ├── alerts/ # AlertDispatcher: store + publish scored events
|
|
│ │ │ ├── detection/ # RuleEngine, InferenceEngine, ensemble scoring
|
|
│ │ │ ├── enrichment/ # GeoIP lookups via MaxMind
|
|
│ │ │ ├── features/ # Feature extraction, Redis aggregation, encoding
|
|
│ │ │ └── ingestion/ # Log tailer, parsers, 4-stage pipeline
|
|
│ │ ├── models/ # SQLModel ORM (ThreatEvent, ModelMetadata)
|
|
│ │ ├── schemas/ # Pydantic request/response models
|
|
│ │ ├── services/ # Database query logic (threat_service, stats_service)
|
|
│ │ ├── config.py # Pydantic Settings (env vars, defaults, validation)
|
|
│ │ └── factory.py # App factory with async lifespan
|
|
│ ├── ml/ # Training pipeline (orchestrator, splitting, export, validation)
|
|
│ ├── cli/ # Typer CLI (train, retrain, replay commands)
|
|
│ ├── tests/ # pytest suite (parsers, features, detection, integration)
|
|
│ └── alembic/ # Database migrations
|
|
├── frontend/
|
|
│ ├── src/
|
|
│ │ ├── api/ # React Query hooks, Zod schemas, Axios client
|
|
│ │ ├── components/ # AlertFeed, SeverityBadge, MethodBadge, StatCard, ThreatDetail
|
|
│ │ ├── core/ # Router config, shell layout, query client
|
|
│ │ └── pages/ # Dashboard, Threats (table + filters), Models
|
|
│ └── vite.config.ts
|
|
├── dev-log/ # Synthetic nginx traffic generator for testing
|
|
├── infra/ # Dockerfiles, redis.conf, nginx.conf
|
|
├── compose.yml # Production 5-service stack
|
|
├── dev.compose.yml # Development stack with hot-reload
|
|
└── justfile # Task runner commands
|
|
```
|
|
|
|
## Next Steps
|
|
|
|
1. **Understand the concepts** - Read [01-CONCEPTS.md](./01-CONCEPTS.md) to learn anomaly detection, ensemble methods, and feature engineering for HTTP traffic
|
|
2. **Study the architecture** - Read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) to see how the 4-stage pipeline, Redis windowing, and ONNX inference fit together
|
|
3. **Walk through the code** - Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for a detailed walkthrough of the ingestion pipeline, rule engine, ML training, and frontend
|
|
4. **Extend the project** - Read [04-CHALLENGES.md](./04-CHALLENGES.md) for ideas like adding new model types, integrating with SIEM tools, or building an active learning workflow
|
|
|
|
## Common Issues
|
|
|
|
**Backend fails to start with "database connection refused"**
|
|
```
|
|
sqlalchemy.exc.OperationalError: connection to server at "localhost" ... refused
|
|
```
|
|
Solution: Make sure PostgreSQL is running. If using Docker, check `just dev-ps` to verify the postgres container is healthy. The backend depends on `postgres:service_healthy`, so it will wait, but if postgres itself failed to start, check the logs with `just dev-logs postgres`.
|
|
|
|
**"onnxruntime not installed" warning at startup**
|
|
This is expected if you haven't installed the ML dependencies. The system falls back to rules-only mode automatically. To enable ML detection, run `cd backend && uv sync --group ml` and then train the models with `just vigil-train`.
|
|
|
|
**WebSocket alerts not appearing on the dashboard**
|
|
Check that Redis is running (`just dev-logs redis`). The WebSocket relay depends on Redis pub/sub. Also verify the backend is processing logs by hitting `http://localhost:8000/health` and checking the `stats` field in the response.
|
|
|
|
## Related Projects
|
|
|
|
If you found this interesting, check out:
|
|
- [SIEM Dashboard](../../intermediate/siem-dashboard/) - Build a Security Information and Event Management dashboard with Flask, MongoDB, and Redis Streams for log aggregation and correlation
|
|
- [Honeypot Network](../honeypot-network/) - Deploy deceptive services that generate the kind of attack traffic this project detects
|
|
- [API Security Scanner](../../intermediate/api-security-scanner/) - Active vulnerability scanning that complements this project's passive detection approach
|