165 lines
8.6 KiB
Markdown
165 lines
8.6 KiB
Markdown
```css
|
|
█████╗ ███╗ ██╗ ██████╗ ███████╗██╗ ██╗ ██╗███████╗
|
|
██╔══██╗████╗ ██║██╔════╝ ██╔════╝██║ ██║ ██║██╔════╝
|
|
███████║██╔██╗ ██║██║ ███╗█████╗ ██║ ██║ ██║███████╗
|
|
██╔══██║██║╚██╗██║██║ ██║██╔══╝ ██║ ██║ ██║╚════██║
|
|
██║ ██║██║ ╚████║╚██████╔╝███████╗███████╗╚██████╔╝███████║
|
|
╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═════╝ ╚══════╝╚══════╝ ╚═════╝ ╚══════╝
|
|
██╗ ██╗██╗ ██████╗ ██╗██╗
|
|
██║ ██║██║██╔════╝ ██║██║
|
|
██║ ██║██║██║ ███╗██║██║
|
|
╚██╗ ██╔╝██║██║ ██║██║██║
|
|
╚████╔╝ ██║╚██████╔╝██║███████╗
|
|
╚═══╝ ╚═╝ ╚═════╝ ╚═╝╚══════╝
|
|
```
|
|
|
|
[](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/advanced/ai-threat-detection)
|
|
[](https://www.python.org)
|
|
[](https://react.dev)
|
|
[](https://www.gnu.org/licenses/agpl-3.0)
|
|
[](https://www.docker.com)
|
|
[](https://pytorch.org)
|
|
|
|
> AI-powered threat detection engine that analyzes nginx access logs using a 3-model ML ensemble to classify HTTP traffic as benign or malicious in real time.
|
|
|
|
<p align="center">
|
|
<a href="https://youtu.be/H8WZYS8c78g">
|
|
<img src="https://img.shields.io/badge/Watch_on-YouTube-FF0000?logo=youtube&logoColor=white" alt="Watch on YouTube">
|
|
</a>
|
|
</p>
|
|
|
|
<p align="center">
|
|
<a href="https://youtu.be/H8WZYS8c78g">
|
|
<img src="https://img.youtube.com/vi/H8WZYS8c78g/maxresdefault.jpg" alt="Video Thumbnail" width="800">
|
|
</a>
|
|
</p>
|
|
|
|
*Learn docs located here: [Learn](https://github.com/CarterPerez-dev/Cybersecurity-Projects/edit/main/PROJECTS/advanced/ai-threat-detection/learn/)*
|
|
|
|
## What It Does
|
|
|
|
- 3-model ML ensemble (Autoencoder + Random Forest + Isolation Forest) exported to ONNX for fast CPU inference
|
|
- 4-stage async pipeline: parse raw logs, extract 35-dimensional feature vectors, score with rules + ML, dispatch alerts
|
|
- Rule engine with ModSecurity CRS-inspired patterns (SQLi, XSS, path traversal, command injection, Log4Shell, SSRF)
|
|
- Real-time WebSocket alert feed with live severity scoring and threat classification
|
|
- Auto-trains on first deploy with synthetic data, retrains from the dashboard using your actual stored events
|
|
- Deploys as a Docker sidecar alongside any nginx-based infrastructure with zero code changes
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
git clone https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
|
|
cd PROJECTS/advanced/ai-threat-detection
|
|
cp .env.example .env
|
|
docker compose -f dev.compose.yml up -d
|
|
```
|
|
|
|
First startup takes ~2 minutes while models train. After that, models persist to a volume and subsequent starts are instant.
|
|
|
|
Once healthy, visit `http://localhost:46969` for the dashboard.
|
|
|
|
> [!TIP]
|
|
> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` to see all available commands.
|
|
>
|
|
> Install: `curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin`
|
|
|
|
## Simulate Attacks
|
|
|
|
A built-in dev-log application generates realistic nginx traffic for testing:
|
|
|
|
```bash
|
|
docker compose -f dev-log/compose.yml up -d
|
|
python dev-log/simulate.py mixed -n 100
|
|
```
|
|
|
|
Attack modes: `normal`, `sqli`, `xss`, `traversal`, `cmdi`, `log4shell`, `ssrf`, `scanner`, `flood`, `mixed`
|
|
|
|
## Stack
|
|
|
|
**Backend:** FastAPI (async), PostgreSQL 18, Redis 7.4, ONNX Runtime, PyTorch, scikit-learn, MLflow
|
|
|
|
**Frontend:** React 19, TypeScript, Vite, Sass, TanStack Query, Zustand
|
|
|
|
**Infra:** Docker Compose, multi-stage builds, auto-training entrypoint, shared volume log tailing
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ nginx logs │
|
|
│ (shared volume) │
|
|
└──────────────────────────┬──────────────────────────────────┘
|
|
│
|
|
PollingObserver
|
|
│
|
|
┌───────▼───────┐
|
|
│ Raw Queue │
|
|
└───────┬───────┘
|
|
│
|
|
┌────────────▼────────────┐
|
|
│ Stage 1: Parse │
|
|
│ nginx combined format │
|
|
└────────────┬────────────┘
|
|
│
|
|
┌────────────▼────────────┐
|
|
│ Stage 2: Features │
|
|
│ 35-dim vector + GeoIP │
|
|
│ + windowed aggregates │
|
|
└────────────┬────────────┘
|
|
│
|
|
┌────────────▼────────────┐
|
|
│ Stage 3: Detection │
|
|
│ Rules + ML Ensemble │
|
|
│ AE(40%) RF(40%) IF(20%)│
|
|
└────────────┬────────────┘
|
|
│
|
|
┌────────────▼────────────┐
|
|
│ Stage 4: Dispatch │
|
|
│ PostgreSQL + Redis │
|
|
│ pub/sub → WebSocket │
|
|
└─────────────────────────┘
|
|
```
|
|
|
|
Threat scores range from 0.0 to 1.0:
|
|
|
|
- **HIGH** (0.7+): Stored, alerted via WebSocket, block recommendation
|
|
- **MEDIUM** (0.5-0.7): Stored, monitored
|
|
- **LOW** (<0.5): Logged for pattern analysis
|
|
|
|
## ML Pipeline
|
|
|
|
Models auto-train on first container startup using synthetic attack patterns (SQLi, XSS, path traversal, scanners, etc.) and are exported to ONNX. Validation gates enforce F1 >= 0.80 and PR-AUC >= 0.85 before deployment.
|
|
|
|
The retrain endpoint (`POST /models/retrain`) pulls real events from the database:
|
|
- **Reviewed events** use human-verified labels as ground truth
|
|
- **Unreviewed events** use score-based heuristics (high score = likely attack, low score = likely normal)
|
|
- If insufficient real data exists, synthetic samples fill the gap
|
|
|
|
## API
|
|
|
|
| Endpoint | Description |
|
|
|----------|-------------|
|
|
| `GET /health` | Health check |
|
|
| `GET /stats` | Threat statistics and severity breakdown |
|
|
| `GET /threats` | Paginated threat events with filters |
|
|
| `GET /models/status` | Active models, detection mode, metrics |
|
|
| `POST /models/retrain` | Trigger retraining from stored events |
|
|
| `POST /ingest/batch` | Manual log line ingestion |
|
|
| `WS /ws/alerts` | Real-time threat alert stream |
|
|
|
|
All endpoints (except health and WebSocket) require `X-API-Key` header.
|
|
|
|
## Status
|
|
|
|
This project is fully functional and actively being polished. The core system (pipeline, ML ensemble, dashboard, real-time alerts, auto-training, attack simulation) is complete and working end to end.
|
|
|
|
Learn modules are coming soon and will cover:
|
|
- Security theory behind anomaly detection and ML-based WAFs
|
|
- Architecture deep-dive into the async pipeline and ensemble scoring
|
|
- Line-by-line implementation walkthrough
|
|
- Extension challenges (GeoIP blocking, custom rule authoring, active learning)
|
|
|
|
## License
|
|
|
|
AGPL 3.0
|