Cybersecurity-Projects/README.md at main

8.6 KiB

Raw Permalink Blame History

 █████╗ ███╗   ██╗ ██████╗ ███████╗██╗     ██╗   ██╗███████╗
██╔══██╗████╗  ██║██╔════╝ ██╔════╝██║     ██║   ██║██╔════╝
███████║██╔██╗ ██║██║  ███╗█████╗  ██║     ██║   ██║███████╗
██╔══██║██║╚██╗██║██║   ██║██╔══╝  ██║     ██║   ██║╚════██║
██║  ██║██║ ╚████║╚██████╔╝███████╗███████╗╚██████╔╝███████║
╚═╝  ╚═╝╚═╝  ╚═══╝ ╚═════╝ ╚══════╝╚══════╝ ╚═════╝ ╚══════╝
██╗   ██╗██╗ ██████╗ ██╗██╗
██║   ██║██║██╔════╝ ██║██║
██║   ██║██║██║  ███╗██║██║
╚██╗ ██╔╝██║██║   ██║██║██║
 ╚████╔╝ ██║╚██████╔╝██║███████╗
  ╚═══╝  ╚═╝ ╚═════╝ ╚═╝╚══════╝

AI-powered threat detection engine that analyzes nginx access logs using a 3-model ML ensemble to classify HTTP traffic as benign or malicious in real time.

Learn docs located here: Learn

What It Does

3-model ML ensemble (Autoencoder + Random Forest + Isolation Forest) exported to ONNX for fast CPU inference
4-stage async pipeline: parse raw logs, extract 35-dimensional feature vectors, score with rules + ML, dispatch alerts
Rule engine with ModSecurity CRS-inspired patterns (SQLi, XSS, path traversal, command injection, Log4Shell, SSRF)
Real-time WebSocket alert feed with live severity scoring and threat classification
Auto-trains on first deploy with synthetic data, retrains from the dashboard using your actual stored events
Deploys as a Docker sidecar alongside any nginx-based infrastructure with zero code changes

Quick Start

git clone https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
cd PROJECTS/advanced/ai-threat-detection
cp .env.example .env
docker compose -f dev.compose.yml up -d

First startup takes ~2 minutes while models train. After that, models persist to a volume and subsequent starts are instant.

Once healthy, visit http://localhost:46969 for the dashboard.

[!TIP] This project uses just as a command runner. Type just to see all available commands.

Install: curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin

Simulate Attacks

A built-in dev-log application generates realistic nginx traffic for testing:

docker compose -f dev-log/compose.yml up -d
python dev-log/simulate.py mixed -n 100

Attack modes: normal, sqli, xss, traversal, cmdi, log4shell, ssrf, scanner, flood, mixed

Stack

Backend: FastAPI (async), PostgreSQL 18, Redis 7.4, ONNX Runtime, PyTorch, scikit-learn, MLflow

Frontend: React 19, TypeScript, Vite, Sass, TanStack Query, Zustand

Infra: Docker Compose, multi-stage builds, auto-training entrypoint, shared volume log tailing

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        nginx logs                           │
│                     (shared volume)                         │
└──────────────────────────┬──────────────────────────────────┘
                           │
                    PollingObserver
                           │
                   ┌───────▼───────┐
                   │   Raw Queue   │
                   └───────┬───────┘
                           │
              ┌────────────▼────────────┐
              │     Stage 1: Parse      │
              │  nginx combined format  │
              └────────────┬────────────┘
                           │
              ┌────────────▼────────────┐
              │   Stage 2: Features     │
              │  35-dim vector + GeoIP  │
              │  + windowed aggregates  │
              └────────────┬────────────┘
                           │
              ┌────────────▼────────────┐
              │   Stage 3: Detection    │
              │  Rules + ML Ensemble    │
              │  AE(40%) RF(40%) IF(20%)│
              └────────────┬────────────┘
                           │
              ┌────────────▼────────────┐
              │   Stage 4: Dispatch     │
              │  PostgreSQL + Redis     │
              │  pub/sub → WebSocket    │
              └─────────────────────────┘

Threat scores range from 0.0 to 1.0:

HIGH (0.7+): Stored, alerted via WebSocket, block recommendation
MEDIUM (0.5-0.7): Stored, monitored
LOW (<0.5): Logged for pattern analysis

ML Pipeline

Models auto-train on first container startup using synthetic attack patterns (SQLi, XSS, path traversal, scanners, etc.) and are exported to ONNX. Validation gates enforce F1 >= 0.80 and PR-AUC >= 0.85 before deployment.

The retrain endpoint (POST /models/retrain) pulls real events from the database:

Reviewed events use human-verified labels as ground truth
Unreviewed events use score-based heuristics (high score = likely attack, low score = likely normal)
If insufficient real data exists, synthetic samples fill the gap

API

Endpoint	Description
`GET /health`	Health check
`GET /stats`	Threat statistics and severity breakdown
`GET /threats`	Paginated threat events with filters
`GET /models/status`	Active models, detection mode, metrics
`POST /models/retrain`	Trigger retraining from stored events
`POST /ingest/batch`	Manual log line ingestion
`WS /ws/alerts`	Real-time threat alert stream

All endpoints (except health and WebSocket) require X-API-Key header.

Status

This project is fully functional and actively being polished. The core system (pipeline, ML ensemble, dashboard, real-time alerts, auto-training, attack simulation) is complete and working end to end.

Learn modules are coming soon and will cover:

Security theory behind anomaly detection and ML-based WAFs
Architecture deep-dive into the async pipeline and ensemble scoring
Line-by-line implementation walkthrough
Extension challenges (GeoIP blocking, custom rule authoring, active learning)

License

AGPL 3.0

8.6 KiB Raw Permalink Blame History