diff --git a/PROJECTS/advanced/ai-threat-detection/.env.example b/PROJECTS/advanced/ai-threat-detection/.env.example
index 79e2ed40..a0f16ce9 100644
--- a/PROJECTS/advanced/ai-threat-detection/.env.example
+++ b/PROJECTS/advanced/ai-threat-detection/.env.example
@@ -32,6 +32,14 @@ GEOIP_DB_PATH=/usr/share/GeoIP/GeoLite2-City.mmdb
# Nginx Log Path (inside container)
NGINX_LOG_PATH=/var/log/nginx/access.log
+# ML Auto-Training
+# Models auto-train with synthetic data on first startup (~1-2 min)
+# Set to "true" to disable auto-training and start in rules-only mode
+SKIP_AUTO_TRAIN=false
+
+# ML Model Directory (inside container)
+MODEL_DIR=/app/data/models
+
# Pipeline Tuning
RAW_QUEUE_SIZE=1000
PARSED_QUEUE_SIZE=500
@@ -39,3 +47,12 @@ FEATURE_QUEUE_SIZE=200
ALERT_QUEUE_SIZE=100
BATCH_SIZE=32
BATCH_TIMEOUT_MS=50
+
+# Dev-Log Target App
+# Start with: just devlog-up
+# Simulate traffic: just devlog-simulate mixed 100
+# The dev-log nginx writes to a shared Docker volume (vigil_dev_nginx_logs)
+# that the vigil backend automatically reads from
+#
+# To use your own nginx logs instead, edit dev.compose.yml backend volumes:
+# - /path/to/your/nginx/logs:/var/log/nginx:ro
diff --git a/PROJECTS/advanced/ai-threat-detection/README.md b/PROJECTS/advanced/ai-threat-detection/README.md
index c7748236..a27d5e79 100644
--- a/PROJECTS/advanced/ai-threat-detection/README.md
+++ b/PROJECTS/advanced/ai-threat-detection/README.md
@@ -1,50 +1,152 @@
-# AngelusVigil
+```css
+ █████╗ ███╗ ██╗ ██████╗ ███████╗██╗ ██╗ ██╗███████╗
+██╔══██╗████╗ ██║██╔════╝ ██╔════╝██║ ██║ ██║██╔════╝
+███████║██╔██╗ ██║██║ ███╗█████╗ ██║ ██║ ██║███████╗
+██╔══██║██║╚██╗██║██║ ██║██╔══╝ ██║ ██║ ██║╚════██║
+██║ ██║██║ ╚████║╚██████╔╝███████╗███████╗╚██████╔╝███████║
+╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═════╝ ╚══════╝╚══════╝ ╚═════╝ ╚══════╝
+██╗ ██╗██╗ ██████╗ ██╗██╗
+██║ ██║██║██╔════╝ ██║██║
+██║ ██║██║██║ ███╗██║██║
+╚██╗ ██╔╝██║██║ ██║██║██║
+ ╚████╔╝ ██║╚██████╔╝██║███████╗
+ ╚═══╝ ╚═╝ ╚═════╝ ╚═╝╚══════╝
+```
-> **Status: IN PROGRESS** — Phase 1 complete, Phase 2 (ML models) next
+[](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/advanced/ai-threat-detection)
+[](https://www.python.org)
+[](https://react.dev)
+[](https://www.gnu.org/licenses/agpl-3.0)
+[](https://www.docker.com)
+[](https://pytorch.org)
-AI-powered threat detection engine that analyzes web server access logs using machine learning to classify HTTP traffic as benign or malicious in real-time.
+> AI-powered threat detection engine that analyzes nginx access logs using a 3-model ML ensemble to classify HTTP traffic as benign or malicious in real time.
-Deploys as a Docker sidecar alongside any nginx-based infrastructure. Zero code changes to the monitored application.
+*This is a quick overview — learn modules with security theory, architecture deep-dives, and full walkthroughs are coming soon.*
-## Progress
+## What It Does
-| Phase | Description | Status |
-|-------|-------------|--------|
-| Phase 1 | Core pipeline, rule-based detection, API, Docker | Complete |
-| Phase 2 | ML ensemble (autoencoder + RF + IF), ONNX inference | Next |
-| Phase 3 | Production hardening, monitoring, retraining | Planned |
-| Phase 4 | Dashboard, active learning, explainability | Planned |
-
-## Tech Stack
-
-| Layer | Technology |
-|-------|-----------|
-| API | FastAPI (async) |
-| ML | PyTorch autoencoder + scikit-learn (RF + IF) |
-| Inference | ONNX Runtime (CPU) |
-| Database | PostgreSQL 18 |
-| Cache | Redis 7.4 |
-| GeoIP | MaxMind GeoLite2 |
+- 3-model ML ensemble (Autoencoder + Random Forest + Isolation Forest) exported to ONNX for fast CPU inference
+- 4-stage async pipeline: parse raw logs, extract 35-dimensional feature vectors, score with rules + ML, dispatch alerts
+- Rule engine with ModSecurity CRS-inspired patterns (SQLi, XSS, path traversal, command injection, Log4Shell, SSRF)
+- Real-time WebSocket alert feed with live severity scoring and threat classification
+- Auto-trains on first deploy with synthetic data, retrains from the dashboard using your actual stored events
+- Deploys as a Docker sidecar alongside any nginx-based infrastructure with zero code changes
## Quick Start
```bash
+git clone https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
+cd PROJECTS/advanced/ai-threat-detection
+cp .env.example .env
docker compose -f dev.compose.yml up -d
-curl http://localhost:36969/health
```
+First startup takes ~2 minutes while models train. After that, models persist to a volume and subsequent starts are instant.
+
+Once healthy, visit `http://localhost:46969` for the dashboard.
+
+> [!TIP]
+> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` to see all available commands.
+>
+> Install: `curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin`
+
+## Simulate Attacks
+
+A built-in dev-log application generates realistic nginx traffic for testing:
+
+```bash
+docker compose -f dev-log/compose.yml up -d
+python dev-log/simulate.py mixed -n 100
+```
+
+Attack modes: `normal`, `sqli`, `xss`, `traversal`, `cmdi`, `log4shell`, `ssrf`, `scanner`, `flood`, `mixed`
+
+## Stack
+
+**Backend:** FastAPI (async), PostgreSQL 18, Redis 7.4, ONNX Runtime, PyTorch, scikit-learn, MLflow
+
+**Frontend:** React 19, TypeScript, Vite, Sass, TanStack Query, Zustand
+
+**Infra:** Docker Compose, multi-stage builds, auto-training entrypoint, shared volume log tailing
+
## Architecture
-3-model ensemble (autoencoder + Random Forest + Isolation Forest) scores each request through a weighted fusion producing a unified threat score [0.0, 1.0]:
+```
+┌─────────────────────────────────────────────────────────────┐
+│ nginx logs │
+│ (shared volume) │
+└──────────────────────────┬──────────────────────────────────┘
+ │
+ PollingObserver
+ │
+ ┌───────▼───────┐
+ │ Raw Queue │
+ └───────┬───────┘
+ │
+ ┌────────────▼────────────┐
+ │ Stage 1: Parse │
+ │ nginx combined format │
+ └────────────┬────────────┘
+ │
+ ┌────────────▼────────────┐
+ │ Stage 2: Features │
+ │ 35-dim vector + GeoIP │
+ │ + windowed aggregates │
+ └────────────┬────────────┘
+ │
+ ┌────────────▼────────────┐
+ │ Stage 3: Detection │
+ │ Rules + ML Ensemble │
+ │ AE(40%) RF(40%) IF(20%)│
+ └────────────┬────────────┘
+ │
+ ┌────────────▼────────────┐
+ │ Stage 4: Dispatch │
+ │ PostgreSQL + Redis │
+ │ pub/sub → WebSocket │
+ └─────────────────────────┘
+```
-- **HIGH** (0.7+): Store + alert + block recommendation
-- **MEDIUM** (0.5-0.7): Store + monitor
-- **LOW** (<0.5): Log only
+Threat scores range from 0.0 to 1.0:
-Currently running rule-based detection (ModSecurity CRS patterns) as cold-start fallback until ML models are trained in Phase 2.
+- **HIGH** (0.7+): Stored, alerted via WebSocket, block recommendation
+- **MEDIUM** (0.5-0.7): Stored, monitored
+- **LOW** (<0.5): Logged for pattern analysis
-See `learn/` for detailed documentation.
+## ML Pipeline
+
+Models auto-train on first container startup using synthetic attack patterns (SQLi, XSS, path traversal, scanners, etc.) and are exported to ONNX. Validation gates enforce F1 >= 0.80 and PR-AUC >= 0.85 before deployment.
+
+The retrain endpoint (`POST /models/retrain`) pulls real events from the database:
+- **Reviewed events** use human-verified labels as ground truth
+- **Unreviewed events** use score-based heuristics (high score = likely attack, low score = likely normal)
+- If insufficient real data exists, synthetic samples fill the gap
+
+## API
+
+| Endpoint | Description |
+|----------|-------------|
+| `GET /health` | Health check |
+| `GET /stats` | Threat statistics and severity breakdown |
+| `GET /threats` | Paginated threat events with filters |
+| `GET /models/status` | Active models, detection mode, metrics |
+| `POST /models/retrain` | Trigger retraining from stored events |
+| `POST /ingest/batch` | Manual log line ingestion |
+| `WS /ws/alerts` | Real-time threat alert stream |
+
+All endpoints (except health and WebSocket) require `X-API-Key` header.
+
+## Status
+
+This project is fully functional and actively being polished. The core system (pipeline, ML ensemble, dashboard, real-time alerts, auto-training, attack simulation) is complete and working end to end.
+
+Learn modules are coming soon and will cover:
+- Security theory behind anomaly detection and ML-based WAFs
+- Architecture deep-dive into the async pipeline and ensemble scoring
+- Line-by-line implementation walkthrough
+- Extension challenges (GeoIP blocking, custom rule authoring, active learning)
## License
-AGPLv3 - See [LICENSE](LICENSE)
+AGPL 3.0
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/api/deps.py b/PROJECTS/advanced/ai-threat-detection/backend/app/api/deps.py
index 005c475f..ec1b6674 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/api/deps.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/api/deps.py
@@ -5,9 +5,23 @@ deps.py
from collections.abc import AsyncIterator
-from fastapi import Request
+from fastapi import Header, HTTPException, Request
from sqlalchemy.ext.asyncio import AsyncSession
+from app.config import settings
+
+
+def require_api_key(
+ x_api_key: str | None = Header(default=None, alias="X-API-Key"),
+) -> None:
+ """
+ Enforce API key authentication when api_key is configured in settings
+ """
+ if not settings.api_key:
+ return
+ if x_api_key != settings.api_key:
+ raise HTTPException(status_code=401, detail="Invalid API key")
+
async def get_session(request: Request) -> AsyncIterator[AsyncSession]:
"""
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/api/ingest.py b/PROJECTS/advanced/ai-threat-detection/backend/app/api/ingest.py
index 9eb0b95c..283d2ab0 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/api/ingest.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/api/ingest.py
@@ -5,9 +5,11 @@ ingest.py
import asyncio
-from fastapi import APIRouter, Request
+from fastapi import APIRouter, Depends, Request
from pydantic import BaseModel
+from app.api.deps import require_api_key
+
router = APIRouter(prefix="/ingest", tags=["ingest"])
@@ -19,7 +21,7 @@ class BatchIngestRequest(BaseModel):
lines: list[str]
-@router.post("/batch", status_code=200)
+@router.post("/batch", status_code=200, dependencies=[Depends(require_api_key)])
async def ingest_batch(
body: BatchIngestRequest,
request: Request,
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/api/models_api.py b/PROJECTS/advanced/ai-threat-detection/backend/app/api/models_api.py
index 4b01584c..33c9b539 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/api/models_api.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/api/models_api.py
@@ -3,19 +3,30 @@
models_api.py
"""
+import logging
import uuid
-from fastapi import APIRouter, Request
-from sqlalchemy import select
-from sqlalchemy.ext.asyncio import AsyncSession
+from fastapi import APIRouter, BackgroundTasks, Request
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+from app.config import settings
from app.models.model_metadata import ModelMetadata
+from app.models.threat_event import ThreatEvent
+
+logger = logging.getLogger(__name__)
router = APIRouter(prefix="/models", tags=["models"])
+SCORE_ATTACK_THRESHOLD = 0.5
+SCORE_NORMAL_CEILING = 0.3
+MIN_TRAINING_SAMPLES = 200
+SYNTHETIC_SUPPLEMENT_NORMAL = 500
+SYNTHETIC_SUPPLEMENT_ATTACK = 250
+
@router.get("/status")
-async def model_status(request: Request, ) -> dict[str, object]:
+async def model_status(request: Request) -> dict[str, object]:
"""
Return the status of active ML models
"""
@@ -36,18 +47,176 @@ async def model_status(request: Request, ) -> dict[str, object]:
@router.post("/retrain", status_code=202)
-async def retrain() -> dict[str, object]:
+async def retrain(
+ request: Request,
+ background_tasks: BackgroundTasks,
+) -> dict[str, object]:
"""
- Trigger an async model retraining job
+ Dispatch a model retraining job using real stored
+ threat events supplemented with synthetic data
"""
- return {
- "status": "accepted",
- "job_id": uuid.uuid4().hex,
- }
+ session_factory = getattr(request.app.state, "session_factory", None)
+ if session_factory is None:
+ return {"status": "error", "job_id": ""}
+
+ job_id = uuid.uuid4().hex
+ background_tasks.add_task(
+ _retrain_from_db,
+ job_id,
+ session_factory,
+ )
+ return {"status": "accepted", "job_id": job_id}
+
+
+async def _retrain_from_db(
+ job_id: str,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """
+ Pull stored threat events, build training arrays,
+ supplement with synthetic data if needed, and run
+ the full training pipeline
+ """
+ import asyncio
+ import dataclasses
+ from pathlib import Path
+
+ import numpy as np
+
+ from ml.orchestrator import TrainingOrchestrator
+
+ logger.info("Retrain job %s: loading stored events", job_id)
+
+ async with session_factory() as session:
+ count = (await session.execute(
+ select(func.count()).select_from(ThreatEvent)
+ )).scalar_one()
+
+ if count == 0:
+ logger.warning(
+ "Retrain job %s: no stored events, using synthetic only",
+ job_id,
+ )
+ _fallback_synthetic(job_id)
+ return
+
+ rows = (await session.execute(
+ select(ThreatEvent)
+ )).scalars().all()
+
+ vectors: list[list[float]] = []
+ labels: list[int] = []
+
+ for event in rows:
+ if not event.feature_vector:
+ continue
+
+ if event.reviewed and event.review_label:
+ label = 1 if event.review_label == "true_positive" else 0
+ elif event.threat_score >= SCORE_ATTACK_THRESHOLD:
+ label = 1
+ elif event.threat_score < SCORE_NORMAL_CEILING:
+ label = 0
+ else:
+ continue
+
+ vectors.append(event.feature_vector)
+ labels.append(label)
+
+ logger.info(
+ "Retrain job %s: %d usable events from DB "
+ "(normal=%d, attack=%d)",
+ job_id,
+ len(vectors),
+ labels.count(0),
+ labels.count(1),
+ )
+
+ from ml.synthetic import generate_mixed_dataset
+
+ if len(vectors) < MIN_TRAINING_SAMPLES:
+ syn_X, syn_y = generate_mixed_dataset(
+ SYNTHETIC_SUPPLEMENT_NORMAL,
+ SYNTHETIC_SUPPLEMENT_ATTACK,
+ )
+ X = np.concatenate([
+ np.array(vectors, dtype=np.float32),
+ syn_X,
+ ]) if vectors else syn_X
+ y = np.concatenate([
+ np.array(labels, dtype=np.int32),
+ syn_y,
+ ]) if labels else syn_y
+ logger.info(
+ "Retrain job %s: supplemented with %d synthetic samples",
+ job_id,
+ len(syn_X),
+ )
+ else:
+ X = np.array(vectors, dtype=np.float32)
+ y = np.array(labels, dtype=np.int32)
+
+ output_dir = Path(settings.model_dir)
+ loop = asyncio.get_running_loop()
+ result = await loop.run_in_executor(
+ None,
+ lambda: TrainingOrchestrator(output_dir=output_dir).run(X, y),
+ )
+
+ logger.info(
+ "Retrain job %s complete: passed_gates=%s",
+ job_id,
+ result.passed_gates,
+ )
+
+ try:
+ from cli.main import _write_metadata
+
+ metrics: dict[str, object] = (
+ dataclasses.asdict(result.ensemble_metrics)
+ if result.ensemble_metrics else {}
+ )
+ await _write_metadata(
+ output_dir,
+ len(X),
+ metrics,
+ result.mlflow_run_id,
+ result.ae_metrics.get("ae_threshold"),
+ )
+ except Exception:
+ logger.exception(
+ "Retrain job %s: failed to write metadata",
+ job_id,
+ )
+
+
+def _fallback_synthetic(job_id: str) -> None:
+ """
+ Run training with synthetic data only when no real
+ events exist
+ """
+ import subprocess
+ import sys
+
+ logger.info("Retrain job %s: falling back to synthetic training", job_id)
+ subprocess.Popen(
+ [
+ sys.executable,
+ "-m",
+ "cli.main",
+ "train",
+ "--synthetic-normal",
+ "1000",
+ "--synthetic-attack",
+ "500",
+ ],
+ start_new_session=True,
+ )
async def _get_active_models(
- session: AsyncSession, ) -> list[dict[str, object]]:
+ session: AsyncSession,
+) -> list[dict[str, object]]:
"""
Query all active model metadata records
"""
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/core/alerts/dispatcher.py b/PROJECTS/advanced/ai-threat-detection/backend/app/core/alerts/dispatcher.py
index 913c15cd..27d10f28 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/core/alerts/dispatcher.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/core/alerts/dispatcher.py
@@ -80,6 +80,7 @@ class AlertDispatcher:
alert = WebSocketAlert(
timestamp=scored.entry.timestamp,
source_ip=scored.entry.ip,
+ request_method=scored.entry.method,
request_path=scored.entry.path,
threat_score=scored.final_score,
severity=severity,
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/core/detection/rules.py b/PROJECTS/advanced/ai-threat-detection/backend/app/core/detection/rules.py
index c4400b73..7b56f605 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/core/detection/rules.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/core/detection/rules.py
@@ -18,6 +18,7 @@ from app.core.features.patterns import (
XSS,
)
from app.core.features.signatures import SCANNER_USER_AGENTS
+from app.core.detection.ensemble import classify_severity as _classify_severity
from app.core.ingestion.parsers import ParsedLogEntry
@@ -74,17 +75,6 @@ class RuleResult:
component_scores: dict[str, float] = field(default_factory=dict)
-def _classify_severity(score: float) -> str:
- """
- Map a threat score to a severity label.
- """
- if score >= 0.7:
- return "HIGH"
- if score >= 0.5:
- return "MEDIUM"
- return "LOW"
-
-
class RuleEngine:
"""
Cold-start rule-based detection engine inspired by ModSecurity CRS.
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/core/features/aggregator.py b/PROJECTS/advanced/ai-threat-detection/backend/app/core/features/aggregator.py
index 929472b9..9ddc540a 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/core/features/aggregator.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/core/features/aggregator.py
@@ -19,7 +19,7 @@ def _hash_member(value: str) -> str:
"""
Produce a compact 16-char hex digest for sorted set deduplication.
"""
- return hashlib.md5(value.encode()).hexdigest()[:16]
+ return hashlib.md5(value.encode(), usedforsecurity=False).hexdigest()[:16]
class WindowAggregator:
@@ -27,7 +27,7 @@ class WindowAggregator:
Per-IP sliding window feature aggregator backed by Redis sorted sets.
"""
- def __init__(self, redis_client: aioredis.Redis[bytes]) -> None:
+ def __init__(self, redis_client: aioredis.Redis[str]) -> None:
self._redis = redis_client
async def record_and_aggregate(
@@ -91,17 +91,18 @@ class WindowAggregator:
results = await pipe.execute()
- read_start = len(keys) * 2
- req_count_1m = results[read_start]
- req_count_5m = results[read_start + 1]
- req_count_10m = results[read_start + 2]
- unique_paths_5m = results[read_start + 3]
- unique_uas_10m = results[read_start + 4]
- statuses_5m = results[read_start + 5]
- sizes_5m = results[read_start + 6]
- methods_5m = results[read_start + 7]
- depths_5m = results[read_start + 8]
- requests_with_scores = results[read_start + 9]
+ (
+ _zadd_req, _zadd_paths, _zadd_statuses, _zadd_uas,
+ _zadd_sizes, _zadd_methods, _zadd_depths,
+ _trim_req, _trim_paths, _trim_statuses, _trim_uas,
+ _trim_sizes, _trim_methods, _trim_depths,
+ req_count_1m, req_count_5m, req_count_10m,
+ unique_paths_5m, unique_uas_10m,
+ statuses_5m, sizes_5m, methods_5m, depths_5m,
+ requests_with_scores,
+ _exp_req, _exp_paths, _exp_statuses, _exp_uas,
+ _exp_sizes, _exp_methods, _exp_depths,
+ ) = results
irt_mean, irt_std = _inter_request_time_stats(requests_with_scores)
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/pipeline.py b/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/pipeline.py
index a31650d8..ddf1cc48 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/pipeline.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/pipeline.py
@@ -81,7 +81,7 @@ class Pipeline:
def __init__(
self,
- redis_client: aioredis.Redis[bytes],
+ redis_client: aioredis.Redis[str],
rule_engine: RuleEngine,
geoip: GeoIPService | None = None,
on_result: (Callable[[ScoredRequest], Awaitable[None]] | None) = None,
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/tailer.py b/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/tailer.py
index cbc34c1a..ebc0004b 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/tailer.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/core/ingestion/tailer.py
@@ -15,7 +15,7 @@ from watchdog.events import (
FileMovedEvent,
FileSystemEventHandler,
)
-from watchdog.observers import Observer
+from watchdog.observers.polling import PollingObserver
logger = logging.getLogger(__name__)
@@ -171,7 +171,7 @@ class LogTailer:
) -> None:
self._log_path = log_path
self._handler = _LogHandler(log_path, queue, loop)
- self._observer = Observer()
+ self._observer = PollingObserver(timeout=2)
self._started = False
def start(self) -> None:
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/factory.py b/PROJECTS/advanced/ai-threat-detection/backend/app/factory.py
index 27971a6c..d689c65e 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/factory.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/factory.py
@@ -68,7 +68,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
app.state.detection_mode = "hybrid" if app.state.models_loaded else "rules"
pipeline = Pipeline(
- redis_client=redis_client, # type: ignore[arg-type]
+ redis_client=redis_client,
rule_engine=RuleEngine(),
geoip=geoip,
on_result=dispatcher.dispatch,
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/schemas/websocket.py b/PROJECTS/advanced/ai-threat-detection/backend/app/schemas/websocket.py
index 8c7a4efa..3d83cbee 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/schemas/websocket.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/schemas/websocket.py
@@ -17,6 +17,7 @@ class WebSocketAlert(BaseModel):
event: Literal["threat"] = "threat"
timestamp: datetime
source_ip: str
+ request_method: str
request_path: str
threat_score: float
severity: str
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/app/services/stats_service.py b/PROJECTS/advanced/ai-threat-detection/backend/app/services/stats_service.py
index 25c7d4e7..40255ae4 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/app/services/stats_service.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/app/services/stats_service.py
@@ -49,7 +49,7 @@ async def get_stats(
sev_rows = (await session.execute(sev_q)).all()
sev_map = {row[0]: row[1] for row in sev_rows}
- threats_detected = sev_map.get("HIGH", 0) + sev_map.get("MEDIUM", 0)
+ threats_detected = total
ip_q = (
select(ThreatEvent.source_ip,
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/cli/main.py b/PROJECTS/advanced/ai-threat-detection/backend/cli/main.py
index ab47502d..050d8db6 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/cli/main.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/cli/main.py
@@ -42,8 +42,14 @@ async def _write_metadata(
create_async_engine,
)
+ from app.models import model_metadata as _reg # noqa: F401
+ from sqlmodel import SQLModel
+
engine = create_async_engine(settings.database_url)
try:
+ async with engine.begin() as conn:
+ await conn.run_sync(SQLModel.metadata.create_all)
+
factory = async_sessionmaker(
engine,
class_=AsyncSession,
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/ml/data_loader.py b/PROJECTS/advanced/ai-threat-detection/backend/ml/data_loader.py
index 4e9454ba..8feec3f5 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/ml/data_loader.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/ml/data_loader.py
@@ -4,9 +4,10 @@ data_loader.py
"""
import logging
+import random
import re
from dataclasses import dataclass
-from datetime import datetime, UTC
+from datetime import UTC, datetime, timedelta
from pathlib import Path
import numpy as np
@@ -27,6 +28,17 @@ _DEFAULT_IP = "192.168.1.100"
_DEFAULT_UA = ("Mozilla/5.0 (compatible; Konqueror/3.5; Linux)"
" KHTML/3.5.8 (like Gecko)")
+_BASE_TIMESTAMP = datetime(2010, 6, 1, tzinfo=UTC)
+_TRAINING_WINDOW_DAYS = 90
+
+
+def _synthetic_timestamp() -> datetime:
+ """
+ Generate a realistic training timestamp spread over 90 days
+ """
+ offset_secs = random.randint(0, _TRAINING_WINDOW_DAYS * 86400)
+ return _BASE_TIMESTAMP + timedelta(seconds=offset_secs)
+
@dataclass
class CSICRequest:
@@ -146,7 +158,7 @@ def csic_to_parsed_entry(req: CSICRequest) -> ParsedLogEntry:
return ParsedLogEntry(
ip=_DEFAULT_IP,
- timestamp=datetime.now(UTC),
+ timestamp=_synthetic_timestamp(),
method=req.method,
path=req.path,
query_string=query,
diff --git a/PROJECTS/advanced/ai-threat-detection/backend/ml/orchestrator.py b/PROJECTS/advanced/ai-threat-detection/backend/ml/orchestrator.py
index 729321d9..4396b46b 100644
--- a/PROJECTS/advanced/ai-threat-detection/backend/ml/orchestrator.py
+++ b/PROJECTS/advanced/ai-threat-detection/backend/ml/orchestrator.py
@@ -5,7 +5,6 @@ orchestrator.py
import json
import logging
-import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any
@@ -135,13 +134,8 @@ class TrainingOrchestrator:
"ensemble_roc_auc": ensemble.roc_auc,
})
passed = ensemble.passed_gates
- except Exception as exc:
+ except Exception:
logger.exception("Ensemble validation failed")
- print(
- f" WARNING: validation raised"
- f" {type(exc).__name__}: {exc}",
- file=sys.stderr,
- )
ensemble = None
passed = False
diff --git a/PROJECTS/advanced/ai-threat-detection/compose.yml b/PROJECTS/advanced/ai-threat-detection/compose.yml
index d1818124..e114396d 100644
--- a/PROJECTS/advanced/ai-threat-detection/compose.yml
+++ b/PROJECTS/advanced/ai-threat-detection/compose.yml
@@ -71,7 +71,7 @@ services:
interval: 30s
timeout: 10s
retries: 3
- start_period: 10s
+ start_period: 180s
restart: always
frontend:
diff --git a/PROJECTS/advanced/ai-threat-detection/dev-log/Dockerfile b/PROJECTS/advanced/ai-threat-detection/dev-log/Dockerfile
new file mode 100644
index 00000000..b7674904
--- /dev/null
+++ b/PROJECTS/advanced/ai-threat-detection/dev-log/Dockerfile
@@ -0,0 +1,12 @@
+# ©AngelaMos | 2026
+# Dockerfile
+
+FROM python:3.14-slim
+
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/
+RUN uv pip install --system fastapi uvicorn
+
+WORKDIR /app
+COPY app.py .
+
+CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
diff --git a/PROJECTS/advanced/ai-threat-detection/dev-log/app.py b/PROJECTS/advanced/ai-threat-detection/dev-log/app.py
new file mode 100644
index 00000000..3850b1f6
--- /dev/null
+++ b/PROJECTS/advanced/ai-threat-detection/dev-log/app.py
@@ -0,0 +1,87 @@
+"""
+©AngelaMos | 2026
+app.py
+"""
+
+from fastapi import FastAPI, Request
+from fastapi.responses import HTMLResponse, JSONResponse
+
+app = FastAPI(title="DevLog Target App")
+
+USERS = [
+ {"id": 1, "name": "alice", "email": "alice@example.com", "role": "admin"},
+ {"id": 2, "name": "bob", "email": "bob@example.com", "role": "user"},
+ {"id": 3, "name": "carol", "email": "carol@example.com", "role": "user"},
+]
+
+PRODUCTS = [
+ {"id": 1, "name": "Widget", "price": 29.99},
+ {"id": 2, "name": "Gadget", "price": 49.99},
+ {"id": 3, "name": "Doohickey", "price": 9.99},
+]
+
+
+@app.get("/", response_class=HTMLResponse)
+async def index() -> str:
+ return "DevLog Target
"
+
+
+@app.get("/health")
+async def health() -> dict[str, str]:
+ return {"status": "healthy"}
+
+
+@app.get("/api/users")
+async def list_users() -> list[dict[str, object]]:
+ return USERS
+
+
+@app.get("/api/users/{user_id}")
+async def get_user(user_id: int) -> JSONResponse:
+ for u in USERS:
+ if u["id"] == user_id:
+ return JSONResponse(u)
+ return JSONResponse({"error": "not found"}, status_code=404)
+
+
+@app.post("/api/login")
+async def login(request: Request) -> JSONResponse:
+ return JSONResponse({"token": "eyJhbGciOiJIUzI1NiJ9.fake.token"})
+
+
+@app.get("/api/search")
+async def search(q: str = "") -> dict[str, object]:
+ return {"query": q, "results": [], "total": 0}
+
+
+@app.get("/api/products")
+async def list_products() -> list[dict[str, object]]:
+ return PRODUCTS
+
+
+@app.get("/api/products/{product_id}")
+async def get_product(product_id: int) -> JSONResponse:
+ for p in PRODUCTS:
+ if p["id"] == product_id:
+ return JSONResponse(p)
+ return JSONResponse({"error": "not found"}, status_code=404)
+
+
+@app.post("/api/checkout")
+async def checkout() -> dict[str, object]:
+ return {"order_id": "ORD-12345", "status": "confirmed"}
+
+
+@app.get("/admin")
+async def admin_panel() -> JSONResponse:
+ return JSONResponse({"error": "forbidden"}, status_code=403)
+
+
+@app.get("/admin/dashboard")
+async def admin_dashboard() -> JSONResponse:
+ return JSONResponse({"error": "forbidden"}, status_code=403)
+
+
+@app.get("/static/{path:path}")
+async def static_files(path: str) -> JSONResponse:
+ return JSONResponse({"error": "not found"}, status_code=404)
diff --git a/PROJECTS/advanced/ai-threat-detection/dev-log/compose.yml b/PROJECTS/advanced/ai-threat-detection/dev-log/compose.yml
new file mode 100644
index 00000000..98d26ea0
--- /dev/null
+++ b/PROJECTS/advanced/ai-threat-detection/dev-log/compose.yml
@@ -0,0 +1,38 @@
+# ©AngelaMos | 2026
+# compose.yml
+
+services:
+ app:
+ build: .
+ container_name: vigil-devlog-app
+ networks:
+ - devlog
+ healthcheck:
+ test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
+ interval: 5s
+ timeout: 3s
+ retries: 3
+ start_period: 5s
+
+ nginx:
+ image: nginx:alpine
+ container_name: vigil-devlog-nginx
+ command: sh -c "rm -f /var/log/nginx/access.log /var/log/nginx/error.log && exec nginx -g 'daemon off;'"
+ ports:
+ - "58319:80"
+ volumes:
+ - ./nginx.conf:/etc/nginx/nginx.conf:ro
+ - nginx_logs:/var/log/nginx
+ depends_on:
+ app:
+ condition: service_healthy
+ networks:
+ - devlog
+
+networks:
+ devlog:
+ driver: bridge
+
+volumes:
+ nginx_logs:
+ name: vigil_dev_nginx_logs
diff --git a/PROJECTS/advanced/ai-threat-detection/dev-log/nginx.conf b/PROJECTS/advanced/ai-threat-detection/dev-log/nginx.conf
new file mode 100644
index 00000000..7ac766c5
--- /dev/null
+++ b/PROJECTS/advanced/ai-threat-detection/dev-log/nginx.conf
@@ -0,0 +1,28 @@
+# ©AngelaMos | 2026
+# nginx.conf
+
+events {
+ worker_connections 64;
+}
+
+http {
+ access_log /var/log/nginx/access.log combined;
+ error_log /var/log/nginx/error.log warn;
+
+ upstream target_app {
+ server app:8000;
+ }
+
+ server {
+ listen 80;
+ server_name _;
+
+ location / {
+ proxy_pass http://target_app;
+ proxy_set_header Host $host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ }
+ }
+}
diff --git a/PROJECTS/advanced/ai-threat-detection/dev-log/simulate.py b/PROJECTS/advanced/ai-threat-detection/dev-log/simulate.py
new file mode 100755
index 00000000..d054c5c1
--- /dev/null
+++ b/PROJECTS/advanced/ai-threat-detection/dev-log/simulate.py
@@ -0,0 +1,298 @@
+#!/usr/bin/env python3
+"""
+©AngelaMos | 2026
+simulate.py
+"""
+
+import argparse
+import random
+import sys
+import time
+import urllib.error
+import urllib.request
+
+DEFAULT_TARGET = "http://localhost:58319"
+
+NORMAL_PATHS = [
+ "/",
+ "/health",
+ "/api/users",
+ "/api/users/1",
+ "/api/users/2",
+ "/api/users/3",
+ "/api/products",
+ "/api/products/1",
+ "/api/products/2",
+ "/api/search?q=shoes",
+ "/api/search?q=electronics",
+ "/api/search?q=sale+items",
+ "/static/css/main.css",
+ "/static/js/app.js",
+ "/static/images/logo.png",
+]
+
+SQLI_PAYLOADS = [
+ "/api/users?id=1' OR '1'='1",
+ "/api/users?id=1' OR '1'='1'--",
+ "/api/search?q=' UNION SELECT username,password FROM users--",
+ "/api/search?q='; DROP TABLE users;--",
+ "/api/users?id=1; SELECT * FROM information_schema.tables",
+ "/api/login?user=admin'--&pass=x",
+ "/api/products?id=1 UNION SELECT null,null,null",
+ "/api/search?q=' OR 1=1#",
+ "/api/users?id=0 UNION ALL SELECT concat(user,0x3a,password) FROM mysql.user",
+ "/api/search?q=1' AND (SELECT COUNT(*) FROM users) > 0--",
+ "/api/users?id=1' WAITFOR DELAY '0:0:5'--",
+ "/api/products?sort=name; INSERT INTO admin VALUES('hacker','pwned')",
+]
+
+XSS_PAYLOADS = [
+ "/api/search?q=",
+ "/api/search?q=
",
+ "/api/search?q=