feat: sbom generator & vulnerability matcher + docstrings across 6 projects
Add bomber CLI tool (Go) — scans dependencies across Go/Node/Python ecosystems, generates SPDX 2.3 and CycloneDX 1.5 SBOMs, and matches against OSV/NVD vulnerability databases with policy engine for CI/CD. Add file-level docstrings to ai-threat-detection, firewall-rule-engine, hash-cracker, linux-cis-hardening-auditor, binary-analysis-tool, and credential-enumeration.
This commit is contained in:
CarterPerez-dev
parent
f95785ac8d
commit
d277d50a93
|
|
@ -1,5 +1,15 @@
|
|||
# ©AngelaMos | 2026
|
||||
# alembic.ini
|
||||
#
|
||||
# Alembic database migration configuration
|
||||
#
|
||||
# Points script_location to the alembic/ directory and sets
|
||||
# the default asyncpg connection URL (overridden at runtime
|
||||
# by env.py from settings). Configures Python logging with
|
||||
# WARN level for root and sqlalchemy.engine, INFO for
|
||||
# alembic, all routed to a stderr console handler with
|
||||
# generic format. Connects to alembic/env.py,
|
||||
# alembic/versions/, app/config
|
||||
|
||||
[alembic]
|
||||
script_location = alembic
|
||||
|
|
|
|||
|
|
@ -1,6 +1,21 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
env.py
|
||||
|
||||
Alembic migration environment with async PostgreSQL engine
|
||||
support
|
||||
|
||||
Configures SQLModel.metadata as the target for autogenerate,
|
||||
imports model registrations (ModelMetadata, ThreatEvent) to
|
||||
ensure table definitions are available. run_migrations_
|
||||
offline generates SQL scripts without a connection. run_
|
||||
migrations_online creates an async engine with NullPool and
|
||||
executes migrations via run_sync. Mode is selected based on
|
||||
context.is_offline_mode()
|
||||
|
||||
Connects to:
|
||||
app/config - settings.database_url
|
||||
app/models - SQLModel table registrations
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,15 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__main__.py
|
||||
|
||||
Uvicorn entry point for the AngelusVigil API server
|
||||
|
||||
Launches app.main:app via uvicorn using host, port, and
|
||||
reload settings from app.config.settings
|
||||
|
||||
Connects to:
|
||||
config.py - settings.host, settings.port, settings.debug
|
||||
main.py - ASGI application instance
|
||||
"""
|
||||
|
||||
import uvicorn
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
API package containing FastAPI route modules for health,
|
||||
ingest, threats, stats, models, and websocket endpoints
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,20 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
deps.py
|
||||
|
||||
FastAPI dependency injection providers for API key
|
||||
authentication and async database sessions
|
||||
|
||||
require_api_key checks the X-API-Key header against
|
||||
settings.api_key, returning 401 if mismatched (no-op
|
||||
when api_key is unconfigured). get_session yields an
|
||||
AsyncSession from the app-level session_factory stored
|
||||
on app.state during lifespan initialization
|
||||
|
||||
Connects to:
|
||||
config.py - settings.api_key
|
||||
factory.py - app.state.session_factory
|
||||
api/ - injected via Depends() in route handlers
|
||||
"""
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
|
|
|
|||
|
|
@ -1,6 +1,21 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
health.py
|
||||
|
||||
Health and readiness probe endpoints for container
|
||||
orchestration
|
||||
|
||||
GET /health returns liveness status with uptime_seconds
|
||||
and pipeline_running flag. GET /ready checks database
|
||||
connectivity (SELECT 1) and Redis ping, reports
|
||||
models_loaded status, and returns 503 if any dependency
|
||||
is down. Both endpoints read from app.state set during
|
||||
lifespan
|
||||
|
||||
Connects to:
|
||||
factory.py - app.state.startup_time,
|
||||
pipeline_running, db_engine
|
||||
core/redis_manager - redis_manager.ping()
|
||||
"""
|
||||
|
||||
import time
|
||||
|
|
|
|||
|
|
@ -1,6 +1,21 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
ingest.py
|
||||
|
||||
Batch log ingestion endpoint for pushing raw log lines
|
||||
into the detection pipeline
|
||||
|
||||
POST /ingest/batch accepts a BatchIngestRequest (list of
|
||||
raw log line strings), pushes each into the pipeline's
|
||||
raw_queue via put_nowait, stops on QueueFull, and returns
|
||||
the count of successfully queued lines. Protected by
|
||||
require_api_key dependency
|
||||
|
||||
Connects to:
|
||||
deps.py - require_api_key
|
||||
core/ingestion/
|
||||
pipeline.py - pipeline.raw_queue
|
||||
factory.py - app.state.pipeline
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,28 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
models_api.py
|
||||
|
||||
ML model status and retraining endpoints
|
||||
|
||||
GET /models/status returns models_loaded flag, detection
|
||||
_mode (hybrid or rules), and active model metadata from
|
||||
the database. POST /models/retrain dispatches a
|
||||
background retraining job that loads stored ThreatEvents,
|
||||
labels them using review_label or score thresholds
|
||||
(SCORE_ATTACK_THRESHOLD 0.5, SCORE_NORMAL_CEILING 0.3),
|
||||
supplements with synthetic data if below MIN_TRAINING_
|
||||
SAMPLES (200), runs TrainingOrchestrator, and writes
|
||||
model metadata. _fallback_synthetic spawns a subprocess
|
||||
CLI train command when no real events exist
|
||||
|
||||
Connects to:
|
||||
config.py - settings.model_dir, ensemble
|
||||
weights
|
||||
models/model_metadata - ModelMetadata queries
|
||||
models/threat_event - ThreatEvent training data
|
||||
ml/orchestrator - TrainingOrchestrator
|
||||
ml/synthetic - generate_mixed_dataset
|
||||
cli/main - _write_metadata
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,6 +1,19 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
stats.py
|
||||
|
||||
Threat statistics endpoint returning aggregated metrics
|
||||
for a configurable time window
|
||||
|
||||
GET /stats accepts a range query parameter (default
|
||||
"24h") and delegates to stats_service.get_stats for
|
||||
database aggregation, returning a StatsResponse
|
||||
|
||||
Connects to:
|
||||
deps.py - get_session dependency
|
||||
schemas/stats - StatsResponse model
|
||||
services/stats_
|
||||
service - get_stats business logic
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
|
|
|
|||
|
|
@ -1,6 +1,23 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
threats.py
|
||||
|
||||
Threat event CRUD endpoints with filtering and
|
||||
pagination
|
||||
|
||||
GET /threats lists events with optional severity,
|
||||
source_ip, since/until datetime filters, and limit/
|
||||
offset pagination (max 100). GET /threats/{threat_id}
|
||||
fetches a single event by UUID, returning 404 if not
|
||||
found. Both delegate to threat_service for database
|
||||
queries
|
||||
|
||||
Connects to:
|
||||
deps.py - get_session dependency
|
||||
schemas/threats - ThreatEventResponse,
|
||||
ThreatListResponse
|
||||
services/threat_
|
||||
service - get_threats, get_threat_by_id
|
||||
"""
|
||||
|
||||
import uuid
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
websocket.py
|
||||
|
||||
WebSocket endpoint streaming real-time threat alerts via
|
||||
Redis pub/sub relay
|
||||
|
||||
WS /ws/alerts accepts a client connection, subscribes to
|
||||
the ALERTS_CHANNEL via a per-client Redis pubsub instance,
|
||||
and runs two concurrent tasks: _relay forwards published
|
||||
messages as WebSocket text frames, _receive drains client
|
||||
messages until disconnect. asyncio.wait with FIRST_
|
||||
COMPLETED cancels the other task on disconnect, then
|
||||
unsubscribes and closes the pubsub. Per-client subscribers
|
||||
ensure correct multi-worker behavior
|
||||
|
||||
Connects to:
|
||||
core/alerts - ALERTS_CHANNEL constant
|
||||
core/redis_manager- redis_manager.client
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,27 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
config.py
|
||||
|
||||
Pydantic-settings application configuration loaded from
|
||||
environment variables and .env file
|
||||
|
||||
Defines the Settings model with defaults for: server
|
||||
(host 0.0.0.0, port 8000, debug, log_level), database
|
||||
(postgresql+asyncpg URL), Redis URL, GeoIP MaxMind
|
||||
database path, nginx log path, pipeline queue sizes
|
||||
(raw 1000, parsed 500, feature 200, alert 100), batch
|
||||
settings (size 32, timeout 50ms), and ML configuration
|
||||
(model_dir, detection_mode, ensemble weights for
|
||||
autoencoder/random-forest/isolation-forest at 0.40/0.40
|
||||
/0.20, ae_threshold_percentile 99.5, MLflow tracking
|
||||
URI). Exports a module-level singleton settings instance
|
||||
|
||||
Connects to:
|
||||
factory.py - consumed in lifespan and create_app
|
||||
__main__.py - server host/port/reload
|
||||
core/ingestion/ - queue sizes, log path
|
||||
core/detection/ - model_dir, ensemble weights
|
||||
core/enrichment/ - geoip_db_path
|
||||
"""
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Core package containing detection, ingestion, feature
|
||||
engineering, enrichment, and alert subsystems
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Alerts package defining the ALERTS_CHANNEL constant for
|
||||
Redis pub/sub real-time threat notification
|
||||
"""
|
||||
|
||||
ALERTS_CHANNEL = "alerts"
|
||||
|
|
|
|||
|
|
@ -1,6 +1,26 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
dispatcher.py
|
||||
|
||||
Alert dispatcher routing scored threat events to storage,
|
||||
Redis pub/sub, and structured logging
|
||||
|
||||
AlertDispatcher.dispatch receives a ScoredRequest from the
|
||||
pipeline, classifies severity via classify_severity, logs
|
||||
every event, and for MEDIUM+ severity persists to
|
||||
PostgreSQL via create_threat_event and publishes a
|
||||
WebSocketAlert JSON payload to the ALERTS_CHANNEL for
|
||||
real-time WebSocket relay
|
||||
|
||||
Connects to:
|
||||
core/alerts/__init__ - ALERTS_CHANNEL
|
||||
core/detection/
|
||||
ensemble - classify_severity
|
||||
core/ingestion/
|
||||
pipeline - ScoredRequest dataclass
|
||||
schemas/websocket - WebSocketAlert model
|
||||
services/threat_
|
||||
service - create_threat_event
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Detection package containing the rule engine, ONNX
|
||||
inference engine, and ensemble scoring utilities
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,27 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
ensemble.py
|
||||
|
||||
Score normalization, fusion, and severity classification
|
||||
utilities for the ML ensemble
|
||||
|
||||
normalize_ae_score maps autoencoder reconstruction error
|
||||
to [0,1] using 2x threshold scaling. normalize_if_score
|
||||
inverts sklearn isolation forest scores to [0,1].
|
||||
fuse_scores computes a weighted average across available
|
||||
model scores. blend_scores combines ML ensemble and rule
|
||||
engine scores with configurable ml_weight (default 0.7).
|
||||
classify_severity maps unified score to HIGH (>=0.7),
|
||||
MEDIUM (>=0.5), or LOW
|
||||
|
||||
Connects to:
|
||||
core/detection/
|
||||
inference - raw model scores passed to normalizers
|
||||
core/detection/
|
||||
rules - classify_severity used for rule results
|
||||
core/ingestion/
|
||||
pipeline - fuse_scores and blend_scores in
|
||||
scoring stage
|
||||
"""
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,27 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
inference.py
|
||||
|
||||
ONNX-based inference engine for the 3-model ML ensemble
|
||||
|
||||
InferenceEngine loads autoencoder (ae.onnx), random
|
||||
forest (rf.onnx), and isolation forest (if.onnx) sessions
|
||||
plus RobustScaler parameters (scaler.json) and anomaly
|
||||
threshold (threshold.json) from a model directory. predict
|
||||
runs all 3 models on a batch of feature vectors: applies
|
||||
_scale_for_ae to normalize autoencoder input, computes
|
||||
reconstruction MSE for ae scores, extracts attack
|
||||
probability from skl2onnx RF output format via _extract_
|
||||
rf_proba, and returns raw IF decision scores. Returns
|
||||
None when models are unavailable. Each ONNX session uses
|
||||
single-threaded execution (inter/intra_op_num_threads=1)
|
||||
|
||||
Connects to:
|
||||
config.py - settings.model_dir
|
||||
factory.py - _load_inference_engine at startup
|
||||
core/ingestion/
|
||||
pipeline - batch inference in scoring stage
|
||||
ml/export_onnx - produces the ONNX model files
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,32 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
rules.py
|
||||
|
||||
Cold-start rule-based detection engine inspired by
|
||||
ModSecurity Core Rule Set
|
||||
|
||||
RuleEngine.score_request evaluates requests against 7
|
||||
regex-based _PatternRules (LOG4SHELL 0.95, COMMAND_
|
||||
INJECTION 0.90, SQL_INJECTION 0.85, XSS 0.80, FILE_
|
||||
INCLUSION 0.75, SSRF 0.70, PATH_TRAVERSAL 0.60),
|
||||
double-encoding detection (0.40), scanner user-agent
|
||||
signature matching (0.35), and 2 _ThresholdRules
|
||||
(RATE_ANOMALY >100 req/min 0.30, HIGH_ERROR_RATE >50%
|
||||
0.25). Final score takes the highest match plus 0.05
|
||||
boost per additional rule, capped at 1.0. Returns a
|
||||
RuleResult with threat_score, severity, matched_rules,
|
||||
and component_scores
|
||||
|
||||
Connects to:
|
||||
core/features/
|
||||
patterns - compiled regex patterns (SQLI,
|
||||
XSS, LOG4SHELL, etc.)
|
||||
core/features/
|
||||
signatures - SCANNER_USER_AGENTS list
|
||||
core/detection/
|
||||
ensemble - classify_severity
|
||||
core/ingestion/
|
||||
parsers - ParsedLogEntry
|
||||
"""
|
||||
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Enrichment package providing GeoIP lookup services for
|
||||
IP-to-location resolution
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,24 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
geoip.py
|
||||
|
||||
Async GeoIP lookup service backed by MaxMind GeoLite2-City
|
||||
database
|
||||
|
||||
GeoIPService loads a .mmdb reader on init, returning None
|
||||
for missing databases. lookup resolves an IP to a GeoResult
|
||||
(country ISO code, city, lat, lon), skipping private/
|
||||
loopback addresses and unknown entries. swap_reader
|
||||
atomically replaces the database reader for hot-reload
|
||||
after .mmdb updates. All blocking geoip2 calls run in a
|
||||
thread via asyncio.to_thread
|
||||
|
||||
Connects to:
|
||||
config.py - settings.geoip_db_path
|
||||
factory.py - initialized and closed in
|
||||
lifespan
|
||||
core/ingestion/
|
||||
pipeline - lookup called in feature_worker
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Feature engineering package with per-request extraction,
|
||||
windowed aggregation, encoding, patterns, and signatures
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,27 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
aggregator.py
|
||||
|
||||
Per-IP sliding window feature aggregator backed by Redis
|
||||
sorted sets
|
||||
|
||||
WindowAggregator.record_and_aggregate records each request
|
||||
into 7 Redis sorted sets (requests, paths, statuses, UAs,
|
||||
sizes, methods, depths) keyed by IP, trims entries older
|
||||
than KEY_TTL (900s), and computes 12 windowed features in
|
||||
a single pipelined round-trip: req_count at 1m/5m/10m
|
||||
windows, error_rate_5m (4xx/5xx ratio), unique_paths_5m,
|
||||
unique_uas_10m, method_entropy_5m (Shannon), avg_response
|
||||
_size_5m, status_diversity_5m (distinct codes), path_depth
|
||||
_variance_5m, and inter_request_time mean/std in ms.
|
||||
Members are MD5-hashed for deduplication where needed
|
||||
|
||||
Connects to:
|
||||
core/ingestion/
|
||||
pipeline - called in feature_worker stage
|
||||
core/features/
|
||||
mappings - WINDOWED_FEATURE_NAMES defines the
|
||||
12 windowed feature keys
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
|
|
|
|||
|
|
@ -1,6 +1,23 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
encoder.py
|
||||
|
||||
Feature vector encoder transforming a combined feature
|
||||
dict into a 35-element float vector for ML inference
|
||||
|
||||
encode_for_inference iterates FEATURE_ORDER, applying
|
||||
boolean 0/1 encoding for 7 BOOLEAN_FEATURES, ordinal
|
||||
lookup via CATEGORICAL_ENCODERS for http_method, status_
|
||||
class, and file_extension, deterministic country code
|
||||
encoding via _encode_country (A-Z ordinal to 1-676), and
|
||||
direct float cast for all numeric features
|
||||
|
||||
Connects to:
|
||||
core/features/
|
||||
mappings - FEATURE_ORDER, BOOLEAN_FEATURES,
|
||||
CATEGORICAL_ENCODERS
|
||||
core/ingestion/
|
||||
pipeline - called after feature merge
|
||||
"""
|
||||
|
||||
from app.core.features.mappings import (
|
||||
|
|
|
|||
|
|
@ -1,6 +1,31 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
extractor.py
|
||||
|
||||
Stateless per-request feature extraction producing 23
|
||||
features from a parsed log entry
|
||||
|
||||
extract_request_features computes: http_method, path_depth,
|
||||
path_entropy (Shannon), path_length, query_string_length,
|
||||
query_param_count, has_encoded_chars, has_double_encoding,
|
||||
status_code, status_class (Nxx), response_size, hour_of_
|
||||
day, day_of_week, is_weekend, ua_length, ua_entropy,
|
||||
is_known_bot, is_known_scanner, has_attack_pattern,
|
||||
special_char_ratio, file_extension, country_code, and
|
||||
is_private_ip. Pattern detection uses compiled regexes
|
||||
from patterns module, bot/scanner detection uses signature
|
||||
sets
|
||||
|
||||
Connects to:
|
||||
core/features/
|
||||
patterns - ATTACK_COMBINED, DOUBLE_ENCODED,
|
||||
ENCODED_CHARS
|
||||
core/features/
|
||||
signatures - BOT_USER_AGENTS, SCANNER_USER_AGENTS
|
||||
core/ingestion/
|
||||
parsers - ParsedLogEntry input
|
||||
core/ingestion/
|
||||
pipeline - called in feature_worker stage
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
|
|
|
|||
|
|
@ -1,6 +1,25 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
mappings.py
|
||||
|
||||
Feature encoding mappings, canonical feature order, and
|
||||
classification constants for the 35-feature ML input spec
|
||||
|
||||
Defines METHOD_MAP (7 HTTP verbs), STATUS_CLASS_MAP (5
|
||||
response classes), EXTENSION_MAP (25 file extensions),
|
||||
FEATURE_ORDER (35-element canonical list: 23 per-request
|
||||
+ 12 windowed), CATEGORICAL_ENCODERS routing http_method/
|
||||
status_class/file_extension to their ordinal maps,
|
||||
WINDOWED_FEATURE_NAMES (last 12 features), and BOOLEAN_
|
||||
FEATURES (7 binary flags). These mappings ensure training
|
||||
and inference use identical encoding
|
||||
|
||||
Connects to:
|
||||
core/features/
|
||||
encoder - FEATURE_ORDER, BOOLEAN_FEATURES,
|
||||
CATEGORICAL_ENCODERS
|
||||
ml/data_loader - FEATURE_ORDER for column alignment
|
||||
ml/synthetic - FEATURE_ORDER for sample generation
|
||||
"""
|
||||
|
||||
METHOD_MAP: dict[str, int] = {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,32 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
patterns.py
|
||||
|
||||
Compiled regex patterns for web attack detection covering
|
||||
7 OWASP categories plus encoding anomalies
|
||||
|
||||
Defines case-insensitive compiled patterns for: SQLI
|
||||
(union select, sleep, benchmark, information_schema, hex
|
||||
literals, comment injection), XSS (script tags, event
|
||||
handlers, javascript/vbscript URIs, DOM sinks like
|
||||
document.cookie/write, eval/alert/prompt), PATH_TRAVERSAL
|
||||
(../ sequences, %2e encoding, sensitive file paths like
|
||||
etc/passwd, .git/config, .env), COMMAND_INJECTION
|
||||
(semicolon/pipe chaining to shell commands, $() and
|
||||
backtick substitution, ${} expansion), FILE_INCLUSION
|
||||
(php://, file://, data://, phar:// wrapper schemes), SSRF
|
||||
(cloud metadata IPs 169.254.169.254, localhost with paths,
|
||||
dict:// and gopher://), LOG4SHELL (${jndi, ${lower, ${:-
|
||||
patterns). Also provides ENCODED_CHARS, DOUBLE_ENCODED for
|
||||
evasion detection, and ATTACK_COMBINED unioning all 7
|
||||
patterns
|
||||
|
||||
Connects to:
|
||||
core/features/
|
||||
extractor - ATTACK_COMBINED, DOUBLE_ENCODED,
|
||||
ENCODED_CHARS
|
||||
core/detection/
|
||||
rules - individual patterns for scored rules
|
||||
"""
|
||||
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,6 +1,23 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
signatures.py
|
||||
|
||||
User-agent signature sets for bot and security scanner
|
||||
detection
|
||||
|
||||
BOT_USER_AGENTS contains 34 lowercase search engine and
|
||||
crawler identifiers (googlebot, bingbot, gptbot, claudebot,
|
||||
etc.) for benign bot classification. SCANNER_USER_AGENTS
|
||||
contains 41 lowercase security tool signatures (nikto,
|
||||
sqlmap, nmap, burp, nuclei, metasploit, hydra, etc.) for
|
||||
hostile scanner detection. Both are frozensets matched via
|
||||
substring search against lowercased user-agent strings
|
||||
|
||||
Connects to:
|
||||
core/features/
|
||||
extractor - is_known_bot, is_known_scanner features
|
||||
core/detection/
|
||||
rules - SCANNER_USER_AGENTS for UA rule scoring
|
||||
"""
|
||||
|
||||
BOT_USER_AGENTS: frozenset[str] = frozenset({
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Ingestion package with log parsing, file tailing, and the
|
||||
four-stage async processing pipeline
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,28 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
parsers.py
|
||||
|
||||
Nginx combined-format log line parser with fast string-
|
||||
split primary and compiled regex fallback
|
||||
|
||||
ParsedLogEntry is a frozen slotted dataclass holding ip,
|
||||
timestamp, method, path, query_string, status_code,
|
||||
response_size, referer, user_agent, and raw_line.
|
||||
parse_combined tries _parse_split first (splitting on
|
||||
quote boundaries for speed), falling back to _parse_regex
|
||||
with a compiled _COMBINED_RE pattern. Both extract the
|
||||
request line, split URI into path and query_string, parse
|
||||
timestamp via strptime with timezone, and handle dash
|
||||
placeholders for size and referer
|
||||
|
||||
Connects to:
|
||||
core/ingestion/
|
||||
pipeline - parse_combined in parse_worker stage
|
||||
core/detection/
|
||||
rules - ParsedLogEntry consumed by RuleEngine
|
||||
core/features/
|
||||
extractor - ParsedLogEntry consumed by feature
|
||||
extraction
|
||||
"""
|
||||
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,6 +1,34 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
pipeline.py
|
||||
|
||||
Four-stage async pipeline transforming raw nginx log lines
|
||||
into scored threat candidates
|
||||
|
||||
Stage 1 (_parse_worker): parses raw lines via parse_
|
||||
combined into ParsedLogEntry. Stage 2 (_feature_worker):
|
||||
enriches with GeoIP lookup, extracts 23 per-request
|
||||
features, aggregates 12 windowed features via Redis-backed
|
||||
WindowAggregator, and encodes the merged 35-dim float
|
||||
vector. Stage 3 (_detection_worker): scores via RuleEngine,
|
||||
optionally runs ML ensemble inference (normalize AE/IF
|
||||
scores, fuse with configurable weights, blend with rule
|
||||
score at 0.7 ML weight). Stage 4 (_dispatch_worker):
|
||||
forwards ScoredRequests via the on_result callback. Stages
|
||||
are connected by sized asyncio.Queues with poison-pill
|
||||
shutdown propagation. EnrichedRequest and ScoredRequest
|
||||
dataclasses carry data between stages
|
||||
|
||||
Connects to:
|
||||
core/ingestion/parsers - parse_combined
|
||||
core/enrichment/geoip - GeoIPService.lookup
|
||||
core/features/extractor - extract_request_features
|
||||
core/features/aggregator - WindowAggregator
|
||||
core/features/encoder - encode_for_inference
|
||||
core/detection/rules - RuleEngine.score_request
|
||||
core/detection/inference - InferenceEngine.predict
|
||||
core/detection/ensemble - normalize/fuse/blend scores
|
||||
core/alerts/dispatcher - on_result callback
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,26 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
tailer.py
|
||||
|
||||
Watchdog-based nginx log file tailer with rotation
|
||||
detection pushing raw lines into an asyncio queue
|
||||
|
||||
_LogHandler extends FileSystemEventHandler to tail a single
|
||||
target file: _open_target seeks to EOF, on_modified reads
|
||||
new lines via _read_new_lines and checks inode changes for
|
||||
rotation, on_moved handles rename-based rotation (access
|
||||
.log -> access.log.1), on_created handles new-file
|
||||
rotation. Lines are pushed via call_soon_threadsafe into
|
||||
the asyncio queue, with QueueFull drops logged. LogTailer
|
||||
wraps _LogHandler with a PollingObserver (2s interval)
|
||||
watching the target's parent directory, providing start/
|
||||
stop lifecycle and is_active property
|
||||
|
||||
Connects to:
|
||||
factory.py - started/stopped in lifespan
|
||||
core/ingestion/
|
||||
pipeline - feeds pipeline.raw_queue
|
||||
config.py - settings.nginx_log_path
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,21 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
redis_manager.py
|
||||
|
||||
Async Redis connection lifecycle manager with module-level
|
||||
singleton
|
||||
|
||||
RedisManager wraps redis.asyncio connection creation
|
||||
(from_url with decode_responses), graceful close, client
|
||||
property access, and PING health check. The module
|
||||
exports redis_manager as a singleton used by factory
|
||||
lifespan, alert dispatcher, and websocket endpoint
|
||||
|
||||
Connects to:
|
||||
config.py - settings.redis_url
|
||||
factory.py - connect/disconnect in lifespan
|
||||
api/websocket - client for pub/sub
|
||||
api/health - ping() for readiness probe
|
||||
"""
|
||||
|
||||
import redis.asyncio as aioredis
|
||||
|
|
|
|||
|
|
@ -1,6 +1,36 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
factory.py
|
||||
|
||||
FastAPI application factory with async lifespan managing
|
||||
database, Redis, pipeline, and ML model initialization
|
||||
|
||||
lifespan creates the async SQLAlchemy engine and session
|
||||
factory, runs SQLModel.metadata.create_all, connects
|
||||
Redis, initializes GeoIPService, constructs the Alert
|
||||
Dispatcher, attempts to load the ONNX InferenceEngine
|
||||
(falling back to rules-only mode), builds the Pipeline
|
||||
with configured queue sizes and ensemble weights, starts
|
||||
the LogTailer if the nginx log directory exists, and
|
||||
stores all components on app.state. On shutdown it stops
|
||||
the tailer, pipeline, GeoIP, Redis, and disposes the DB
|
||||
engine. _load_inference_engine lazily imports onnxruntime
|
||||
-backed InferenceEngine, returning None if the dependency
|
||||
is missing or no models exist. create_app assembles the
|
||||
FastAPI instance and mounts all six API routers (health,
|
||||
ingest, threats, stats, models, websocket)
|
||||
|
||||
Connects to:
|
||||
config.py - settings for all config values
|
||||
core/ingestion/pipeline - Pipeline
|
||||
core/ingestion/tailer - LogTailer
|
||||
core/detection/rules - RuleEngine
|
||||
core/detection/inference- InferenceEngine (optional)
|
||||
core/alerts/dispatcher - AlertDispatcher
|
||||
core/enrichment/geoip - GeoIPService
|
||||
core/redis_manager - redis_manager
|
||||
api/ - all route modules
|
||||
models/ - SQLModel registration
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,11 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
main.py
|
||||
|
||||
ASGI application instance created by the factory
|
||||
|
||||
Connects to:
|
||||
factory.py - create_app builds the FastAPI instance
|
||||
"""
|
||||
|
||||
from app.factory import create_app
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Models package exporting SQLModel table classes for
|
||||
ThreatEvent and ModelMetadata
|
||||
"""
|
||||
|
||||
from app.models.model_metadata import ModelMetadata
|
||||
|
|
|
|||
|
|
@ -1,6 +1,18 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
base.py
|
||||
|
||||
Abstract SQLModel base class providing UUID primary key
|
||||
and timezone-aware created_at timestamp
|
||||
|
||||
TimestampedModel defines id as a uuid4 primary key and
|
||||
created_at as a DateTime(timezone=True) column with
|
||||
CURRENT_TIMESTAMP server default. All domain models
|
||||
inherit from this base
|
||||
|
||||
Connects to:
|
||||
models/threat_event - ThreatEvent inherits
|
||||
models/model_metadata - ModelMetadata inherits
|
||||
"""
|
||||
|
||||
import uuid
|
||||
|
|
|
|||
|
|
@ -1,6 +1,21 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
model_metadata.py
|
||||
|
||||
SQLModel table tracking ML model versions, training
|
||||
metrics, and deployment status
|
||||
|
||||
ModelMetadata stores model_type, version, training_samples,
|
||||
metrics (JSON), artifact_path, is_active flag, optional
|
||||
mlflow_run_id, threshold, and notes. A partial index on
|
||||
model_type filtered by is_active=TRUE enables fast lookup
|
||||
of the currently deployed model per type
|
||||
|
||||
Connects to:
|
||||
models/base - inherits TimestampedModel
|
||||
api/models_api - queried for /models/status,
|
||||
written after retrain
|
||||
cli/main - _write_metadata inserts records
|
||||
"""
|
||||
|
||||
from sqlalchemy import Column, Index, JSON, text
|
||||
|
|
|
|||
|
|
@ -1,6 +1,25 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
threat_event.py
|
||||
|
||||
SQLModel table for detected threat events with full
|
||||
request context and ML metadata
|
||||
|
||||
ThreatEvent stores source_ip, request_method, request_path,
|
||||
status_code, response_size, user_agent, threat_score,
|
||||
severity, component_scores (JSON), geo fields (country,
|
||||
city, lat, lon), feature_vector (JSON float array),
|
||||
matched_rules (JSON string array), model_version,
|
||||
reviewed flag, and review_label for analyst feedback.
|
||||
Indexed on created_at, source_ip, severity, threat_score,
|
||||
and a partial index on reviewed=FALSE for triage queries
|
||||
|
||||
Connects to:
|
||||
models/base - inherits TimestampedModel
|
||||
services/threat_service - CRUD operations
|
||||
api/models_api - training data source for
|
||||
retrain
|
||||
core/alerts/dispatcher - persisted on MEDIUM+ severity
|
||||
"""
|
||||
|
||||
from sqlalchemy import (
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Pydantic schemas package for API request/response
|
||||
validation across stats, threats, and websocket endpoints
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,18 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
stats.py
|
||||
|
||||
Pydantic response models for the /stats endpoint
|
||||
|
||||
SeverityBreakdown holds high/medium/low threat counts.
|
||||
IPStatEntry and PathStatEntry pair a source_ip or path
|
||||
with a count. StatsResponse aggregates time_range,
|
||||
threats_stored, threats_detected, severity_breakdown,
|
||||
top_source_ips (top 10), and top_attacked_paths (top 10)
|
||||
|
||||
Connects to:
|
||||
api/stats - StatsResponse as response_model
|
||||
services/stats_service - constructs StatsResponse
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
|
|
|||
|
|
@ -1,6 +1,21 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
threats.py
|
||||
|
||||
Pydantic response models for the /threats endpoints
|
||||
|
||||
GeoInfo holds optional country, city, lat, lon from GeoIP
|
||||
lookups. ThreatEventResponse is the full event schema with
|
||||
UUID id, timestamps, request details, threat_score,
|
||||
severity (Literal HIGH/MEDIUM/LOW), component_scores,
|
||||
geo info, matched_rules, model_version, and review status
|
||||
(from_attributes enabled for ORM conversion). Threat
|
||||
ListResponse wraps paginated items with total/limit/offset
|
||||
|
||||
Connects to:
|
||||
api/threats - response_model for list and
|
||||
detail endpoints
|
||||
services/threat_service - _to_response builds these
|
||||
"""
|
||||
|
||||
import uuid
|
||||
|
|
|
|||
|
|
@ -1,6 +1,18 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
websocket.py
|
||||
|
||||
Pydantic model for real-time WebSocket threat alert
|
||||
payloads
|
||||
|
||||
WebSocketAlert carries event type (Literal "threat"),
|
||||
timestamp, source_ip, request_method, request_path,
|
||||
threat_score, severity, and component_scores. Serialized
|
||||
via model_dump_json for Redis pub/sub broadcast
|
||||
|
||||
Connects to:
|
||||
core/alerts/dispatcher - constructs and publishes alerts
|
||||
api/websocket - relayed to connected clients
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Service layer package with threat event CRUD and
|
||||
statistics aggregation business logic
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
stats_service.py
|
||||
|
||||
Threat statistics aggregation service computing time-
|
||||
windowed metrics from stored events
|
||||
|
||||
get_stats accepts a time_range string (1h, 6h, 24h, 7d,
|
||||
30d) mapped to timedeltas via _RANGE_MAP, queries threat
|
||||
events since the cutoff, and returns a StatsResponse with
|
||||
total count, severity breakdown (HIGH/MEDIUM/LOW counts
|
||||
via GROUP BY), top 10 source IPs, and top 10 attacked
|
||||
paths ordered by frequency
|
||||
|
||||
Connects to:
|
||||
models/threat_event - ThreatEvent queries
|
||||
schemas/stats - StatsResponse, SeverityBreakdown,
|
||||
IPStatEntry, PathStatEntry
|
||||
api/stats - called from GET /stats endpoint
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, UTC
|
||||
|
|
|
|||
|
|
@ -1,6 +1,29 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
threat_service.py
|
||||
|
||||
Threat event CRUD service for database persistence and
|
||||
retrieval
|
||||
|
||||
get_threats builds a filtered, paginated query with
|
||||
optional severity, source_ip, since/until datetime
|
||||
filters, ordered by created_at DESC. get_threat_by_id
|
||||
fetches a single event by UUID. create_threat_event
|
||||
persists a ScoredRequest as a ThreatEvent with full
|
||||
request context, GeoIP data, feature vector, matched
|
||||
rules, and severity classification. _to_response converts
|
||||
ThreatEvent ORM models to ThreatEventResponse schemas
|
||||
with nested GeoInfo
|
||||
|
||||
Connects to:
|
||||
models/threat_event - ThreatEvent table operations
|
||||
schemas/threats - ThreatEventResponse, GeoInfo,
|
||||
ThreatListResponse
|
||||
core/detection/ensemble - classify_severity for create
|
||||
core/ingestion/pipeline - ScoredRequest input type
|
||||
api/threats - called from list/detail
|
||||
endpoints
|
||||
core/alerts/dispatcher - called on MEDIUM+ dispatch
|
||||
"""
|
||||
|
||||
import uuid
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
CLI package providing the Typer-based vigil command-line
|
||||
interface for server, training, replay, and diagnostics
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,29 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
main.py
|
||||
|
||||
Typer CLI application with serve, train, replay, config,
|
||||
and health commands
|
||||
|
||||
serve launches uvicorn with configurable host/port/reload.
|
||||
train loads CSIC 2010 dataset and/or synthetic data, runs
|
||||
TrainingOrchestrator, exports ONNX models, and writes
|
||||
metadata to the database via _write_metadata (creates an
|
||||
async engine, calls save_model_metadata). replay sends
|
||||
historical log lines in batches to a running server's
|
||||
/ingest/batch endpoint via httpx. config prints all
|
||||
settings with secrets redacted (_redact_url masks
|
||||
credentials in database URLs). health pings /health and
|
||||
displays status, uptime, and pipeline state
|
||||
|
||||
Connects to:
|
||||
app/config - settings for serve defaults
|
||||
app/main - uvicorn target "app.main:app"
|
||||
ml/orchestrator - TrainingOrchestrator for train
|
||||
ml/data_loader - load_csic_dataset for CSIC data
|
||||
ml/synthetic - generate_mixed_dataset
|
||||
ml/metadata - save_model_metadata
|
||||
api/ingest - /ingest/batch for replay
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,4 +1,7 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
ML package with autoencoder, classifier training, ONNX
|
||||
export, data loading, experiment tracking, and validation
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
autoencoder.py
|
||||
|
||||
PyTorch symmetric autoencoder for HTTP request anomaly
|
||||
detection
|
||||
|
||||
ThreatAutoencoder has a 35->24->12->6 encoder and 6->12
|
||||
->24->35 decoder with BatchNorm1d, LeakyReLU(0.2), and
|
||||
Dropout(0.2) between each linear layer. Trained on normal
|
||||
traffic only so that high reconstruction error (compute_
|
||||
reconstruction_error via per-sample MSE) indicates
|
||||
anomalous requests. encode/decode expose bottleneck access
|
||||
for analysis
|
||||
|
||||
Connects to:
|
||||
ml/export_onnx - exported to ae.onnx
|
||||
ml/orchestrator - trained in _train_autoencoder
|
||||
ml/scaler - input normalized before training
|
||||
"""
|
||||
|
||||
import torch
|
||||
|
|
|
|||
|
|
@ -1,6 +1,28 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
data_loader.py
|
||||
|
||||
CSIC 2010 HTTP dataset loader with feature extraction for
|
||||
ML training
|
||||
|
||||
parse_csic_file reads a CSIC dataset file, splits on HTTP
|
||||
request line boundaries, and produces CSICRequest objects
|
||||
(method, path, query_string, headers, body, label).
|
||||
csic_to_parsed_entry converts CSICRequests to
|
||||
ParsedLogEntrys with synthetic defaults (private IP,
|
||||
random timestamp over 90 days, 200 status). load_csic_
|
||||
dataset loads normal (label=0) and attack (label=1)
|
||||
files, extracts 23 per-request features, zeros 12
|
||||
windowed features, encodes to 35-dim vectors, and returns
|
||||
(X, y) numpy arrays. load_csic_normal loads a single
|
||||
normal-only file
|
||||
|
||||
Connects to:
|
||||
core/features/extractor - extract_request_features
|
||||
core/features/encoder - encode_for_inference
|
||||
core/features/mappings - WINDOWED_FEATURE_NAMES
|
||||
core/ingestion/parsers - ParsedLogEntry
|
||||
cli/main - loaded in train command
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
download_csic.py
|
||||
|
||||
CSIC 2010 dataset downloader with progress display and
|
||||
integrity checking
|
||||
|
||||
download_csic fetches normalTrafficTraining.txt, normal
|
||||
TrafficTest.txt, and anomalousTrafficTest.txt from the
|
||||
Universidad de la Republica GitLab mirror via httpx
|
||||
streaming, writing to data/datasets/csic2010/. Skips
|
||||
files that already exist above MIN_FILE_BYTES (1MB).
|
||||
Shows download progress (percentage or MB), computes
|
||||
SHA-256 via _compute_sha256, and warns on suspiciously
|
||||
small downloads
|
||||
|
||||
Connects to:
|
||||
ml/data_loader - downloaded files consumed by
|
||||
parse_csic_file
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
|
|
|
|||
|
|
@ -1,6 +1,19 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
experiment.py
|
||||
|
||||
MLflow experiment context manager with automatic system
|
||||
metadata logging
|
||||
|
||||
VigilExperiment wraps mlflow.start_run/end_run as a context
|
||||
manager, recording Python version, platform, and git commit
|
||||
hash on entry, and setting status/error tags on exit.
|
||||
Provides log_params, log_metrics (with optional step), and
|
||||
log_artifact convenience methods. _get_git_hash shells out
|
||||
to git rev-parse --short HEAD
|
||||
|
||||
Connects to:
|
||||
ml/orchestrator - used to wrap the full training run
|
||||
"""
|
||||
|
||||
import platform
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
export_onnx.py
|
||||
|
||||
ONNX model export functions for the 3-model ML ensemble
|
||||
|
||||
export_autoencoder converts a PyTorch ThreatAutoencoder to
|
||||
ONNX with dynamic batch dimension, opset 17, constant
|
||||
folding, and named I/O (features/reconstructed). export_
|
||||
random_forest and export_isolation_forest convert sklearn
|
||||
estimators to ONNX via skl2onnx with FloatTensorType input
|
||||
and target opset {"": 17, "ai.onnx.ml": 3}. All functions
|
||||
create parent directories and return the output Path
|
||||
|
||||
Connects to:
|
||||
ml/autoencoder - ThreatAutoencoder model class
|
||||
ml/orchestrator - called after training completes
|
||||
core/detection/
|
||||
inference - loads the exported ONNX files
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
metadata.py
|
||||
|
||||
Model metadata persistence for tracking trained model
|
||||
versions and deployment status
|
||||
|
||||
compute_model_version produces a 12-char hex version from
|
||||
the SHA-256 of an ONNX artifact file. save_model_metadata
|
||||
iterates MODEL_TYPES (ae.onnx -> autoencoder, rf.onnx ->
|
||||
random_forest, if.onnx -> isolation_forest), deactivates
|
||||
any previously active version of each type, and inserts
|
||||
new ModelMetadata rows with version, training_samples,
|
||||
metrics, artifact_path, mlflow_run_id, and threshold
|
||||
|
||||
Connects to:
|
||||
models/model_metadata - ModelMetadata ORM model
|
||||
cli/main - called from _write_metadata
|
||||
api/models_api - called after retrain
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
|
|
|
|||
|
|
@ -1,6 +1,32 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
orchestrator.py
|
||||
|
||||
End-to-end training pipeline orchestrator for the 3-model
|
||||
ML ensemble
|
||||
|
||||
TrainingOrchestrator.run accepts (X, y) arrays, calls
|
||||
prepare_training_data for stratified splitting with SMOTE,
|
||||
trains the autoencoder on normal-only data, random forest
|
||||
on labeled data, and isolation forest on normal-only data,
|
||||
exports all three to ONNX (ae.onnx, rf.onnx, if.onnx)
|
||||
plus scaler.json and threshold.json, runs validate_ensemble
|
||||
against the held-out test set with PR-AUC and F1 quality
|
||||
gates, and logs all parameters, metrics, and artifacts to
|
||||
MLflow via VigilExperiment. Returns a TrainingResult
|
||||
dataclass aggregating per-model metrics, gate status,
|
||||
output directory, and MLflow run ID
|
||||
|
||||
Connects to:
|
||||
ml/experiment - VigilExperiment context manager
|
||||
ml/export_onnx - ONNX export functions
|
||||
ml/splitting - prepare_training_data
|
||||
ml/train_autoencoder - train_autoencoder
|
||||
ml/train_classifiers - train_random_forest,
|
||||
train_isolation_forest
|
||||
ml/validation - validate_ensemble
|
||||
cli/main - called from train command
|
||||
api/models_api - called from retrain endpoint
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,27 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
scaler.py
|
||||
|
||||
IQR-based feature scaler with JSON persistence for the
|
||||
autoencoder preprocessing stage
|
||||
|
||||
FeatureScaler wraps sklearn RobustScaler (median/IQR
|
||||
normalization) to handle outlier-heavy HTTP traffic data.
|
||||
Provides fit, transform, fit_transform, and
|
||||
inverse_transform mirroring the sklearn API. save_json
|
||||
serializes center and scale arrays to a human-readable
|
||||
JSON file (avoiding pickle for security and portability),
|
||||
and load_json reconstructs a fitted scaler from that file.
|
||||
Only the autoencoder uses this scaler since tree-based
|
||||
models (random forest, isolation forest) are
|
||||
scale-invariant
|
||||
|
||||
Connects to:
|
||||
ml/train_autoencoder - fitted during AE training
|
||||
ml/orchestrator - scaler.json saved alongside models
|
||||
core/detection/
|
||||
inference - loaded at inference time for AE
|
||||
input normalization
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,23 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
splitting.py
|
||||
|
||||
Stratified train/val/test splitting with SMOTE
|
||||
oversampling for imbalanced attack data
|
||||
|
||||
prepare_training_data performs a 70/15/15 stratified split
|
||||
preserving class ratios, extracts the normal-only subset
|
||||
from training data for the autoencoder and isolation
|
||||
forest, and conditionally applies SMOTE oversampling to
|
||||
the training set when the minority class ratio falls below
|
||||
the target strategy (default 0.3). SMOTE is skipped if the
|
||||
minority class has fewer than k_neighbors+1 samples.
|
||||
Returns a TrainingSplit dataclass with X_train, y_train,
|
||||
X_val, y_val, X_test, y_test, and X_normal_train arrays
|
||||
|
||||
Connects to:
|
||||
ml/orchestrator - called at the start of the training
|
||||
pipeline
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
|
|
|||
|
|
@ -1,6 +1,31 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
synthetic.py
|
||||
|
||||
Synthetic HTTP traffic generator for ML training and
|
||||
testing with realistic attack payloads
|
||||
|
||||
Provides per-category generators for 6 attack types:
|
||||
generate_sqli_requests (22 SQL injection payloads),
|
||||
generate_xss_requests (21 XSS vectors), generate_
|
||||
traversal_requests (15 path traversal payloads),
|
||||
generate_log4shell_requests (10 JNDI lookup variants),
|
||||
generate_ssrf_requests (11 cloud metadata and internal
|
||||
service targets), and generate_scanner_requests (11
|
||||
vulnerability scanner user-agents). generate_normal_
|
||||
requests produces benign traffic across 31 realistic
|
||||
paths. generate_mixed_dataset orchestrates all generators,
|
||||
converts ParsedLogEntry objects to 35-dim feature vectors
|
||||
via extract_request_features and encode_for_inference with
|
||||
zeroed windowed features, and returns (X, y) numpy arrays
|
||||
|
||||
Connects to:
|
||||
core/features/extractor - extract_request_features
|
||||
core/features/encoder - encode_for_inference
|
||||
core/features/mappings - WINDOWED_FEATURE_NAMES
|
||||
core/ingestion/parsers - ParsedLogEntry
|
||||
cli/main - used when no CSIC dataset is
|
||||
available
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,6 +1,27 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
train_autoencoder.py
|
||||
|
||||
PyTorch autoencoder training loop with early stopping and
|
||||
anomaly threshold calibration
|
||||
|
||||
train_autoencoder takes normal-only traffic vectors, splits
|
||||
off a 15% validation set, fits a FeatureScaler (IQR-based)
|
||||
on training data, builds DataLoaders, and trains a
|
||||
ThreatAutoencoder (35->24->12->6->12->24->35) using MSE
|
||||
loss with AdamW optimizer (weight decay 1e-5),
|
||||
ReduceLROnPlateau scheduler (factor 0.5, patience 5),
|
||||
gradient clipping at max_norm 1.0, and early stopping
|
||||
(default patience 10). After training, computes per-sample
|
||||
reconstruction error on the validation set and sets the
|
||||
anomaly threshold at the 99.5th percentile. Returns the
|
||||
trained model, fitted scaler, calibrated threshold, and
|
||||
train/val loss history
|
||||
|
||||
Connects to:
|
||||
ml/autoencoder - ThreatAutoencoder model class
|
||||
ml/scaler - FeatureScaler for input normalization
|
||||
ml/orchestrator - called during pipeline execution
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
|
|
|||
|
|
@ -1,6 +1,24 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
train_classifiers.py
|
||||
|
||||
Sklearn classifier training for the random forest and
|
||||
isolation forest ensemble members
|
||||
|
||||
train_random_forest builds a 200-tree balanced-weight
|
||||
RandomForestClassifier with max_depth 20, wraps it in
|
||||
CalibratedClassifierCV with isotonic calibration (3-fold
|
||||
CV) for well-calibrated probability outputs, evaluates on
|
||||
a held-out 20% calibration split, and returns the
|
||||
calibrated model with accuracy, precision, recall, F1, and
|
||||
PR-AUC metrics. train_isolation_forest fits a 200-tree
|
||||
IsolationForest on normal-only traffic with automatic
|
||||
contamination estimation, returning the model and sample
|
||||
count
|
||||
|
||||
Connects to:
|
||||
ml/orchestrator - called during pipeline execution
|
||||
ml/export_onnx - models exported to ONNX after training
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
|
|
|||
|
|
@ -1,6 +1,28 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
validation.py
|
||||
|
||||
Post-training ensemble validation with quality gates for
|
||||
deployment readiness
|
||||
|
||||
validate_ensemble loads all 3 ONNX models via
|
||||
InferenceEngine, runs batch prediction on held-out test
|
||||
data, normalizes per-model raw scores (AE reconstruction
|
||||
error against threshold, IF anomaly scores), fuses them
|
||||
via weighted average (default weights: AE 0.4, RF 0.4,
|
||||
IF 0.2), applies a 0.5 binary threshold, and computes
|
||||
precision, recall, F1, PR-AUC, and ROC-AUC. Quality
|
||||
gates require PR-AUC >= 0.85 and F1 >= 0.80 for
|
||||
passed_gates to be True. Returns a ValidationResult
|
||||
dataclass with all metrics, confusion matrix, and
|
||||
per-gate pass/fail details
|
||||
|
||||
Connects to:
|
||||
core/detection/ensemble - normalize_ae_score,
|
||||
normalize_if_score, fuse_scores
|
||||
core/detection/inference - InferenceEngine ONNX runtime
|
||||
ml/orchestrator - called after training to gate
|
||||
deployment
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,5 +1,19 @@
|
|||
# ©AngelaMos | 2026
|
||||
# pyproject.toml
|
||||
#
|
||||
# Python project metadata, dependencies, and tool
|
||||
# configuration for AngelusVigil
|
||||
#
|
||||
# Declares the angelusvigil package (Python 3.14+) with
|
||||
# core dependencies (FastAPI, uvicorn, SQLAlchemy, asyncpg,
|
||||
# Redis, Pydantic, watchdog, geoip2, typer), dev extras
|
||||
# (pytest, ruff, mypy, pylint, coverage, fakeredis), and ml
|
||||
# extras (torch, scikit-learn, onnxruntime, mlflow, pandas,
|
||||
# imbalanced-learn). Uses hatchling as the build backend
|
||||
# with app, cli, and ml packages. Configures ruff (line 95,
|
||||
# Python 3.14 target), mypy (strict mode), pylint (4 jobs
|
||||
# with pydantic plugin), and pytest (asyncio auto mode).
|
||||
# Connects to all backend source modules
|
||||
|
||||
[project]
|
||||
name = "angelusvigil"
|
||||
|
|
|
|||
|
|
@ -1,4 +1,17 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
__init__.py
|
||||
|
||||
Test suite package for the ai-threat-detection backend
|
||||
|
||||
Contains unit, integration, and end-to-end tests covering
|
||||
the full stack: API endpoints, ingestion pipeline, feature
|
||||
extraction, rule engine, ML training and inference,
|
||||
ensemble scoring, ONNX export, model metadata persistence,
|
||||
CLI commands, and GeoIP enrichment. Uses pytest-asyncio for
|
||||
async tests, fakeredis for Redis isolation, and in-memory
|
||||
SQLite via aiosqlite for database tests
|
||||
|
||||
Connects to:
|
||||
tests/conftest - shared fixtures for DB and HTTP client
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -2,7 +2,22 @@
|
|||
©AngelaMos | 2026
|
||||
conftest.py
|
||||
|
||||
Shared pytest fixtures for in-memory SQLite database and HTTPX test client setup.
|
||||
Shared pytest fixtures for in-memory SQLite database and
|
||||
HTTPX async test client setup
|
||||
|
||||
test_settings overrides Settings for the test environment
|
||||
with an in-memory SQLite URL and dummy paths. db_engine
|
||||
creates a StaticPool aiosqlite engine with all tables via
|
||||
SQLModel.metadata.create_all. db_session yields an
|
||||
AsyncSession bound to the shared engine. db_client builds
|
||||
a full HTTPX AsyncClient with ASGITransport wrapping the
|
||||
FastAPI app, overriding get_session to use the in-memory
|
||||
database with auto-commit
|
||||
|
||||
Connects to:
|
||||
app/config - Settings override
|
||||
app/factory - create_app for ASGI transport
|
||||
app/api/deps - get_session dependency override
|
||||
"""
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
|
|
|
|||
|
|
@ -2,7 +2,23 @@
|
|||
©AngelaMos | 2026
|
||||
test_api.py
|
||||
|
||||
Tests the FastAPI REST endpoints for threats, stats, health, readiness, and model management.
|
||||
Tests the FastAPI REST endpoints for health, threats, stats,
|
||||
and model management using an in-memory database
|
||||
|
||||
Validates /health returns status, uptime, and pipeline flag.
|
||||
Tests /threats CRUD: empty list returns zero total, random
|
||||
UUID returns 404, seeded event is fetchable by ID with all
|
||||
fields, and severity filter returns only matching items.
|
||||
Tests /stats returns zeroed counts on empty window.
|
||||
Tests /models/status returns detection_mode and
|
||||
active_models list, and POST /models/retrain returns 202
|
||||
with a 32-char job ID
|
||||
|
||||
Connects to:
|
||||
api/health - liveness endpoint
|
||||
api/threats - threat CRUD
|
||||
api/stats - statistics endpoint
|
||||
api/models_api - model status and retrain
|
||||
"""
|
||||
|
||||
import uuid
|
||||
|
|
|
|||
|
|
@ -2,7 +2,21 @@
|
|||
©AngelaMos | 2026
|
||||
test_autoencoder.py
|
||||
|
||||
Tests the ThreatAutoencoder architecture: shapes, output range, reconstruction error, and training behavior.
|
||||
Tests the ThreatAutoencoder PyTorch architecture for shape
|
||||
correctness, output range, reconstruction error, and
|
||||
training behavior
|
||||
|
||||
Validates output shape matches input (batch, 35), encoder
|
||||
bottleneck compresses to 6 dimensions, single-sample
|
||||
forward pass succeeds in eval mode, decoder output is
|
||||
unbounded (matching RobustScaler range), reconstruction
|
||||
error returns one positive scalar per sample, trained model
|
||||
reconstructs normal data better than anomalies after 50
|
||||
epochs, eval mode produces deterministic output (dropout
|
||||
off), and variable batch sizes (1, 8, 32, 128) are handled
|
||||
|
||||
Connects to:
|
||||
ml/autoencoder - ThreatAutoencoder
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
|
|
|||
|
|
@ -1,6 +1,20 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_cli.py
|
||||
|
||||
Tests the Typer CLI command help output, argument
|
||||
validation, and metadata persistence wiring
|
||||
|
||||
TestCLICommands validates train --help shows csic-dir and
|
||||
synthetic options, nonexistent csic-dir exits with error,
|
||||
replay/serve/config/health --help exit cleanly with
|
||||
expected content, and missing replay log file fails.
|
||||
TestCLITrainMetadata mocks the orchestrator to verify that
|
||||
train emits a warning when DB metadata write is unavailable
|
||||
|
||||
Connects to:
|
||||
cli/main - Typer app with serve, train, replay, config,
|
||||
health commands
|
||||
"""
|
||||
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -2,7 +2,17 @@
|
|||
©AngelaMos | 2026
|
||||
test_config_ml.py
|
||||
|
||||
Tests ML-related settings defaults: detection mode, ensemble weights, model paths, and MLflow URI.
|
||||
Tests ML-related settings defaults for detection mode,
|
||||
ensemble weights, model paths, and MLflow tracking URI
|
||||
|
||||
Validates that the default detection_mode is 'rules',
|
||||
ensemble weights (AE + RF + IF) sum to exactly 1.0,
|
||||
model_dir defaults to 'data/models', ae_threshold_
|
||||
percentile defaults to 99.5, and mlflow_tracking_uri
|
||||
defaults to 'file:./mlruns'
|
||||
|
||||
Connects to:
|
||||
app/config - Settings pydantic-settings model
|
||||
"""
|
||||
|
||||
from app.config import settings
|
||||
|
|
|
|||
|
|
@ -1,6 +1,23 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_data_loader.py
|
||||
|
||||
Tests CSIC 2010 dataset parsing, CSICRequest-to-
|
||||
ParsedLogEntry conversion, and end-to-end dataset loading
|
||||
|
||||
TestParseCSICFile validates HTTP request block splitting,
|
||||
method/path/query/header extraction, POST body capture,
|
||||
attack label assignment, malformed block skipping, and
|
||||
empty file handling using inline CSIC-format fixtures.
|
||||
TestCSICToParsedEntry verifies synthesized defaults (IP,
|
||||
timestamp, status) and POST body query string merging.
|
||||
TestLoadCSICDataset confirms 35-column X shape, dual-label
|
||||
y arrays, correct per-file label counts, and finite feature
|
||||
values
|
||||
|
||||
Connects to:
|
||||
ml/data_loader - parse_csic_file, csic_to_parsed_entry,
|
||||
load_csic_dataset
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
|
|
|||
|
|
@ -2,7 +2,25 @@
|
|||
©AngelaMos | 2026
|
||||
test_detection.py
|
||||
|
||||
Tests the rule engine's threat scoring, severity classification, and attack pattern matching.
|
||||
Tests the RuleEngine threat scoring, severity
|
||||
classification, and OWASP attack pattern matching
|
||||
|
||||
Validates normal requests score LOW below 0.5, SQL
|
||||
injection in query strings scores HIGH with SQL_INJECTION
|
||||
rule, XSS payloads trigger XSS rule, path traversal
|
||||
triggers PATH_TRAVERSAL, command injection triggers
|
||||
COMMAND_INJECTION at HIGH severity, scanner UAs fire
|
||||
SCANNER_UA, high request rates fire RATE_ANOMALY, multiple
|
||||
rules aggregate to higher scores, scores are clamped to
|
||||
[0, 1], severity thresholds align with architecture
|
||||
(LOW < 0.5, MEDIUM >= 0.5, HIGH >= 0.7), component_scores
|
||||
match matched_rules, FILE_INCLUSION detects PHP stream
|
||||
wrappers, and DOUBLE_ENCODING detects %25-prefixed
|
||||
sequences
|
||||
|
||||
Connects to:
|
||||
core/detection/rules - RuleEngine
|
||||
core/ingestion/parsers - ParsedLogEntry
|
||||
"""
|
||||
|
||||
from datetime import datetime, UTC
|
||||
|
|
|
|||
|
|
@ -2,7 +2,23 @@
|
|||
©AngelaMos | 2026
|
||||
test_ensemble.py
|
||||
|
||||
Tests ensemble score normalization, weighted fusion, ML/rule blending, and severity classification.
|
||||
Tests ensemble score normalization, weighted fusion, ML/rule
|
||||
blending, and severity classification functions
|
||||
|
||||
TestScoreNormalization validates AE error below threshold
|
||||
maps below 0.5, 3x threshold caps at 1.0, zero error maps
|
||||
to 0.0, negative IF score maps above 0.5, positive below
|
||||
0.5, and zero maps to 0.5. TestEnsembleFusion validates
|
||||
weighted average computation, all-zero scores fuse to 0.0,
|
||||
all-one scores fuse to 1.0, and partial model support.
|
||||
TestBlendScores validates ML/rule blending at various
|
||||
weights and clamping. TestClassifySeverity validates HIGH
|
||||
at >= 0.7, MEDIUM at [0.5, 0.7), LOW below 0.5
|
||||
|
||||
Connects to:
|
||||
core/detection/ensemble - normalize_ae_score,
|
||||
normalize_if_score, fuse_scores,
|
||||
blend_scores, classify_severity
|
||||
"""
|
||||
|
||||
from app.core.detection.ensemble import (
|
||||
|
|
|
|||
|
|
@ -2,7 +2,19 @@
|
|||
©AngelaMos | 2026
|
||||
test_experiment.py
|
||||
|
||||
Tests the VigilExperiment MLflow wrapper: run lifecycle, param/metric logging, and status tagging.
|
||||
Tests the VigilExperiment MLflow context manager for run
|
||||
lifecycle, parameter/metric logging, and status tagging
|
||||
|
||||
Uses a tmp_path MLflow tracking URI for isolation.
|
||||
Validates run ID is set on context entry and None before,
|
||||
log_params writes string values, log_metrics stores floats,
|
||||
log_artifact uploads files to the artifact list,
|
||||
python_version and platform system metadata tags are auto-
|
||||
logged, successful exit tags status='completed', and
|
||||
exception exit tags status='failed' with the error message
|
||||
|
||||
Connects to:
|
||||
ml/experiment - VigilExperiment
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
|
|
|||
|
|
@ -2,7 +2,22 @@
|
|||
©AngelaMos | 2026
|
||||
test_export_onnx.py
|
||||
|
||||
Tests ONNX export for the autoencoder, random forest, and isolation forest models.
|
||||
Tests ONNX export and inference parity for the autoencoder,
|
||||
random forest, and isolation forest models
|
||||
|
||||
TestAutoencoderExport validates file creation, PyTorch-to-
|
||||
ONNX output match within 1e-5 tolerance, and dynamic batch
|
||||
dimension (1, 16, 64). TestRandomForestExport validates
|
||||
file creation and ONNX inference returning class predictions
|
||||
and probabilities. TestIsolationForestExport validates file
|
||||
creation and ONNX anomaly scores matching sklearn
|
||||
decision_function within 1e-4 tolerance
|
||||
|
||||
Connects to:
|
||||
ml/export_onnx - export_autoencoder,
|
||||
export_random_forest,
|
||||
export_isolation_forest
|
||||
ml/autoencoder - ThreatAutoencoder for AE export
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
|
|
|||
|
|
@ -2,7 +2,31 @@
|
|||
©AngelaMos | 2026
|
||||
test_features.py
|
||||
|
||||
Tests per-request feature extraction, Redis sliding-window aggregation, and feature encoding.
|
||||
Tests the 23 per-request feature extractor, Redis sliding-
|
||||
window aggregator (12 windowed features), and 35-dim
|
||||
feature encoder
|
||||
|
||||
Validates all 23 feature keys are returned, path_depth
|
||||
counts segments, path_entropy distinguishes random vs
|
||||
simple paths, query param count and length, percent-
|
||||
encoding and double-encoding detection, status class
|
||||
grouping, temporal features (hour, day, weekend), bot
|
||||
and scanner UA detection, attack pattern detection (SQLi,
|
||||
XSS, traversal), special char ratio, private IP, file
|
||||
extension, and country code passthrough. WindowAggregator
|
||||
tests use fakeredis to validate single/multi-request
|
||||
counts, error rate calculation, unique paths/UAs, TTL
|
||||
setting, and window boundary exclusion. Encoder tests
|
||||
validate 35-element output, method/status ordinal mapping,
|
||||
boolean-to-float, numerical passthrough, and unknown
|
||||
categorical fallback
|
||||
|
||||
Connects to:
|
||||
core/features/extractor - extract_request_features
|
||||
core/features/aggregator - WindowAggregator
|
||||
core/features/encoder - encode_for_inference
|
||||
core/features/mappings - FEATURE_ORDER, METHOD_MAP,
|
||||
STATUS_CLASS_MAP
|
||||
"""
|
||||
|
||||
import time
|
||||
|
|
|
|||
|
|
@ -2,7 +2,19 @@
|
|||
©AngelaMos | 2026
|
||||
test_geoip.py
|
||||
|
||||
Tests the GeoIP lookup service including private IP handling and missing database fallback.
|
||||
Tests the GeoIPService MaxMind lookup including private IP
|
||||
handling, error cases, and missing database fallback
|
||||
|
||||
Validates GeoResult field storage, successful lookup
|
||||
returning country/city/lat/lon, private and loopback IPs
|
||||
returning None without hitting the reader, AddressNotFound
|
||||
Error returning None, None reader returning None, missing
|
||||
city name handled gracefully, non-existent .mmdb path sets
|
||||
reader to None, and valid .mmdb path opens the reader via
|
||||
mock
|
||||
|
||||
Connects to:
|
||||
core/enrichment/geoip - GeoIPService, GeoResult
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
|
|
|||
|
|
@ -2,7 +2,23 @@
|
|||
©AngelaMos | 2026
|
||||
test_inference.py
|
||||
|
||||
Tests the InferenceEngine: model loading, predict output shapes, score ranges, and missing-model handling.
|
||||
Tests the ONNX InferenceEngine for model loading, batch
|
||||
prediction, score ranges, and error handling
|
||||
|
||||
Uses a model_dir fixture with all 3 exported ONNX models,
|
||||
scaler.json, and threshold.json. Validates is_loaded=True
|
||||
with all models, is_loaded=False for nonexistent and
|
||||
partial directories, predict returns None when not loaded,
|
||||
predict returns ae/rf/if score dicts, AE scores are non-
|
||||
negative, RF probabilities are in [0, 1], single-sample
|
||||
prediction works, threshold loads from JSON, and partial
|
||||
model sets (AE only) report not loaded
|
||||
|
||||
Connects to:
|
||||
core/detection/inference - InferenceEngine
|
||||
ml/export_onnx - model export for fixture
|
||||
ml/scaler - FeatureScaler for fixture
|
||||
ml/autoencoder - ThreatAutoencoder for fixture
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -2,7 +2,24 @@
|
|||
©AngelaMos | 2026
|
||||
test_integration.py
|
||||
|
||||
End-to-end tests covering the full path from log file write through tailer, pipeline, and database storage.
|
||||
End-to-end tests covering the full path from log file
|
||||
write through tailer, pipeline, and database storage
|
||||
|
||||
integration_env fixture creates a temp log file, in-memory
|
||||
SQLite, fake Redis, AlertDispatcher, RuleEngine, Pipeline,
|
||||
and LogTailer wired together. Tests write nginx-format log
|
||||
lines (normal, SQLi, XSS, path traversal) to the file and
|
||||
poll the database for stored ThreatEvent rows. Validates
|
||||
that MEDIUM+ threats are persisted, LOW severity requests
|
||||
are not stored, and stored events have correct severity,
|
||||
score, matched_rules, feature_vector length, and source_ip
|
||||
|
||||
Connects to:
|
||||
core/ingestion/tailer - LogTailer
|
||||
core/ingestion/pipeline - Pipeline
|
||||
core/alerts/dispatcher - AlertDispatcher
|
||||
core/detection/rules - RuleEngine
|
||||
models/threat_event - ThreatEvent
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,24 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_metadata.py
|
||||
|
||||
Tests SHA-256 model version hashing and async metadata
|
||||
persistence to the database
|
||||
|
||||
TestComputeModelVersion verifies 12-char hex output,
|
||||
deterministic hashing (same file = same version), and
|
||||
distinct versions for different files. TestSaveModel
|
||||
Metadata uses an in-memory SQLite session and fake ONNX
|
||||
artifacts to validate 3-row creation (one per model type),
|
||||
is_active flag on new rows, correct model_type values
|
||||
(autoencoder, random_forest, isolation_forest), previous
|
||||
active row deactivation on re-save, and inactive row
|
||||
preservation (6 total rows after two saves)
|
||||
|
||||
Connects to:
|
||||
ml/metadata - compute_model_version,
|
||||
save_model_metadata
|
||||
models/model_metadata - ModelMetadata ORM model
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_ml_integration.py
|
||||
|
||||
Tests the ML inference engine wired into the ingestion
|
||||
pipeline in hybrid detection mode
|
||||
|
||||
Uses a trained_model_dir fixture with ONNX models to build
|
||||
a pipeline with InferenceEngine. Validates hybrid detection
|
||||
mode is set when ML models are present, final_score is in
|
||||
[0, 1], rules-only mode falls back to rule score as final
|
||||
score, attack lines score higher than benign in hybrid
|
||||
mode, and rule_result is preserved alongside ML scores
|
||||
|
||||
Connects to:
|
||||
core/detection/inference - InferenceEngine
|
||||
core/detection/rules - RuleEngine
|
||||
core/ingestion/pipeline - Pipeline, ScoredRequest
|
||||
ml/export_onnx - model export for fixture
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,20 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_orchestrator.py
|
||||
|
||||
Tests the TrainingOrchestrator pipeline from data splitting
|
||||
through model export, validation, and MLflow logging
|
||||
|
||||
Verifies all 5 output files are produced (ae.onnx, rf.onnx,
|
||||
if.onnx, scaler.json, threshold.json), TrainingResult
|
||||
dataclass structure, scaler.json keys (center, scale,
|
||||
n_features), threshold.json float value, per-model metrics
|
||||
presence (ae_threshold, rf f1, if n_samples), ensemble
|
||||
validation metrics, MLflow run ID capture (32-char hex),
|
||||
and passed_gates boolean type
|
||||
|
||||
Connects to:
|
||||
ml/orchestrator - TrainingOrchestrator, TrainingResult
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -2,7 +2,19 @@
|
|||
©AngelaMos | 2026
|
||||
test_parsers.py
|
||||
|
||||
Tests nginx combined log line parsing via parse_combined.
|
||||
Tests nginx combined-format log line parsing via the
|
||||
parse_combined function
|
||||
|
||||
Validates full field extraction (IP, timestamp, method,
|
||||
path, query string, status code, response size, referer,
|
||||
user agent, raw line), IPv4 and IPv6 address handling,
|
||||
dash-referer normalization to empty string, multi-parameter
|
||||
query strings with special characters, malformed and empty
|
||||
line None returns, dash response size normalization to
|
||||
zero, and full-length IPv6 address parsing
|
||||
|
||||
Connects to:
|
||||
core/ingestion/parsers - parse_combined, ParsedLogEntry
|
||||
"""
|
||||
|
||||
from datetime import datetime, UTC
|
||||
|
|
|
|||
|
|
@ -2,7 +2,21 @@
|
|||
©AngelaMos | 2026
|
||||
test_pipeline.py
|
||||
|
||||
Tests the async ingestion pipeline: parsing, feature extraction, rule scoring, and shutdown.
|
||||
Tests the async ingestion pipeline across all 4 stages:
|
||||
parsing, feature extraction, rule scoring, and dispatch
|
||||
|
||||
Uses a fakeredis-backed Pipeline with a results collector
|
||||
callback. Validates that valid log lines flow end-to-end
|
||||
producing a ScoredRequest with correct IP, method, 35-dim
|
||||
feature vector, and LOW severity. Confirms malformed lines
|
||||
are dropped without crashing, backpressure works with
|
||||
maxsize=1 queues, stop() drains remaining items with all
|
||||
tasks completing cleanly, and SQLi payloads score HIGH with
|
||||
SQL_INJECTION rule match
|
||||
|
||||
Connects to:
|
||||
core/ingestion/pipeline - Pipeline, ScoredRequest
|
||||
core/detection/rules - RuleEngine
|
||||
"""
|
||||
|
||||
import fakeredis.aioredis
|
||||
|
|
|
|||
|
|
@ -2,7 +2,21 @@
|
|||
©AngelaMos | 2026
|
||||
test_scaler.py
|
||||
|
||||
Tests the FeatureScaler: fitting, transform correctness, JSON serialization, and round-trip loading.
|
||||
Tests the FeatureScaler IQR-based normalization for
|
||||
fitting, transform correctness, JSON round-trip, and error
|
||||
handling
|
||||
|
||||
Validates n_features is stored after fit, transform
|
||||
preserves shape and float32 dtype, median of scaled
|
||||
features is near zero, inverse_transform recovers original
|
||||
values within 1e-5, save_json creates a valid JSON file
|
||||
with center/scale/n_features keys, load_json round-trip
|
||||
produces identical transform output within 1e-6, transform
|
||||
before fit raises RuntimeError, and fit_transform
|
||||
convenience method works
|
||||
|
||||
Connects to:
|
||||
ml/scaler - FeatureScaler
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,20 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_splitting.py
|
||||
|
||||
Tests stratified train/val/test splitting with SMOTE
|
||||
oversampling for imbalanced datasets
|
||||
|
||||
Validates TrainingSplit dataclass return, 70/15/15 split
|
||||
proportions within tolerance, stratified class distribution
|
||||
preservation in val/test sets, SMOTE minority ratio near
|
||||
target strategy (0.3), val/test sizes unaffected by SMOTE,
|
||||
X_normal_train containing only class-0 rows, small dataset
|
||||
(50 samples) success, single-class ValueError, and SMOTE
|
||||
skip when minority count is below k_neighbors threshold
|
||||
|
||||
Connects to:
|
||||
ml/splitting - prepare_training_data, TrainingSplit
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
|
|
|
|||
|
|
@ -1,6 +1,24 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_synthetic.py
|
||||
|
||||
Tests synthetic HTTP traffic generators and mixed dataset
|
||||
assembly for ML training
|
||||
|
||||
TestGenerators validates all 7 per-type generators (SQLi,
|
||||
XSS, traversal, Log4Shell, SSRF, scanner, normal) return
|
||||
correct counts, contain expected payload patterns (OR/UNION
|
||||
for SQLi, script/alert for XSS, ../ for traversal), return
|
||||
ParsedLogEntry instances, and pass through feature
|
||||
extraction and encoding to 35-dim vectors. TestMixedDataset
|
||||
verifies correct X shape (n, 35), dual-label y, matching
|
||||
label counts, and finite feature values
|
||||
|
||||
Connects to:
|
||||
ml/synthetic - all generate_* functions,
|
||||
generate_mixed_dataset
|
||||
core/features/extractor - extract_request_features
|
||||
core/features/encoder - encode_for_inference
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
|
|
|
|||
|
|
@ -2,7 +2,24 @@
|
|||
©AngelaMos | 2026
|
||||
test_training.py
|
||||
|
||||
Tests training pipelines for the autoencoder, random forest, and isolation forest models.
|
||||
Tests training functions for the autoencoder, random forest,
|
||||
and isolation forest models
|
||||
|
||||
TestAutoencoderTraining validates train_autoencoder returns
|
||||
model/threshold/scaler/history, threshold is positive,
|
||||
history has correct epoch count, higher percentile yields
|
||||
higher threshold, and returned model is in eval mode.
|
||||
TestRandomForestTraining validates model/metrics return,
|
||||
predict_proba availability, required metric keys (f1,
|
||||
pr_auc, accuracy, precision, recall), probability range,
|
||||
and metric value range. TestIsolationForestTraining
|
||||
validates model return, score_samples availability,
|
||||
n_samples metric, and normal/outlier score separation
|
||||
|
||||
Connects to:
|
||||
ml/train_autoencoder - train_autoencoder
|
||||
ml/train_classifiers - train_random_forest,
|
||||
train_isolation_forest
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
|
|
|
|||
|
|
@ -1,6 +1,24 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_training_e2e.py
|
||||
|
||||
End-to-end training integration test from synthetic data
|
||||
generation through ONNX inference and score fusion
|
||||
|
||||
test_full_training_produces_loadable_models generates a
|
||||
500-normal/200-attack synthetic dataset, runs the full
|
||||
TrainingOrchestrator pipeline with 3 epochs, verifies all
|
||||
5 output files (ae.onnx, rf.onnx, if.onnx, scaler.json,
|
||||
threshold.json), loads models via InferenceEngine, runs
|
||||
batch prediction, normalizes and fuses per-model scores,
|
||||
blends with rule scores, and asserts all values are in
|
||||
[0, 1]. Validates passed_gates is a boolean
|
||||
|
||||
Connects to:
|
||||
ml/orchestrator - TrainingOrchestrator
|
||||
ml/synthetic - generate_mixed_dataset
|
||||
core/detection/ensemble - normalize, fuse, blend
|
||||
core/detection/inference - InferenceEngine
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
|
|
|||
|
|
@ -1,6 +1,22 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
test_validation.py
|
||||
|
||||
Tests post-training ensemble validation with quality gates
|
||||
|
||||
Uses a trained_model_dir fixture with all 3 ONNX models,
|
||||
scaler, and threshold, plus a separable_test_data fixture
|
||||
with well-separated normal/attack clusters. Validates
|
||||
ValidationResult structure, metric ranges (precision,
|
||||
recall, f1, pr_auc, roc_auc all in [0, 1]), 2x2 confusion
|
||||
matrix shape, gate_details keys (pr_auc, f1), gate pass
|
||||
with low thresholds, gate fail with high thresholds, and
|
||||
custom ensemble weight acceptance
|
||||
|
||||
Connects to:
|
||||
ml/validation - validate_ensemble, ValidationResult
|
||||
ml/export_onnx - model export for fixture setup
|
||||
ml/scaler - FeatureScaler for fixture setup
|
||||
"""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,5 +1,20 @@
|
|||
# ©AngelaMos | 2026
|
||||
# Production Docker Compose
|
||||
# compose.yml
|
||||
#
|
||||
# Production Docker Compose stack for AngelusVigil
|
||||
#
|
||||
# Orchestrates 5 services on the vigil_network bridge:
|
||||
# postgres (18-alpine with healthcheck and persistent
|
||||
# volume), redis (7.4-alpine with custom redis.conf),
|
||||
# backend (FastAPI with asyncpg, Redis, nginx log tail,
|
||||
# GeoIP, and model data volumes), frontend (Vite
|
||||
# production build served via nginx on the host port),
|
||||
# and geoip-updater (MaxMind weekly refresh). Joins the
|
||||
# external certgames_net network and mounts the external
|
||||
# certgames_nginx_logs volume for real-time log access.
|
||||
# Connects to infra/docker/fastapi.prod,
|
||||
# infra/docker/vite.prod, infra/redis/redis.conf,
|
||||
# infra/nginx/vigil.conf
|
||||
|
||||
services:
|
||||
postgres:
|
||||
|
|
|
|||
|
|
@ -1,5 +1,12 @@
|
|||
# ©AngelaMos | 2026
|
||||
# Dockerfile
|
||||
#
|
||||
# Container image for the dev-log FastAPI target application
|
||||
#
|
||||
# Based on python:3.14-slim with uv copied from the official
|
||||
# astral-sh image. Installs fastapi and uvicorn system-wide,
|
||||
# copies app.py, and runs uvicorn on port 8000. Sits behind
|
||||
# the nginx reverse proxy defined in compose.yml
|
||||
|
||||
FROM python:3.14-slim
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,25 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
app.py
|
||||
|
||||
Minimal FastAPI target application for generating nginx
|
||||
access logs during development
|
||||
|
||||
Exposes realistic REST endpoints that the simulate.py
|
||||
traffic generator hits through the nginx reverse proxy:
|
||||
/ (HTML landing), /health, /api/users (list and by ID),
|
||||
/api/login (POST returning a fake JWT), /api/search with
|
||||
query parameter, /api/products (list and by ID),
|
||||
/api/checkout (POST), /admin and /admin/dashboard (403
|
||||
forbidden), and /static/{path} (404). Designed to produce
|
||||
diverse nginx combined-format log lines for testing the
|
||||
ingestion pipeline and rule engine
|
||||
|
||||
Connects to:
|
||||
dev-log/nginx.conf - proxied behind nginx
|
||||
dev-log/simulate.py - traffic generator targets these
|
||||
endpoints
|
||||
dev-log/compose.yml - containerized as vigil-devlog-app
|
||||
"""
|
||||
|
||||
from fastapi import FastAPI, Request
|
||||
|
|
|
|||
|
|
@ -1,5 +1,15 @@
|
|||
# ©AngelaMos | 2026
|
||||
# compose.yml
|
||||
#
|
||||
# Docker Compose stack for the dev-log traffic generation
|
||||
# environment
|
||||
#
|
||||
# Runs two services on a bridge network: app (FastAPI target
|
||||
# built from the local Dockerfile with a /health check) and
|
||||
# nginx (alpine image reverse-proxying port 58319 to the
|
||||
# app, writing combined-format access logs to a named volume
|
||||
# vigil_dev_nginx_logs). The nginx container clears stale
|
||||
# log files on startup for clean sessions
|
||||
|
||||
services:
|
||||
app:
|
||||
|
|
|
|||
|
|
@ -1,5 +1,15 @@
|
|||
# ©AngelaMos | 2026
|
||||
# nginx.conf
|
||||
#
|
||||
# Nginx reverse proxy configuration for the dev-log traffic
|
||||
# generation environment
|
||||
#
|
||||
# Listens on port 80, proxies all requests to the upstream
|
||||
# FastAPI app on port 8000, and writes combined-format
|
||||
# access logs to /var/log/nginx/access.log. Sets X-Real-IP,
|
||||
# X-Forwarded-For, and X-Forwarded-Proto headers for the
|
||||
# backend. The log output is mounted as a named volume in
|
||||
# compose.yml for consumption by the ingestion pipeline
|
||||
|
||||
events {
|
||||
worker_connections 64;
|
||||
|
|
|
|||
|
|
@ -2,6 +2,26 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
simulate.py
|
||||
|
||||
HTTP traffic simulator for generating realistic attack and
|
||||
normal log patterns against the dev-log target application
|
||||
|
||||
Provides 10 traffic modes via argparse: normal (benign
|
||||
browsing with GET/POST mix), sqli (12 SQL injection
|
||||
payloads), xss (10 script/event handler vectors),
|
||||
traversal (10 dot-dot-slash and encoding variants), cmdi
|
||||
(7 shell command injection payloads), log4shell (4 JNDI
|
||||
lookup variants), ssrf (5 cloud metadata and internal
|
||||
service targets), scanner (20 recon paths with 11 scanner
|
||||
user-agents), flood (rapid-fire requests), and mixed
|
||||
(50/10/40 normal/scanner/attack split). Uses urllib for
|
||||
HTTP requests with configurable count, delay, and target
|
||||
URL. Checks /health reachability before starting
|
||||
|
||||
Connects to:
|
||||
dev-log/app.py - target endpoints
|
||||
dev-log/nginx.conf - requests proxied through nginx
|
||||
to generate access.log entries
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
|
|
|||
|
|
@ -1,5 +1,18 @@
|
|||
# ©AngelaMos | 2026
|
||||
# dev.compose.yml
|
||||
#
|
||||
# Development Docker Compose stack with exposed ports and
|
||||
# hot reload
|
||||
#
|
||||
# Orchestrates 4 services on the vigil_dev bridge: postgres
|
||||
# (18-alpine on host port 16969 with default devpassword),
|
||||
# redis (7.4-alpine on host port 26969 with appendonly),
|
||||
# backend (FastAPI dev build on host port 36969 with debug
|
||||
# enabled, quiet gitpython, and SKIP_AUTO_TRAIN toggle),
|
||||
# and frontend (Vite dev server on host port 46969 with
|
||||
# source bind-mount for HMR and API proxy to the backend).
|
||||
# Connects to infra/docker/fastapi.dev,
|
||||
# infra/docker/vite.dev
|
||||
|
||||
services:
|
||||
postgres:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,14 @@
|
|||
// ===================
|
||||
// © AngelaMos | 2026
|
||||
// App.tsx
|
||||
//
|
||||
// Root React component with providers and routing
|
||||
//
|
||||
// Wraps the application in QueryClientProvider (TanStack
|
||||
// React Query), provides the browser router via
|
||||
// RouterProvider, renders a dark-themed Sonner toast
|
||||
// container at top-right, and includes ReactQueryDevtools
|
||||
// in development mode
|
||||
// ===================
|
||||
|
||||
import { QueryClientProvider } from '@tanstack/react-query'
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
// ===================
|
||||
// © AngelaMos | 2026
|
||||
// index.ts
|
||||
//
|
||||
// Barrel export for API query hooks
|
||||
//
|
||||
// Re-exports useAlerts (WebSocket alert stream),
|
||||
// useModelStatus and useRetrain (model management),
|
||||
// useStats (dashboard statistics), and useThreats and
|
||||
// useThreat (threat event listing and detail)
|
||||
// ===================
|
||||
|
||||
export * from './useAlerts'
|
||||
|
|
|
|||
|
|
@ -1,6 +1,21 @@
|
|||
// ===================
|
||||
// © AngelaMos | 2026
|
||||
// useAlerts.ts
|
||||
//
|
||||
// WebSocket alert stream hook with Zustand state and
|
||||
// exponential reconnect
|
||||
//
|
||||
// Maintains a Zustand AlertState store holding the alert
|
||||
// ring buffer (capped at ALERTS.MAX_ITEMS), connection
|
||||
// status, and error state. useAlerts opens a WebSocket to
|
||||
// WS_ENDPOINTS.ALERTS, validates incoming JSON frames
|
||||
// against WebSocketAlertSchema via safeParse, stamps each
|
||||
// with a crypto.randomUUID id, and prepends to the store.
|
||||
// On close the hook schedules reconnection with exponential
|
||||
// backoff (RECONNECT_BASE_MS * 2^attempt, capped at
|
||||
// RECONNECT_MAX_MS). Cleanup on unmount closes the socket
|
||||
// and clears the retry timer. Connects to api/types/
|
||||
// websocket.types, config, components/alert-feed
|
||||
// ===================
|
||||
|
||||
import { useEffect, useRef } from 'react'
|
||||
|
|
|
|||
|
|
@ -1,6 +1,17 @@
|
|||
// ===================
|
||||
// © AngelaMos | 2026
|
||||
// useModels.ts
|
||||
//
|
||||
// TanStack Query hooks for model status and retraining
|
||||
//
|
||||
// useModelStatus fetches API_ENDPOINTS.MODELS.STATUS and
|
||||
// validates the response through ModelStatusSchema, using
|
||||
// the standard query strategy. useRetrain posts to
|
||||
// API_ENDPOINTS.MODELS.RETRAIN, validates through
|
||||
// RetrainResponseSchema, shows a Sonner success toast, and
|
||||
// invalidates all QUERY_KEYS.MODELS queries to refresh the
|
||||
// status display. Connects to api/types/models.types,
|
||||
// core/api, config, pages/models
|
||||
// ===================
|
||||
|
||||
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue