# Core Security Concepts This document covers the security and machine learning concepts behind AngelusVigil. These aren't textbook definitions. We'll dig into how each concept is actually implemented in the codebase and why the design choices matter. ## Anomaly Detection ### What It Is Anomaly detection identifies data points that don't fit the expected pattern. In the context of web traffic, "expected" means the normal behavior of legitimate users and applications. Anything that deviates significantly from that baseline is flagged as potentially malicious. This is fundamentally different from signature-based detection (like traditional WAFs), which compares requests against a database of known attack patterns. Signature detection can only catch attacks it already knows about. Anomaly detection can catch attacks it has never seen before, as long as they look different from normal traffic. ### Why It Matters In December 2021, the Log4Shell vulnerability (CVE-2021-44228) was disclosed. Within hours, attackers were sending requests containing `${jndi:ldap://...}` payloads to millions of servers. WAF vendors scrambled to push rule updates, but many organizations were exposed for days before rules were available. An anomaly detection system trained on normal traffic would have flagged these requests immediately. The `${jndi:` substring has high entropy, unusual character distribution, and doesn't match any legitimate path or query pattern. The 2020 SolarWinds attack (SUNBURST) is another example. The attackers used legitimate-looking HTTP requests to blend in with normal Orion platform traffic. But their request timing, path patterns, and behavioral fingerprints were subtly different from real management traffic. A well-tuned anomaly detector monitoring those signals could have raised early warnings. ### How It Works AngelusVigil uses an **autoencoder** for anomaly detection. The concept is simple: train a neural network to compress and reconstruct normal traffic. If a new request can't be reconstructed well (high reconstruction error), it's probably abnormal. ``` Input (35 features) ──► Encoder ──► Bottleneck (6 dims) ──► Decoder ──► Reconstruction (35 features) │ Compare with input │ MSE = anomaly score ``` The autoencoder architecture in this project: ``` 35 ──► 24 ──► 12 ──► 6 ──► 12 ──► 24 ──► 35 input bottleneck output ``` The bottleneck forces the network to learn a compressed representation of normal traffic. Attack requests contain patterns the autoencoder never saw during training, so they reconstruct poorly. The reconstruction error (mean squared error between input and output) becomes the anomaly score. During training, only normal (benign) traffic is used. The model learns to compress and reconstruct legitimate requests. During inference, the reconstruction error is compared against a calibrated threshold (set at the 99.5th percentile of validation set errors). Requests exceeding this threshold are flagged as anomalous. ### Common Attacks That Anomaly Detection Catches 1. **Zero-day exploits** - Novel attack payloads that don't match any known signatures. The Log4Shell example above is a perfect case: the request structure is so unusual that anomaly detection flags it even without a specific rule. 2. **Slow-and-low attacks** - Attackers who spread requests across time to evade rate limits. The windowed aggregation features (unique paths, method entropy, inter-request timing) capture behavioral patterns that per-request rules miss. 3. **Application-layer DDoS** - Requests that are individually valid but collectively abnormal. A legitimate user doesn't hit 200 unique paths in 5 minutes with perfectly uniform 100ms spacing between requests. The per-IP windowed features expose this. 4. **Credential stuffing** - Repeated POST requests to login endpoints from a single IP or a distributed botnet. Per-IP features like `req_count_1m`, `error_rate_5m`, and `unique_paths_5m` build a behavioral profile that catches this even when individual requests look normal. ### Limitations Anomaly detection produces false positives. A legitimate user doing something unusual (bulk API calls, automated testing) will trigger alerts. That's why AngelusVigil uses an ensemble approach rather than relying on the autoencoder alone, and why it supports analyst review labels for active learning. It also can't tell you *what* an attack is. The autoencoder says "this is weird" but doesn't say "this is SQL injection." That's what the rule engine and random forest classifier add. ## Ensemble Learning ### What It Is Ensemble learning combines predictions from multiple models to make a final decision. The idea is that different models have different strengths and weaknesses. Combining them produces better results than any single model alone. AngelusVigil uses three models with weighted voting: ``` ┌──────────────────┐ │ Autoencoder (AE) │ Weight: 0.40 │ Unsupervised │ "How weird is this request?" └────────┬─────────┘ │ ┌──────────┐ ┌───────────┴───────────┐ ┌──────────────┐ │ Feature │───►│ Ensemble Fusion │───►│ Final Score │ │ Vector │ │ Weighted Average │ │ [0.0, 1.0] │ └──────────┘ └───────────┬───────────┘ └──────────────┘ │ ┌────────┴─────────┐ │ Random Forest │ Weight: 0.40 │ Supervised │ "Is this an attack?" └────────┬─────────┘ │ ┌────────┴─────────┐ │ Isolation Forest │ Weight: 0.20 │ Unsupervised │ "Is this an outlier?" └──────────────────┘ ``` ### Why Three Models? Each model brings a different perspective: **Autoencoder (40% weight)** - Trained only on normal traffic. Good at catching anything that looks "off" compared to the baseline. High recall (catches most attacks) but can be noisy (flags unusual but benign requests too). The reconstruction error approach means it doesn't need labeled attack data to learn. **Random Forest (40% weight)** - Trained on labeled data (both normal and attack samples). Good at classifying known attack types with high precision. Uses SMOTE oversampling to handle the class imbalance problem (attacks are rare compared to normal traffic). Can't catch attack types it wasn't trained on. **Isolation Forest (20% weight)** - Another unsupervised approach that isolates anomalies by random partitioning. Anomalies require fewer splits to isolate because they sit in sparse regions of the feature space. Gets lower weight because it overlaps somewhat with the autoencoder's anomaly detection capability but uses a fundamentally different algorithm (tree-based vs. neural network). ### Why It Matters In 2014, JPMorgan Chase was breached through a compromised web server. Their IDS flagged the initial access, but a single model's alert was classified as a false positive and ignored. The attackers maintained access for two months, eventually compromising 76 million household records. Ensemble systems make this kind of single-model failure less likely because multiple independent models would need to agree that traffic is benign. ### How Scores Are Combined The ensemble fusion happens in two stages. First, raw model scores are normalized to [0, 1]: ```python def normalize_ae_score(error: float, threshold: float) -> float: if threshold <= 0: return 0.0 return min(error / (threshold * 2), 1.0) def normalize_if_score(raw_score: float) -> float: return (1 - raw_score) / 2.0 ``` The autoencoder score is normalized against 2x the calibrated threshold. The isolation forest score is inverted (sklearn returns negative values for anomalies, positive for normal) and scaled. Then the normalized scores are fused with weighted averaging: ```python def fuse_scores(scores: dict[str, float], weights: dict[str, float]) -> float: total = 0.0 weight_sum = 0.0 for key, weight in weights.items(): if key in scores: total += scores[key] * weight weight_sum += weight if weight_sum == 0: return 0.0 return total / weight_sum ``` The fusion function gracefully handles missing models. If only the autoencoder and random forest are loaded, the isolation forest weight is excluded and the remaining weights are renormalized. This supports partial model availability during retraining. Finally, the ML ensemble score is blended with the rule engine score: ```python def blend_scores(ml_score: float, rule_score: float, ml_weight: float = 0.7) -> float: return min(ml_score * ml_weight + rule_score * (1.0 - ml_weight), 1.0) ``` ML gets 70% influence, rules get 30%. This means a high-confidence rule match (like a Log4Shell pattern) still contributes significantly to the final score even if the ML models are uncertain. ### Severity Classification The blended score maps to three severity tiers: | Score Range | Severity | Action | |---|---|---| | 0.70 - 1.00 | HIGH | Store + publish alert + WebSocket broadcast | | 0.50 - 0.69 | MEDIUM | Store + publish alert | | 0.00 - 0.49 | LOW | Log only | ## Feature Engineering for HTTP Traffic ### What It Is Feature engineering transforms raw data (nginx log lines) into numeric vectors that ML models can process. The quality of your features determines the ceiling of your model's performance. You can always improve a model's architecture, but you can't overcome bad features. ### Why It Matters A raw nginx log line looks like this: ``` 192.168.1.100 - - [15/Mar/2026:14:22:31 +0000] "GET /api/users?id=1' OR '1'='1 HTTP/1.1" 200 1234 "-" "sqlmap/1.5" ``` A human can immediately spot the SQL injection payload and the sqlmap user agent. But an ML model needs numbers. Feature engineering is the bridge between "text a human can read" and "numbers a model can learn from." ### The 35-Dimensional Feature Vector AngelusVigil extracts two types of features that are concatenated into a single 35-dimensional vector: **23 Per-Request Features** (stateless, computed from a single log entry): | Feature | Type | What It Captures | |---|---|---| | `http_method` | categorical | GET, POST, PUT, DELETE, etc. | | `path_depth` | numeric | Number of `/` segments (deeper paths are unusual) | | `path_entropy` | numeric | Shannon entropy of the path string (attack payloads have high entropy) | | `path_length` | numeric | Raw character count | | `query_string_length` | numeric | Long query strings often carry injection payloads | | `query_param_count` | numeric | Number of `&`-separated parameters | | `has_encoded_chars` | boolean | URL-encoded characters like `%27` (single quote) | | `has_double_encoding` | boolean | Double-encoded chars like `%2527` (evasion technique) | | `status_code` | numeric | HTTP response code | | `status_class` | categorical | 2xx, 3xx, 4xx, 5xx grouping | | `response_size` | numeric | Body size in bytes | | `hour_of_day` | numeric | 0-23, captures time-of-day patterns | | `day_of_week` | numeric | 0-6, captures weekly patterns | | `is_weekend` | boolean | Weekend vs weekday | | `ua_length` | numeric | User-Agent string length | | `ua_entropy` | numeric | Shannon entropy of the UA string | | `is_known_bot` | boolean | Matches known bot signatures (Googlebot, etc.) | | `is_known_scanner` | boolean | Matches scanner signatures (sqlmap, nikto, etc.) | | `has_attack_pattern` | boolean | Matches combined attack regex | | `special_char_ratio` | numeric | Ratio of non-alphanumeric characters in the path | | `file_extension` | categorical | `.php`, `.asp`, `.env`, etc. | | `country_code` | categorical | GeoIP country code | | `is_private_ip` | boolean | RFC 1918 private address | **12 Per-IP Windowed Features** (stateful, computed from Redis sorted sets): | Feature | Window | What It Captures | |---|---|---| | `req_count_1m` | 1 min | Immediate request rate (burst detection) | | `req_count_5m` | 5 min | Short-term sustained rate | | `req_count_10m` | 10 min | Medium-term pattern | | `error_rate_5m` | 5 min | Ratio of 4xx/5xx responses (scanners trigger lots of errors) | | `unique_paths_5m` | 5 min | Path diversity (scanners enumerate many paths) | | `unique_uas_10m` | 10 min | User-Agent diversity (rotating UAs is a bot fingerprint) | | `method_entropy_5m` | 5 min | Shannon entropy of HTTP methods used | | `avg_response_size_5m` | 5 min | Mean response body size | | `status_diversity_5m` | 5 min | Count of distinct status codes | | `path_depth_variance_5m` | 5 min | Variance in path depth | | `inter_request_time_mean` | 10 min | Average time between requests in ms | | `inter_request_time_std` | 10 min | Standard deviation of request intervals | ### Shannon Entropy Shannon entropy measures the randomness of a string. The formula: ``` H(s) = -Σ (count(c) / len(s)) * log2(count(c) / len(s)) ``` Normal paths like `/api/v1/users` have low entropy because characters repeat and follow predictable patterns. Attack payloads like `/api/search?q=%27%20UNION%20SELECT%20*%20FROM%20users--` have high entropy because they contain a wide variety of characters. The implementation in the codebase: ```python def _shannon_entropy(s: str) -> float: if not s: return 0.0 length = len(s) counts = Counter(s) return -sum((c / length) * math.log2(c / length) for c in counts.values()) ``` This is applied to both the request path and the User-Agent string. Legitimate User-Agents like `Mozilla/5.0 (Windows NT 10.0; Win64; x64)` have predictable entropy ranges. Scanner tools like `sqlmap/1.5.2#stable` or randomized UAs used by botnets tend to fall outside those ranges. ### Why Windowed Features Need Redis Per-request features are stateless. You can compute them from a single log line. But behavioral detection requires context: "Is this IP sending requests faster than usual? Has it hit an unusual number of distinct paths?" AngelusVigil stores per-IP request history in Redis sorted sets, with the Unix timestamp as the score. This makes windowed queries efficient: ``` ZADD ip:192.168.1.100:requests ZCOUNT ip:192.168.1.100:requests <5m_ago> +inf ``` The `ZCOUNT` operation returns how many requests from this IP fell within the last 5 minutes, computed in O(log N) time. Seven sorted sets per IP track different dimensions (requests, paths, statuses, user agents, sizes, methods, depths), and all writes + reads happen in a single pipelined round-trip to minimize latency. Keys auto-expire after 15 minutes (`KEY_TTL = 900`), so Redis memory usage stays bounded even under heavy traffic. ## Rule-Based Detection (Cold-Start) ### What It Is Rule-based detection uses handcrafted patterns to identify known attack types. AngelusVigil's rule engine is inspired by the ModSecurity Core Rule Set (CRS), the open source WAF ruleset used by OWASP. ### Why It Matters ML models need training data. On day one, you have no data and no models. The rule engine provides immediate protection with zero training, covering the most common web attack categories. Even after ML models are trained, rules remain valuable. They catch well-known attacks with high confidence and provide explainability ("this request matched the SQL injection pattern") that ML scores alone can't. ### Attack Pattern Rules The rule engine evaluates 9 regex-based patterns against each request URI: | Rule | Score | What It Catches | |---|---|---| | LOG4SHELL | 0.95 | `${jndi:ldap://...}` and variations (CVE-2021-44228) | | COMMAND_INJECTION | 0.90 | Shell metacharacters, backtick execution, pipe chains | | SQL_INJECTION | 0.85 | `UNION SELECT`, `OR 1=1`, comment sequences, `SLEEP()` | | XSS | 0.80 | `