chore: add demos for projects, update haskell-reverse-proxy modules, refresh siem assets
- Add DEMO.md and screenshots for bug-bounty-platform, hash-cracker, linux-cis-hardening-auditor, simple-port-scanner, simple-vulnerability-scanner, systemd-persistence-scanner, base64-tool, caesar-cipher, dns-lookup, metadata-scrubber-tool, network-traffic-analyzer, siem-dashboard - Link DEMO.md from project READMEs - Add .gitignore entries for DEMO-TRACKER and simple-port-scanner build dir - Restructure haskell-reverse-proxy with DDoS, Fingerprint, ML, RateLimit, WAF, Geo, and Honeypot modules; drop superseded research docs and old Makefile - Refresh siem-dashboard dashboard.png and rename alerts.png to alert-detail.png
This commit is contained in:
CarterPerez-dev
parent
83e3b2b762
commit
ef315b072b
|
|
@ -5,5 +5,6 @@
|
|||
.env
|
||||
.angelusvigil/
|
||||
.worktrees/
|
||||
DEMO-TRACKER.md
|
||||
|
||||
PROJECTS/beginner/hash-cracker/docs/
|
||||
|
|
|
|||
|
|
@ -0,0 +1,89 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
██████╗ ██████╗ ██╗ ██╗███╗ ██╗████████╗██╗ ██╗
|
||||
██╔══██╗██╔═══██╗██║ ██║████╗ ██║╚══██╔══╝╚██╗ ██╔╝
|
||||
██████╔╝██║ ██║██║ ██║██╔██╗ ██║ ██║ ╚████╔╝
|
||||
██╔══██╗██║ ██║██║ ██║██║╚██╗██║ ██║ ╚██╔╝
|
||||
██████╔╝╚██████╔╝╚██████╔╝██║ ╚████║ ██║ ██║
|
||||
╚═════╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://bugbounty.carterperez-dev.com">
|
||||
<img src="https://img.shields.io/badge/▶_TRY_IT_LIVE-bugbounty.carterperez--dev.com-DC143C?style=for-the-badge&logo=googlechrome&logoColor=white" alt="Live Demo"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
docker compose up -d → localhost:8420
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Landing](#landing) · [Sign Up](#sign-up) · [Login](#login) · [Programs](#programs) · [Company Programs](#company-programs) · [Settings](#settings) · [User Management](#user-management)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Landing
|
||||
|
||||
Public marketing page introducing the four core flows — browse programs, submit reports, responsible disclosure, and earn rewards
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Sign Up
|
||||
|
||||
Email and password account creation with confirmation field and inline validation
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Login
|
||||
|
||||
Credential-based authentication issuing rotating JWT refresh tokens with multi-device session tracking
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Programs
|
||||
|
||||
Researcher view of active bug bounty programs with reward summary and response SLA per program
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Company Programs
|
||||
|
||||
Company-side program management with status, creation date, and inline view and edit controls per program
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Settings
|
||||
|
||||
Profile management and password rotation with current-password verification
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### User Management
|
||||
|
||||
Admin user directory with role and status filtering, account provisioning, and inline edit and delete actions
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 31 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
|
|
@ -21,3 +21,6 @@ examples/certs/
|
|||
|
||||
# Private progress tracking
|
||||
refer/
|
||||
|
||||
# Development docs (research, status, plans) - not for public repo
|
||||
docs/
|
||||
|
|
|
|||
|
|
@ -1,63 +0,0 @@
|
|||
# Ᾰenebris Makefile - Common development commands
|
||||
|
||||
.PHONY: help build run logs stop restart clean test backend kill-all
|
||||
|
||||
help:
|
||||
@echo "Ᾰenebris Development Commands:"
|
||||
@echo ""
|
||||
@echo " make build - Build the project"
|
||||
@echo " make run - Start Ᾰenebris proxy"
|
||||
@echo " make logs - Watch logs in real-time"
|
||||
@echo " make stop - Stop the proxy"
|
||||
@echo " make restart - Restart the proxy"
|
||||
@echo " make clean - Clean build artifacts"
|
||||
@echo ""
|
||||
@echo " make backend - Start test backend (port 8000)"
|
||||
@echo " make test - Run tests (when implemented)"
|
||||
@echo " make kill-all - Kill proxy + backend"
|
||||
@echo ""
|
||||
|
||||
build:
|
||||
@echo "Building Ᾰenebris..."
|
||||
@stack build
|
||||
|
||||
run:
|
||||
@echo "Starting Ᾰenebris on port 8081..."
|
||||
@nohup stack run examples/config.yaml > aenebris.log 2>&1 &
|
||||
@sleep 2
|
||||
@echo "Proxy started! PID: $$(pgrep -f 'aenebris examples' | head -1)"
|
||||
@echo "Run 'make logs' to watch output"
|
||||
|
||||
logs:
|
||||
@echo "Watching Ᾰenebris logs (Ctrl+C to exit)..."
|
||||
@tail -f aenebris.log
|
||||
|
||||
stop:
|
||||
@echo "Stopping Ᾰenebris..."
|
||||
@pkill -f 'aenebris examples' || echo "Proxy not running"
|
||||
|
||||
restart: stop
|
||||
@sleep 1
|
||||
@make run
|
||||
|
||||
clean:
|
||||
@echo "Cleaning build artifacts..."
|
||||
@stack clean
|
||||
@rm -f aenebris.log nohup.out
|
||||
|
||||
backend:
|
||||
@echo "Starting test backend on port 8000..."
|
||||
@python3 examples/test_backend.py > backend.log 2>&1 &
|
||||
@sleep 1
|
||||
@echo "Backend started! PID: $$(pgrep -f test_backend | head -1)"
|
||||
|
||||
# Run tests (placeholder for now)
|
||||
test:
|
||||
@echo "Running tests..."
|
||||
@stack test
|
||||
|
||||
kill-all:
|
||||
@echo "Killing all processes..."
|
||||
@pkill -f 'aenebris examples' || echo "Proxy not running"
|
||||
@pkill -f test_backend || echo "Backend not running"
|
||||
@echo "All processes stopped"
|
||||
|
|
@ -27,6 +27,19 @@ library
|
|||
, Aenebris.Tunnel
|
||||
, Aenebris.Middleware.Security
|
||||
, Aenebris.Middleware.Redirect
|
||||
, Aenebris.RateLimit
|
||||
, Aenebris.DDoS.EarlyData
|
||||
, Aenebris.DDoS.MemoryShed
|
||||
, Aenebris.DDoS.IPJail
|
||||
, Aenebris.DDoS.ConnLimit
|
||||
, Aenebris.Fingerprint.JA4H
|
||||
, Aenebris.WAF.Rule
|
||||
, Aenebris.WAF.Patterns
|
||||
, Aenebris.WAF.Engine
|
||||
, Aenebris.Honeypot
|
||||
, Aenebris.Geo
|
||||
, Aenebris.ML.Features
|
||||
, Aenebris.ML.Model
|
||||
default-language: Haskell2010
|
||||
build-depends: base >= 4.7 && < 5
|
||||
, warp >= 3.3
|
||||
|
|
@ -45,15 +58,20 @@ library
|
|||
, containers >= 0.6
|
||||
, vector >= 0.12
|
||||
, tls >= 2.1
|
||||
, x509 >= 1.7
|
||||
, x509-store >= 1.6
|
||||
, x509-validation >= 1.6
|
||||
, crypton-x509 >= 1.7
|
||||
, crypton-x509-store >= 1.6
|
||||
, crypton-x509-validation >= 1.6
|
||||
, case-insensitive >= 1.2
|
||||
, directory >= 1.3
|
||||
, data-default-class >= 0.1
|
||||
, network >= 3.1
|
||||
, streaming-commons >= 0.2
|
||||
, splice >= 0.6
|
||||
, crypton >= 1.0
|
||||
, memory >= 0.18
|
||||
, regex-tdfa >= 1.3
|
||||
, geoip2 >= 0.4
|
||||
, iproute >= 1.7
|
||||
ghc-options: -Wall
|
||||
-Wcompat
|
||||
-Widentities
|
||||
|
|
@ -82,8 +100,20 @@ test-suite aenebris-test
|
|||
build-depends: base >= 4.7 && < 5
|
||||
, aenebris
|
||||
, hspec >= 2.0
|
||||
, wai >= 3.2
|
||||
, wai-extra >= 3.0
|
||||
, http-types >= 0.12
|
||||
, stm >= 2.5
|
||||
, bytestring >= 0.11
|
||||
, text >= 2.0
|
||||
, case-insensitive >= 1.2
|
||||
, time >= 1.9
|
||||
, containers >= 0.6
|
||||
, iproute >= 1.7
|
||||
, network >= 3.1
|
||||
, geoip2 >= 0.4
|
||||
, yaml >= 0.11
|
||||
, vector >= 0.12
|
||||
ghc-options: -Wall
|
||||
-threaded
|
||||
-rtsopts
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -1,245 +0,0 @@
|
|||
# HTTP/2 and HTTP/3 in Haskell: Production Implementation Guide
|
||||
|
||||
**The Haskell ecosystem provides production-ready HTTP/2 support through the mature `http2` library and Warp web server, with experimental HTTP/3 capabilities emerging.** HTTP/2 delivers 14-30% performance improvements for most websites, while HTTP/3 adds another 12-50% boost particularly on mobile and high-latency networks. The protocol stack maintained by Kazu Yamamoto achieves nginx-comparable performance and powers major Haskell web applications today.
|
||||
|
||||
For Haskell developers, HTTP/2 is ready for immediate production deployment with Warp 3.1+, offering automatic ALPN negotiation and transparent multiplexing. HTTP/3 support exists through the `quic` and `http3` libraries, though it remains in active development (version 0.2.x) and is best deployed via reverse proxies for production systems. The elimination of head-of-line blocking and 0-RTT connection establishment make HTTP/3 particularly valuable for mobile-first applications and global audiences, while the unified architecture across all three libraries ensures smooth adoption paths.
|
||||
|
||||
## Understanding the protocol evolution and Haskell's position
|
||||
|
||||
HTTP/2 represented a fundamental shift from text-based to binary framing, introducing multiplexing that allows concurrent streams over a single TCP connection. This eliminated HTTP/1.1's "six connections per domain" bottleneck and reduced connection overhead. The `http2` Haskell library implements the complete RFC 7540 specification, including HPACK header compression that achieves 40-80% size reduction, sophisticated priority queues using custom-designed data structures, and comprehensive flow control mechanisms. First released in 2015 and now at version 5.3.10 (June 2025), the library has proven itself through 70+ releases and extensive production deployment in Warp, Yesod, and mighttpd2.
|
||||
|
||||
HTTP/3 takes this evolution further by replacing TCP entirely with QUIC, a UDP-based transport protocol developed initially by Google. This architectural change eliminates transport-layer head-of-line blocking that still affects HTTP/2, reduces connection establishment from 2 RTT to 1 RTT (or 0 RTT on reconnection), and enables connection migration when devices switch networks. The Haskell `quic` library (version 0.2.20, September 2025) implements the complete IETF QUIC specification including RFC 9000, 9001, 9002, and Version 2, while the `http3` library (version 0.1.1) provides the HTTP/3 protocol layer. All three libraries share the same maintainer and architectural philosophy based on Haskell lightweight threads, ensuring consistency across the stack.
|
||||
|
||||
## HTTP/2 protocol deep dive: what Haskell developers need to know
|
||||
|
||||
**Multiplexing operates through a binary framing layer** that sits between the socket and HTTP API. Every HTTP/2 communication splits into frames with a 9-byte header (length, type, flags, stream ID) plus variable payload. Streams represent bidirectional flows of frames within a single connection, with odd-numbered streams initiated by clients and even-numbered by servers. The critical insight is that frames from different streams can interleave freely—a DATA frame from stream 5, followed by a HEADERS frame from stream 3, then another DATA frame from stream 5—all without blocking.
|
||||
|
||||
This multiplexing eliminates the connection limit problem but introduces complexity in stream state management. Streams transition through states (idle → open → half-closed → closed) with specific rules about which frame types are valid in each state. The Haskell `http2` library handles this state machine internally, mapping each HTTP/2 stream to a Haskell lightweight thread. This design choice proves elegant: **one thread per stream (not per connection)** allows natural concurrent processing while maintaining clear isolation between streams.
|
||||
|
||||
Flow control prevents any single stream from monopolizing bandwidth. HTTP/2 implements credit-based flow control at both stream and connection levels, starting with 65,535 bytes for each window. Senders must track available window space and queue DATA frames when exhausted, while receivers send WINDOW_UPDATE frames as they consume data. The `http2` library manages these windows automatically, but developers should understand the implications: slow consumption in application code can cause flow control windows to close, throttling the entire connection.
|
||||
|
||||
**Priority mechanisms in HTTP/2 allow resource ordering through dependency trees and weights**, though real-world deployment shows limited effectiveness. The specification supports complex parent-child relationships with weights 1-256 determining proportional resource sharing among siblings. However, many implementations use simpler schemes—Chrome uses sequential exclusive dependencies, and research shows complex trees suffer from poor interoperability. The `http2` library implements priority queues using a custom "random heap" data structure invented specifically for this purpose, but developers should focus on simple weight-based prioritization rather than complex dependency trees.
|
||||
|
||||
**Server push, once considered HTTP/2's killer feature, is now deprecated** and removed from Chrome 106+ (October 2022) and Firefox 132+ (October 2024). The PUSH_PROMISE frame allowed servers to speculatively send resources before clients requested them, but practice revealed fatal flaws: servers cannot know client cache state, leading to wasted bandwidth; predicting what to push proved nearly impossible; and better alternatives like HTTP 103 Early Hints emerged. The Haskell `http2` library supports server push for compatibility, but new implementations should skip it entirely in favor of preload hints.
|
||||
|
||||
**HPACK compression achieves 40-80% header size reduction** through a combination of static tables (61 predefined common headers), dynamic tables (connection-specific learned patterns), and Huffman encoding. The static table includes entries like index 2 for `:method GET` and index 8 for `:status 200`, allowing entire headers to encode in 1-2 bytes. The dynamic table grows as the connection processes headers, building compression context. Four representation types handle different scenarios: indexed (both name and value in table), literal with incremental indexing (adds to table), literal without indexing (one-time use), and literal never indexed (for sensitive data like Authorization headers).
|
||||
|
||||
HPACK's design specifically mitigates the CRIME attack that plagued generic compression. By avoiding cross-message compression and using static Huffman coding instead of adaptive algorithms, HPACK prevents attackers from using compression ratios to guess secret values. The never-indexed flag ensures sensitive headers never enter the compression context. The Haskell implementation handles HPACK state carefully through STM for thread-safe dynamic table management and precomputed lookup tables for efficient encoding/decoding.
|
||||
|
||||
## HTTP/3 and QUIC: rebuilding the transport layer
|
||||
|
||||
**QUIC's use of UDP instead of TCP represents pragmatic engineering rather than technical preference.** TCP is ossified—implemented in operating system kernels across billions of devices, making updates essentially impossible. Network middleboxes (firewalls, load balancers, NAT devices) are hardcoded for TCP behavior, blocking attempts to deploy new transport protocols. By building on UDP, which already passes through all infrastructure, QUIC can be implemented in user space at the application layer. This enables rapid iteration: Google deployed 18 versions of QUIC in 2 years, something unthinkable with a kernel-level protocol.
|
||||
|
||||
QUIC reimplements all TCP's reliability features in the application layer with improvements. Monotonically increasing packet numbers eliminate retransmission ambiguity that affects TCP RTT calculations. Each packet has a unique number even across retransmissions, allowing precise loss detection. ACK frames can acknowledge multiple packet ranges efficiently with included delay information for accurate measurements. Loss detection uses both packet threshold (3 missing packets) and time threshold (based on smoothed RTT), with lost frames retransmitted in new packets with new numbers.
|
||||
|
||||
**Connection IDs represent QUIC's most innovative feature**, identifying connections independently of the network 4-tuple (source IP/port, destination IP/port). This enables connection migration when IP addresses change—mobile devices switching from WiFi to cellular, laptops roaming between access points, or NAT rebindings. Each endpoint selects Connection IDs for packets sent to it, and multiple IDs can exist per connection. Path validation ensures new paths work: PATH_CHALLENGE frames sent on the new path require PATH_RESPONSE confirmations before switching. New Connection IDs exchanged during migration prevent linkability by observers, enhancing privacy as connections move across networks.
|
||||
|
||||
**0-RTT connection establishment eliminates handshake overhead on resumption**, saving 25-200ms depending on network latency. After an initial 1-RTT connection where the server provides a session ticket, subsequent connections can send application data immediately in 0-RTT packets encrypted with keys derived from cached parameters. This proves particularly valuable for mobile networks where every round trip costs 50-150ms. However, 0-RTT introduces security concerns: the data lacks forward secrecy and is vulnerable to replay attacks. Mitigations include server-side anti-replay mechanisms, rejecting non-idempotent methods (POST, PUT, DELETE) in 0-RTT, and the Early-Data header allowing origin servers to detect and handle with `425 Too Early` status codes. Browsers typically only send safe requests (GET, HEAD) in 0-RTT.
|
||||
|
||||
**HTTP/3 eliminates the transport-layer head-of-line blocking that still affects HTTP/2.** With HTTP/2 over TCP, a lost packet blocks all streams until retransmitted because TCP guarantees ordered byte delivery. At 2% packet loss, HTTP/1.1 with 6 parallel connections can outperform HTTP/2 with its single connection. QUIC provides independent streams with per-stream loss recovery: a lost packet only affects its specific stream while others continue uninterrupted. This architectural difference shows most dramatically on lossy networks—the Kiwee study with 15% packet loss demonstrated HTTP/3 being 52% faster than HTTP/2.
|
||||
|
||||
Connection migration eliminates interruptions when network paths change. Traditional TCP connections identified by 4-tuple break when IP or port changes, requiring full reconnection (TCP + TLS handshakes). QUIC connections survive these changes seamlessly through Connection ID routing and path validation. While helpful for mobile users, research shows switching networks happens less frequently than initially assumed, and congestion control must still probe the new network's capacity. The feature proves most valuable for real-time applications like video conferencing, navigation apps, and gaming where connection continuity matters more than bulk transfer speed.
|
||||
|
||||
**QPACK modifies HPACK's compression approach to handle QUIC's out-of-order delivery.** HPACK assumes all dynamic table updates arrive in order, working perfectly over TCP but breaking over QUIC where header blocks might reference entries not yet received. QPACK introduces separate encoder and decoder streams for table management, required insert counts indicating the highest dynamic table index referenced, and acknowledgment tracking preventing references to unevicted entries. Encoders must balance three strategies: static table references (safe, no blocking), acknowledged dynamic references (safe, good compression), and unacknowledged dynamic references (best compression, may block stream). The `SETTINGS_QPACK_BLOCKED_STREAMS` setting controls this trade-off, with many implementations using only static tables for simplicity.
|
||||
|
||||
## Haskell library ecosystem: maturity and production readiness
|
||||
|
||||
**The `http2` library represents one of Haskell's most mature network implementations**, with production deployment proving its reliability at scale. Current version 5.3.10 (June 26, 2025) shows continuous maintenance through regular updates across 2024-2025. The library provides complete HTTP/2 frame support (DATA, HEADERS, PRIORITY, RST_STREAM, SETTINGS, PUSH_PROMISE, PING, GOAWAY, WINDOW_UPDATE, CONTINUATION), comprehensive HPACK implementation without reference sets, sophisticated priority queue handling with random heaps, and both client and server components.
|
||||
|
||||
Performance benchmarks place Warp with `http2` at nginx-level performance despite being written in Haskell, a remarkable achievement validated by the "Experience Report: Developing High Performance HTTP/2 Server in Haskell" paper at the 2016 Haskell Symposium and the AOSA book chapter. The architecture maps HTTP/2 streams to Haskell lightweight threads using thread pools to minimize spawning overhead. Critical paths use hand-rolled parsers rather than combinator libraries, specialized date formatting with caching, and zero-copy ByteString operations where possible. DoS attack mitigations added in version 3.0+ protect against various HTTP/2-specific attack vectors.
|
||||
|
||||
Real-world usage proves the library's production readiness. Warp (one of the fastest HTTP servers in any language) uses it as the HTTP/2 implementation. Yesod web framework deploys it for all HTTP/2 support. The mighttpd2 production web server serves traffic with it. Dependency on http-semantics, time-manager, and network-control packages provides clean separation of concerns. Known limitations are minor: the library exposes low-level primitives requiring careful usage, HTTP/1.1 Upgrade to HTTP/2 is not supported (only direct HTTP/2 and ALPN), and some features like PING replies are hardcoded.
|
||||
|
||||
**The `quic` library implements complete IETF QUIC specifications** including RFC 9000 (QUIC transport), RFC 9001 (TLS integration), RFC 9002 (loss detection and congestion control), RFC 9287 (bit greasing), RFC 9369 (QUIC Version 2), and RFC 9368 (version negotiation). Current version 0.2.20 (September 3, 2025) shows active maintenance by the same author as `http2`, ensuring architectural consistency. The library provides both QUIC v1 and v2 support, implements RFC-compliant congestion control algorithms, supports both automatic and manual migration, and includes client and server implementations validated with h3spec compliance testing.
|
||||
|
||||
Production readiness assessment places `quic` at 4 out of 5 stars—very good but newer than `http2`. The library successfully deploys in mighttpd2 v4.0.0+ for production HTTP/3 serving. Documentation comes primarily through blog posts and examples rather than comprehensive guides, a gap that reflects the library's relative youth (first released around 2021, compared to `http2`'s 2015 debut). The smaller ecosystem compared to `http2` means fewer dependent packages and less extensive real-world testing, though the fundamental implementation is sound and RFC-compliant.
|
||||
|
||||
**The `http3` library bridges QUIC transport with HTTP/3 protocol**, building on both `quic` and `http2` for shared HTTP semantics. Version 0.1.1 (August 11, 2025) remains in 0.x territory, signaling ongoing development. The library handles HTTP/3 frame encoding/decoding, QPACK header compression, both client and server components, and TLS 1.3 integration (requiring tls library ≥2.1.10). Dependencies on quic ≥0.2.11 and http2 ≥5.3.4 ensure compatibility across the stack. Production readiness assessment gives it 3 out of 5 stars—good and functional but evolving, suitable for early production use with appropriate monitoring.
|
||||
|
||||
## Warp integration: HTTP/2 and HTTP/3 support
|
||||
|
||||
**Warp has supported HTTP/2 natively since version 3.1.0 (July 2015)**, making it one of the earliest HTTP/2 implementations in any language. Current version 3.4.9 (September 13, 2025) integrates http2 library versions 5.1-5.4 directly. The implementation supports direct HTTP/2 (h2c cleartext) and ALPN negotiation over TLS (h2 with warp-tls), but explicitly does not support the HTTP/1.1 Upgrade mechanism. This design choice reflects practical deployment: browsers only use ALPN for HTTP/2, and h2c serves primarily server-to-server communication like gRPC.
|
||||
|
||||
Configuration for HTTP/2 is remarkably simple—it works automatically when using warp-tls with TLS ALPN or when clients connect with the HTTP/2 preface for direct h2c. No special configuration flags are needed:
|
||||
|
||||
```haskell
|
||||
import Network.Wai.Handler.Warp (run)
|
||||
import Network.Wai.Handler.WarpTLS (runTLS, tlsSettings, defaultTlsSettings)
|
||||
|
||||
-- For TLS with automatic HTTP/2 ALPN negotiation
|
||||
main = runTLS tlsSettings defaultSettings app
|
||||
|
||||
-- For cleartext with HTTP/2 support
|
||||
main = run 3000 app
|
||||
```
|
||||
|
||||
The Network.Wai.HTTP2 module provides HTTP/2-specific APIs including `HTTP2Application` for HTTP/2-aware application interfaces, `PushPromise` for server push support (though deprecated), and `promoteApplication` to upgrade HTTP/1.1 apps to support HTTP/2. Performance characteristics match the underlying `http2` library: better than HTTP/1.1 in throughput tests, one thread per stream (not per connection) architecture, efficient thread pool usage, and performance comparable to nginx based on AOSA book benchmarks.
|
||||
|
||||
**HTTP/3 support exists through the experimental warp-quic package** at version 0.0.3 (June 10, 2025). This provides a WAI handler built on `http3` and `quic` libraries, using the same WAI Application interface for consistency. Production readiness assessment gives warp-quic 3 out of 5 stars—experimental with limited deployment history. For production systems, the recommended approach uses an HTTP/3-capable reverse proxy (Nginx 1.25.0+ or Caddy 2.6.0+) in front of Warp:
|
||||
|
||||
```
|
||||
Client → Nginx (HTTP/3 on UDP/443) → Warp (HTTP/2 on TCP/443)
|
||||
```
|
||||
|
||||
This architecture provides HTTP/3 benefits to clients while maintaining Warp's proven HTTP/2 stability internally. The proxy handles protocol translation, UDP processing overhead, and provides mature HTTP/3 implementations while Haskell applications continue using Warp's production-tested interface.
|
||||
|
||||
## Performance analysis: when each protocol matters
|
||||
|
||||
**HTTP/2 delivers 14-30% performance improvements for most websites**, with benefits increasing for high-resource sites and mobile users. Google search saw 8% faster desktop performance and 3.6% faster mobile, with the slowest 1-10% of users experiencing up to 16% improvement. ImageKit's demo loading 100 image tiles showed dramatic visual differences, with HTTP/2's parallel loading versus HTTP/1.1's sequential batches limited by 6 parallel connections. The benefits emerge most clearly for websites with 100+ resources, multiple small files (CSS, JS, images), and high-latency networks where the single multiplexed connection eliminates repeated handshake overhead.
|
||||
|
||||
HTTP/1.1 can still perform better in specific scenarios: high packet loss environments where HTTP/2's head-of-line blocking at the TCP level causes worse performance than HTTP/1.1's multiple independent connections, simple static sites with fewer than 10-20 resources where migration overhead isn't justified, and API endpoints serving single JSON responses where multiplexing provides no benefit. The critical threshold is packet loss—at 2% loss, HTTP/1.1 with 6 connections can outperform HTTP/2's single connection, as lost packets block all streams until retransmitted.
|
||||
|
||||
**HTTP/3 adds another 12-50% improvement on top of HTTP/2, with the largest gains on problematic networks.** Cloudflare's real-world testing measured Time to First Byte improvements of 12.4% (176ms vs 201ms average), though page load times showed HTTP/3 trailing HTTP/2 by 1-4% in good network conditions, attributed to congestion algorithm differences (BBR v1 vs CUBIC). The real benefits emerge on high-latency and lossy networks: Request Metrics benchmarks from New York (1,000 miles) showed HTTP/3 200-300ms faster, while London (transatlantic) showed 600-1,200ms improvements, and Bangalore demonstrated the most dramatic gains with tightly grouped response times.
|
||||
|
||||
Mobile networks demonstrate HTTP/3's transformative impact. Wix's study across millions of websites showed connection setup improvements of up to 33% at the mean, with 75th percentile improvements exceeding 250ms in countries like the Philippines. Largest Contentful Paint (LCP) improved by up to 20% at the 75th percentile, reducing LCP by over 500ms in many cases—approximately one-fifth of Google's 2,500ms target. The Kiwee study with simulated poor conditions (15% packet loss, 100ms latency) measured 52% faster downloads with HTTP/3. YouTube reported 20% less video stalling in countries like India with QUIC.
|
||||
|
||||
The performance hierarchy becomes clear across network conditions: excellent networks (fiber, low latency, no loss) show HTTP/2 providing 14% improvement while HTTP/3 adds marginal 1-4% benefit; moderate networks (typical cellular, 50-100ms RTT, 1-5% loss) see HTTP/2 improving 30-50% with HTTP/3 adding substantial 15-25% more; poor networks (rural/satellite, 100-200ms+ RTT, 5-15% loss) can see HTTP/2 performing worse than HTTP/1.1 due to head-of-line blocking while HTTP/3 delivers dramatic 40-50%+ improvements.
|
||||
|
||||
**Overhead considerations reveal important trade-offs.** HTTP/2 uses approximately the same CPU as HTTP/1.1 without encryption, with the binary protocol reducing parsing overhead versus text-based HTTP/1.1. HTTP/3 and QUIC prove "much more expensive to host" due to UDP packet processing overhead, per-packet encryption versus bulk encryption in TLS over TCP, and user-space implementation versus kernel-space TCP. Memory usage increases with HTTP/2 (maintaining 40,000 sessions requires significant RAM) and further with HTTP/3's connection state management in user space. Connection setup costs decrease from 3 RTT (HTTP/1.1 with TLS 1.2) to 2 RTT (HTTP/2 with TLS 1.3) to 1 RTT (HTTP/3), with 0-RTT mode enabling immediate data transmission on reconnection.
|
||||
|
||||
The overhead justification depends on scale and use case. Facebook uses HTTP/3 client-to-edge but HTTP/2 for data center traffic due to overhead concerns. Netflix sticks with heavily optimized TCP+TLS at their scale. CDN providers like Cloudflare deploy HTTP/3 globally because the user experience benefits justify the CPU cost. The key insight: overhead matters most for internal microservices on reliable networks, while user-facing applications on diverse networks justify the cost through improved experience.
|
||||
|
||||
## Protocol comparison: HTTP/1.1 vs HTTP/2 vs HTTP/3
|
||||
|
||||
| Feature | HTTP/1.1 | HTTP/2 | HTTP/3 |
|
||||
|---------|----------|--------|--------|
|
||||
| **Transport** | TCP | TCP | QUIC over UDP |
|
||||
| **Framing** | Text-based, newline-delimited | Binary frames | Binary frames |
|
||||
| **Multiplexing** | No (6 connections) | Yes (streams in 1 connection) | Yes (streams in 1 connection) |
|
||||
| **Head-of-line blocking** | Application layer (sequential) | Transport layer (TCP) | None (per-stream loss recovery) |
|
||||
| **Connection setup** | 3 RTT (TCP + TLS 1.2) | 2 RTT (TCP + TLS 1.3) | 1 RTT (0 RTT on resume) |
|
||||
| **Header compression** | None | HPACK (40-80% reduction) | QPACK (similar to HPACK) |
|
||||
| **Server push** | No | Yes (deprecated 2022) | Yes (but discouraged) |
|
||||
| **Priority** | No | Weight + dependency tree | Simplified (RFC 9218) |
|
||||
| **Connection migration** | No | No | Yes (Connection IDs) |
|
||||
| **Encryption** | Optional (HTTPS) | Required in browsers (ALPN) | Required (TLS 1.3 mandatory) |
|
||||
| **Browser support** | 100% | 97% | 92% |
|
||||
| **Web adoption** | ~20% | ~47% | ~30% |
|
||||
| **Typical improvement** | Baseline | +14-30% | +12-50% on top of HTTP/2 |
|
||||
| **Best for** | Simple sites, APIs | Modern websites, SPAs | Mobile, global audiences, lossy networks |
|
||||
| **Haskell maturity** | Mature | Production-ready (Warp 3.1+) | Experimental (warp-quic 0.0.3) |
|
||||
|
||||
## Fallback strategies and protocol negotiation
|
||||
|
||||
**Graceful degradation happens automatically through protocol negotiation.** Servers listen simultaneously on TCP/443 for HTTP/1.1 and HTTP/2, and UDP/443 for HTTP/3. Clients discover HTTP/3 via the Alt-Svc header (`Alt-Svc: h3=":443"; ma=86400`) or DNS HTTPS records with ALPN parameters. Browsers race QUIC versus TCP connections, falling back silently within 200-500ms if QUIC fails. The fallback hierarchy flows naturally: HTTP/3 → HTTP/2 → HTTP/1.1, with clients driving selection and servers requiring no detection before connection establishment.
|
||||
|
||||
**Application-Layer Protocol Negotiation (ALPN) from RFC 7301 enables this seamlessness.** The client sends TLS ClientHello with an ALPN extension listing supported protocols in preference order (`["h3", "h2", "http/1.1"]`). The server selects the highest mutually supported protocol and returns it in TLS ServerHello. The connection proceeds with the negotiated protocol. Protocol identifiers are: `h3` for HTTP/3 over QUIC/UDP with TLS 1.3, `h2` for HTTP/2 over TCP with TLS, `h2c` for HTTP/2 cleartext (no TLS, uses Upgrade mechanism), and `http/1.1` for HTTP/1.1 with optional TLS. ALPN is mandatory for HTTP/2 over TLS, and servers must not respond with one protocol then use another.
|
||||
|
||||
The h2c alternative uses HTTP/1.1 Upgrade mechanism for cleartext HTTP/2, primarily serving server-to-server communication like gRPC. Browsers do not implement h2c, making it irrelevant for typical web applications. The Upgrade mechanism sends an HTTP/1.1 request with `Connection: Upgrade, HTTP2-Settings` and `Upgrade: h2c` headers, receiving `HTTP/1.1 101 Switching Protocols` if accepted.
|
||||
|
||||
**Client compatibility handling requires understanding current support status.** HTTP/2 has 97/100 browser compatibility (Chrome 41+, Firefox 36+, Safari 9.3+, Edge all versions), while HTTP/3 reaches 92/100 (Chrome 87+, Firefox 88+, Edge 87+, Safari 14+ with manual enable until 17.6). As of November 2025, 25-30% of web traffic uses HTTP/3, showing rapid adoption growth that doubled in 12 months (2021-2022). Feature detection should use server-side protocol logging rather than user-agent strings, which prove unreliable and easily spoofed.
|
||||
|
||||
Mobile versus desktop considerations prove important: mobile benefits more from HTTP/3 due to high latency and connection migration, with mobile HTTP/3 usage at 25-30% versus desktop's 20-25%. Intermediaries and proxies introduce complications—UDP blocking is common in corporate firewalls requiring TCP fallback, transparent proxies may downgrade to HTTP/1.1, and TLS inspection can break ALPN unless proxies support ALPN forwarding. Testing from mobile, desktop, and corporate networks becomes essential for production deployment.
|
||||
|
||||
## Implementation roadmap for Haskell projects
|
||||
|
||||
**Phase 1: HTTP/2 deployment (immediate, production-ready)**
|
||||
|
||||
Start with Warp 3.1+ and warp-tls for automatic HTTP/2 support. The minimal configuration requires TLS certificates and runs automatically:
|
||||
|
||||
```haskell
|
||||
import Network.Wai
|
||||
import Network.Wai.Handler.Warp
|
||||
import Network.Wai.Handler.WarpTLS
|
||||
|
||||
main :: IO ()
|
||||
main = do
|
||||
let tlsSettings = tlsSettingsChain "cert.pem" ["intermediate.pem"] "key.pem"
|
||||
warpSettings = setPort 443 defaultSettings
|
||||
runTLS tlsSettings warpSettings app
|
||||
```
|
||||
|
||||
Protocol detection happens transparently through ALPN during TLS handshake. Applications can access HTTP/2 data through `Warp.getHTTP2Data req` if needed, though most applications work unchanged across HTTP/1.1 and HTTP/2. Avoid implementing server push (deprecated), use preload hints instead: `<link rel="preload" href="/critical.css" as="style">`.
|
||||
|
||||
Configuration optimization follows HTTP/2 best practices: stop domain sharding (counterproductive with multiplexing), stop excessive bundling (leverage parallel loading), minimize buffering to enable prioritization, and set appropriate concurrent stream limits. Monitoring should track protocol distribution (% HTTP/1.1 vs HTTP/2), connection establishment time, Time to First Byte (TTFB), and successful ALPN negotiations.
|
||||
|
||||
**Phase 2: HTTP/3 experimentation (current state, use with caution)**
|
||||
|
||||
Deploy HTTP/3 through reverse proxy architecture for production systems. Nginx 1.25.0+ (May 2023) or Caddy 2.6.0+ (September 2022) provide mature HTTP/3 implementations:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
# HTTP/2 over TCP
|
||||
listen 443 ssl;
|
||||
listen [::]:443 ssl;
|
||||
http2 on;
|
||||
|
||||
# HTTP/3 over UDP
|
||||
listen 443 quic reuseport;
|
||||
listen [::]:443 quic reuseport;
|
||||
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
add_header Alt-Svc 'h3=":443"; ma=86400' always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:3000; # Warp
|
||||
proxy_http_version 1.1;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Firewall configuration must allow UDP/443: `iptables -A INPUT -p udp --dport 443 -j ACCEPT`. The Alt-Svc header enables HTTP/3 discovery by clients. This architecture provides HTTP/3 benefits to users while maintaining Warp's proven stability internally.
|
||||
|
||||
Direct Haskell HTTP/3 deployment using warp-quic is possible for experimental projects:
|
||||
|
||||
```haskell
|
||||
import Network.Wai.Handler.WarpQUIC
|
||||
|
||||
-- Uses same WAI Application interface
|
||||
main = runWarpQUIC settings app
|
||||
```
|
||||
|
||||
However, production use should wait for version 1.0+ and broader deployment validation. Current version 0.0.3 indicates experimental status with limited field testing.
|
||||
|
||||
**Phase 3: Production hardening**
|
||||
|
||||
Security considerations require TLS 1.3 for HTTP/3, TLS 1.2+ for HTTP/2, and careful 0-RTT handling (only safe requests, monitor for replay attacks). Implement rate limiting on UDP/443 to prevent amplification attacks. Monitor CPU usage, as QUIC processing overhead in user space exceeds kernel-space TCP.
|
||||
|
||||
Testing strategy should include WebPageTest with "3G Fast" profile to verify prioritization, http3check.net for HTTP/3 support verification, curl with `--http3` flag for command-line testing, and Chrome DevTools network tab with Protocol column added. Test from multiple network types: cellular (high latency, packet loss), enterprise (potential UDP blocking), and international (high RTT) to ensure fallback mechanisms work correctly.
|
||||
|
||||
Performance monitoring tracks protocol distribution showing fallback rates, connection establishment time by protocol, TTFB improvements, UDP packet loss rate on HTTP/3 connections, and HTTP/3 → HTTP/2 fallback frequency. Set up alerting for abnormal fallback rates indicating network issues, increased CPU usage from QUIC overhead, or client compatibility problems.
|
||||
|
||||
**Phase 4: Advanced optimization**
|
||||
|
||||
Type-safe protocol handling uses phantom types for compile-time guarantees:
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE DataKinds, KindSignatures #-}
|
||||
|
||||
data Protocol = HTTP1 | HTTP2 | HTTP3
|
||||
|
||||
newtype Request (p :: Protocol) = Request RequestData
|
||||
|
||||
-- Only valid for HTTP2 (compile-time check)
|
||||
pushResource :: Request 'HTTP2 -> PushPromise -> IO ()
|
||||
```
|
||||
|
||||
Resource management uses ResourceT for automatic cleanup:
|
||||
|
||||
```haskell
|
||||
import Control.Monad.Trans.Resource
|
||||
|
||||
app :: Application
|
||||
app req respond = runResourceT $ do
|
||||
(releaseKey, handle) <- allocate
|
||||
(openFile "data.txt" ReadMode)
|
||||
hClose
|
||||
content <- liftIO $ hGetContents handle
|
||||
liftIO $ respond $ responseLBS status200 [] content
|
||||
```
|
||||
|
||||
Custom priority handling implements application-specific prioritization by analyzing request patterns, identifying critical resources, and using HTTP/2 priority frames for time-sensitive content.
|
||||
|
||||
## Critical recommendations for production deployment
|
||||
|
||||
**Deploy HTTP/2 immediately for all public-facing Haskell web applications.** The benefits are proven (14-30% improvement), implementation is mature (Warp 3.1+ since 2015), and browser support is universal (97%). Configuration requires minimal changes (just TLS with warp-tls), and performance matches nginx despite being Haskell code. The `http2` library's 10+ years and 70+ releases demonstrate production readiness.
|
||||
|
||||
**Deploy HTTP/3 through reverse proxies for mobile-heavy or global audiences.** The protocol delivers significant improvements on problematic networks (25-50%+), particularly valuable for developing markets and mobile users. Nginx or Caddy provide mature implementations while Haskell applications continue using proven Warp interfaces. Direct Haskell HTTP/3 via warp-quic should wait until version 1.0+ for production use, though experimentation is encouraged for learning and feedback.
|
||||
|
||||
**Avoid server push entirely**, as it's deprecated and removed from major browsers. Use preload hints (`<link rel="preload">`) or HTTP 103 Early Hints instead. Testing server push capabilities in the `http2` library wastes development time on abandoned technology.
|
||||
|
||||
**Prioritize testing across diverse network conditions.** HTTP/2 and HTTP/3 benefits vary dramatically by network quality, with the largest gains on the slowest connections. Test on cellular networks (high latency, packet loss), from international locations (high RTT), and through enterprise networks (potential UDP blocking) to ensure fallback mechanisms work correctly.
|
||||
|
||||
**Monitor protocol distribution and fallback rates.** Unexpected fallback patterns indicate network issues, client compatibility problems, or configuration errors. Track CPU usage with HTTP/3, as QUIC's user-space implementation requires more processing than kernel TCP. Budget for this overhead when scaling.
|
||||
|
||||
The Haskell HTTP implementation ecosystem provides production-ready building blocks for modern web applications, with HTTP/2 support matching industry leaders and HTTP/3 capabilities emerging through active development. The unified architecture across `http2`, `quic`, and `http3` libraries ensures smooth adoption paths while maintaining the type safety and composability that make Haskell valuable for production systems.
|
||||
|
|
@ -1,964 +0,0 @@
|
|||
# Advanced Load Balancing Implementation Guide for Haskell Reverse Proxies
|
||||
|
||||
**Production-grade load balancing architecture for high-performance Haskell proxies targeting 100k+ req/s, featuring STM-based connection tracking, async health checking, and composable algorithms.**
|
||||
|
||||
Load balancing in Haskell reverse proxies requires careful orchestration of concurrent state management, efficient algorithm selection, and robust failure detection. For the Ᾰenebris project milestone, this guide synthesizes battle-tested patterns from production systems like Keter, Mighty, and modern libraries to deliver type-safe, composable implementations that leverage Haskell's concurrency primitives. The architecture balances functional purity with performance pragmatism, using **IORef for hot paths** and **STM for complex transactions** while maintaining sub-microsecond selection latency.
|
||||
|
||||
## Algorithm implementations optimized for Haskell
|
||||
|
||||
The foundation of any load balancer is its selection algorithm. **Round-robin with IORef achieves ~9.7ns read/write operations**, making it ideal for high-throughput scenarios. The critical design choice is between IORef (minimal overhead) and TVar (composability), with each serving distinct architectural needs.
|
||||
|
||||
### Round-robin: IORef-based implementation
|
||||
|
||||
For pure round-robin without complex state coordination, **IORef with `atomicModifyIORef'` provides optimal performance**:
|
||||
|
||||
```haskell
|
||||
import Data.IORef
|
||||
import Data.Vector (Vector, (!))
|
||||
import qualified Data.Vector as V
|
||||
|
||||
data RoundRobinBalancer a = RoundRobinBalancer
|
||||
{ backends :: Vector a
|
||||
, counter :: IORef Int
|
||||
}
|
||||
|
||||
newRRBalancer :: [a] -> IO (RoundRobinBalancer a)
|
||||
newRRBalancer bs = RoundRobinBalancer (V.fromList bs) <$> newIORef 0
|
||||
|
||||
selectBackend :: RoundRobinBalancer a -> IO a
|
||||
selectBackend balancer = do
|
||||
let backends' = backends balancer
|
||||
len = V.length backends'
|
||||
idx <- atomicModifyIORef' (counter balancer) $ \i ->
|
||||
let next = (i + 1) `mod` len
|
||||
in (next, i)
|
||||
return $ backends' ! idx
|
||||
```
|
||||
|
||||
The strict `atomicModifyIORef'` variant is essential - the lazy version causes space leaks under high load. Vector indexing provides O(1) access, and modulo wraparound handles counter overflow safely. This pattern **scales linearly to 8+ cores** without contention issues.
|
||||
|
||||
**Alternative: Hackage's `roundRobin` package** (version 0.1.2.0) provides a pre-built solution using NonEmpty for type-level guarantees of at least one backend:
|
||||
|
||||
```haskell
|
||||
import Data.RoundRobin
|
||||
|
||||
rr <- newRoundRobin (backend1 :| [backend2, backend3])
|
||||
backend <- select rr -- Thread-safe selection
|
||||
```
|
||||
|
||||
### Least connections: Heap-based and STM approaches
|
||||
|
||||
Least connections requires tracking active connection counts per backend. **Two proven patterns emerge**: heap-based (Rob Pike inspired) and direct STM comparison.
|
||||
|
||||
**Heap-based implementation** (from wagdav/load-balancer):
|
||||
|
||||
```haskell
|
||||
import Data.Heap (MinPrioHeap)
|
||||
import qualified Data.Heap as DH
|
||||
import Control.Concurrent.STM
|
||||
|
||||
type Pool a = MinPrioHeap Int (Worker a)
|
||||
|
||||
data Worker a = Worker Int (TChan (Request a))
|
||||
|
||||
-- Dispatch to least-loaded worker
|
||||
dispatch :: Pool a -> Request a -> IO (Pool a)
|
||||
dispatch pool request = do
|
||||
let ((priority, worker), pool') = fromJust $ view pool
|
||||
schedule worker request
|
||||
return $ insert (priority + 1, worker) pool'
|
||||
|
||||
-- Mark completion and decrement
|
||||
completed :: Pool a -> Worker a -> Pool a
|
||||
completed pool worker =
|
||||
let (matchingWorkers, pool') = partition (\item -> snd item == worker) pool
|
||||
[(priority, w)] = toList matchingWorkers
|
||||
in insert (priority - 1, w) pool'
|
||||
```
|
||||
|
||||
The heap automatically maintains the least-loaded worker at the root with **O(log n) insertion and extraction**. Workers report completion asynchronously via TChan, enabling loose coupling and independent failure handling.
|
||||
|
||||
**Direct STM comparison** for simpler architectures:
|
||||
|
||||
```haskell
|
||||
data Backend = Backend
|
||||
{ backendHost :: String
|
||||
, backendPort :: Int
|
||||
, activeConnections :: TVar Int
|
||||
}
|
||||
|
||||
trackConnection :: Backend -> IO a -> IO a
|
||||
trackConnection backend action =
|
||||
bracket_
|
||||
(atomically $ modifyTVar' (activeConnections backend) (+1))
|
||||
(atomically $ modifyTVar' (activeConnections backend) (subtract 1))
|
||||
action
|
||||
|
||||
selectLeastConnections :: [Backend] -> STM Backend
|
||||
selectLeastConnections backends = do
|
||||
conns <- mapM (readTVar . activeConnections) backends
|
||||
let minConns = minimum conns
|
||||
idx = fromJust $ findIndex (== minConns) conns
|
||||
return $ backends !! idx
|
||||
```
|
||||
|
||||
This pattern **composes naturally with health checks** - the STM transaction can atomically read both connection counts and health status. The tradeoff is performance: STM transactions have O(n) lookup time where n equals TVars accessed, roughly 2x slower than IORef for simple operations but vastly superior for composed logic.
|
||||
|
||||
**Hackage's `load-balancing` package** (version 1.0.1.1) provides production-tested least-connections with round-robin tie-breaking:
|
||||
|
||||
```haskell
|
||||
import Control.Concurrent.LoadDistribution
|
||||
|
||||
lb <- evenlyDistributed (return $ Set.fromList backends)
|
||||
withResource lb $ \maybeBackend ->
|
||||
case maybeBackend of
|
||||
Just backend -> proxyRequest backend
|
||||
Nothing -> handleNoBackends
|
||||
```
|
||||
|
||||
### Weighted distribution: Smooth weighted round-robin
|
||||
|
||||
The **nginx smooth weighted round-robin algorithm produces optimal distribution patterns**, avoiding bursts of identical backend selection. For weights {5, 1, 1}, it generates {a, a, b, a, c, a, a} instead of the naive {c, b, a, a, a, a, a}.
|
||||
|
||||
**Algorithm mechanics**: On each selection, increase each backend's `current_weight` by its `weight`, select the backend with maximum `current_weight`, then reduce the selected backend's weight by the total weight sum.
|
||||
|
||||
```haskell
|
||||
data WeightedBackend a = WeightedBackend
|
||||
{ backend :: a
|
||||
, weight :: Int
|
||||
, currentWeight :: TVar Int
|
||||
}
|
||||
|
||||
smoothWeightedSelect :: [WeightedBackend a] -> IO a
|
||||
smoothWeightedSelect backends = atomically $ do
|
||||
-- Increase all current weights by their base weights
|
||||
forM_ backends $ \wb ->
|
||||
modifyTVar' (currentWeight wb) (+ weight wb)
|
||||
|
||||
-- Find backend with maximum current weight
|
||||
weights <- mapM (readTVar . currentWeight) backends
|
||||
let maxWeight = maximum weights
|
||||
selected = backends !! fromJust (findIndex (== maxWeight) weights)
|
||||
|
||||
-- Reduce selected backend's current weight by total
|
||||
let totalWeight = sum (map weight backends)
|
||||
modifyTVar' (currentWeight selected) (subtract totalWeight)
|
||||
|
||||
return $ backend selected
|
||||
```
|
||||
|
||||
This **STM-based implementation guarantees atomicity** across multiple backend updates. For weights {5, 1, 1}, the execution trace shows smooth distribution:
|
||||
|
||||
```
|
||||
Initial: a=0 b=0 c=0
|
||||
Step 1: a=5 b=1 c=1 → select a → a=-2
|
||||
Step 2: a=3 b=2 c=2 → select a → a=-4
|
||||
Step 3: a=1 b=3 c=3 → select b → b=-4
|
||||
Step 4: a=6 b=-3 c=4 → select a → a=-1
|
||||
```
|
||||
|
||||
**No existing Haskell library implements smooth WRR** - this is a critical implementation gap. The nginx algorithm (commit 52327e0) provides the reference implementation, and the pattern above is production-ready.
|
||||
|
||||
### Performance characteristics and selection criteria
|
||||
|
||||
| Algorithm | Complexity | Concurrency Primitive | Use Case | Throughput |
|
||||
|-----------|------------|----------------------|----------|------------|
|
||||
| Round-robin (IORef) | O(1) | IORef | Simple equal distribution | 100k+ req/s |
|
||||
| Round-robin (TVar) | O(1) | TVar | Composable with health checks | 80k+ req/s |
|
||||
| Least connections (heap) | O(log n) | STM + TChan | Dynamic load awareness | 50k+ req/s |
|
||||
| Least connections (direct) | O(n) | STM | Simple setups, few backends | 40k+ req/s |
|
||||
| Smooth WRR | O(n) | STM | Weighted backends, quality distribution | 60k+ req/s |
|
||||
|
||||
**Decision matrix**: Use IORef round-robin for maximum throughput with equal backends. Use STM-based smooth WRR when backend capacities differ. Use heap-based least connections when request processing time varies significantly (e.g., database queries vs static files).
|
||||
|
||||
## STM patterns for connection tracking
|
||||
|
||||
Software Transactional Memory enables **composable atomic updates across multiple shared variables**, critical for coordinating health checks, connection counts, and metrics. Understanding STM's performance characteristics prevents common pitfalls.
|
||||
|
||||
### TVar architecture for shared state
|
||||
|
||||
**TVar provides transactional guarantees** but with performance tradeoffs. Each `readTVar` or `writeTVar` adds an entry to the transaction log with O(n) lookup cost. The key insight: **keep transactions small and minimize TVars touched per transaction**.
|
||||
|
||||
```haskell
|
||||
data BackendState = BackendState
|
||||
{ bsBackends :: TVar (Map BackendId Backend)
|
||||
, bsMetrics :: TVar Metrics
|
||||
, bsHealthStatus :: TVar (Map BackendId HealthStatus)
|
||||
}
|
||||
|
||||
-- BAD: Touches many TVars in single transaction
|
||||
countAllConnections :: [Backend] -> STM Int
|
||||
countAllConnections backends =
|
||||
sum <$> mapM (readTVar . activeConns) backends
|
||||
|
||||
-- GOOD: Maintain aggregate counter
|
||||
data BackendPool = BackendPool
|
||||
{ totalConnections :: TVar Int
|
||||
, backends :: [Backend]
|
||||
}
|
||||
|
||||
-- Update both atomically
|
||||
updateConnections :: BackendPool -> Backend -> IO ()
|
||||
updateConnections pool backend = atomically $ do
|
||||
modifyTVar' (totalConnections pool) (+1)
|
||||
modifyTVar' (activeConns backend) (+1)
|
||||
```
|
||||
|
||||
**TMVar vs TVar choice**: TVar holds a value always; TMVar can be empty. Use TMVar for **synchronization and signaling** (producer/consumer), TVar for **shared state**. TMVar is just `TVar (Maybe a)` with blocking operations - use the simpler primitive when blocking isn't needed.
|
||||
|
||||
### Avoiding contention through striping
|
||||
|
||||
**Striped pools reduce contention** by partitioning resources across multiple TVars. The `resource-pool` package (used by Yesod) implements this pattern:
|
||||
|
||||
```haskell
|
||||
data Pool a = Pool
|
||||
{ localPools :: SmallArray (LocalPool a)
|
||||
, reaperRef :: IORef ()
|
||||
}
|
||||
|
||||
data LocalPool a = LocalPool
|
||||
{ localPool :: TVar (Stripe a)
|
||||
}
|
||||
|
||||
data Stripe a = Stripe
|
||||
{ available :: Int -- Count of available resources
|
||||
, queue :: Queue a -- Available resources
|
||||
, waiting :: Queue (TMVar (Maybe a)) -- Waiting threads
|
||||
}
|
||||
```
|
||||
|
||||
Each stripe operates independently, **reducing transaction conflicts**. Configure stripe count to match CPU cores (default uses `getNumCapabilities`). This pattern **scales STM performance to 40+ cores** before plateauing.
|
||||
|
||||
### Connection pool implementation with STM
|
||||
|
||||
Production-grade connection tracking combines **bracket for resource safety with STM for atomic updates**:
|
||||
|
||||
```haskell
|
||||
import Control.Concurrent.STM
|
||||
import Control.Exception (bracket_)
|
||||
|
||||
data Backend = Backend
|
||||
{ backendId :: Int
|
||||
, connections :: TVar Int
|
||||
, maxConnections :: Int
|
||||
, healthy :: TVar Bool
|
||||
}
|
||||
|
||||
-- Acquire connection with capacity checking
|
||||
acquireConnection :: Backend -> STM ()
|
||||
acquireConnection backend = do
|
||||
current <- readTVar (connections backend)
|
||||
isHealthy <- readTVar (healthy backend)
|
||||
when (current >= maxConnections backend) retry
|
||||
when (not isHealthy) retry
|
||||
writeTVar (connections backend) (current + 1)
|
||||
|
||||
-- Release connection
|
||||
releaseConnection :: Backend -> STM ()
|
||||
releaseConnection backend =
|
||||
modifyTVar' (connections backend) (subtract 1)
|
||||
|
||||
-- Safe usage pattern
|
||||
withConnection :: Backend -> IO a -> IO a
|
||||
withConnection backend action =
|
||||
bracket_
|
||||
(atomically $ acquireConnection backend)
|
||||
(atomically $ releaseConnection backend)
|
||||
action
|
||||
```
|
||||
|
||||
The `retry` primitive is STM's killer feature - **threads automatically block until the transaction can succeed**. When connections become available or health status changes, waiting threads wake and retry. No manual condition variables or polling needed.
|
||||
|
||||
### Performance optimization strategies
|
||||
|
||||
**Key findings from production systems**:
|
||||
|
||||
1. **Keep transactions small**: Long transactions are vulnerable to starvation. Short transactions repeatedly abort long ones under contention.
|
||||
|
||||
2. **Move pure computation outside transactions**:
|
||||
```haskell
|
||||
-- BAD: Expensive computation inside transaction
|
||||
badPattern = atomically $ do
|
||||
val <- expensiveComputation -- Recomputed on every retry!
|
||||
tvar <- readTVar someTVar
|
||||
|
||||
-- GOOD: Pure computation outside
|
||||
goodPattern = do
|
||||
let val = expensiveComputation
|
||||
atomically $ do
|
||||
tvar <- readTVar someTVar
|
||||
writeTVar someTVar (combine val tvar)
|
||||
```
|
||||
|
||||
3. **Use IORef for simple counters**: For single-variable updates without composition needs, **IORef is 2-3x faster**:
|
||||
- IORef read/write: ~9.7ns
|
||||
- MVar operations: ~15ns
|
||||
- TVar operations: ~20ns (single variable)
|
||||
|
||||
4. **Batch updates when possible**: Instead of N separate transactions, combine related updates:
|
||||
```haskell
|
||||
-- Update multiple backend states atomically
|
||||
updateHealthChecks :: [Backend] -> [(Backend, Bool)] -> STM ()
|
||||
updateHealthChecks backends results =
|
||||
forM_ results $ \(backend, isHealthy) ->
|
||||
writeTVar (healthy backend) isHealthy
|
||||
```
|
||||
|
||||
**GC pressure consideration**: Large pinned arrays (>409 bytes) cause GHC to take a global lock, becoming a bottleneck beyond 16 cores. Pool and reuse buffers in high-performance scenarios.
|
||||
|
||||
## Async health checking with circuit breakers
|
||||
|
||||
Health checking separates the **control plane** (detecting failures) from the **data plane** (routing requests). Async patterns using Control.Concurrent.Async enable **non-blocking health probes** that run independently of request handling.
|
||||
|
||||
### HTTP health check implementation
|
||||
|
||||
Use `http-client` with connection pooling for efficient health checks:
|
||||
|
||||
```haskell
|
||||
import Network.HTTP.Client
|
||||
import Network.HTTP.Client.TLS
|
||||
import System.Timeout
|
||||
import Control.Concurrent.Async
|
||||
|
||||
data HealthCheckConfig = HealthCheckConfig
|
||||
{ hcInterval :: Int -- Seconds between checks
|
||||
, hcTimeout :: Int -- Request timeout (seconds)
|
||||
, hcEndpoint :: String -- Health endpoint path
|
||||
, hcMaxFailures :: Int -- Failures before marking unhealthy
|
||||
, hcRecoveryAttempts :: Int -- Successes before marking healthy
|
||||
}
|
||||
|
||||
defaultConfig :: HealthCheckConfig
|
||||
defaultConfig = HealthCheckConfig
|
||||
{ hcInterval = 10
|
||||
, hcTimeout = 2
|
||||
, hcEndpoint = "/health"
|
||||
, hcMaxFailures = 3
|
||||
, hcRecoveryAttempts = 2
|
||||
}
|
||||
|
||||
performHealthCheck :: Manager -> HealthCheckConfig -> Backend -> IO Bool
|
||||
performHealthCheck manager config backend = do
|
||||
let url = backendUrl backend ++ hcEndpoint config
|
||||
result <- timeout (hcTimeout config * 1000000) $ do
|
||||
req <- parseRequest url
|
||||
response <- httpLbs req manager
|
||||
return $ statusCode (responseStatus response) == 200
|
||||
|
||||
return $ fromMaybe False result
|
||||
```
|
||||
|
||||
**Key optimizations**: Share a single `Manager` across all health checks to leverage connection pooling. Set appropriate `managerConnCount` (default 2 per route is too low for production - use 100+).
|
||||
|
||||
### Periodic scheduling with async
|
||||
|
||||
**Control.Concurrent.Async provides resource-safe concurrent operations**. The `withAsync` combinator automatically cancels threads when they leave scope:
|
||||
|
||||
```haskell
|
||||
import Control.Concurrent (threadDelay)
|
||||
import Control.Concurrent.Async
|
||||
import Control.Monad (forever)
|
||||
|
||||
healthCheckLoop :: Manager -> HealthCheckConfig -> [Backend] -> IO ()
|
||||
healthCheckLoop manager config backends = forever $ do
|
||||
-- Check all backends concurrently
|
||||
results <- mapConcurrently
|
||||
(performHealthCheck manager config)
|
||||
backends
|
||||
|
||||
-- Update backend states
|
||||
zipWithM_ (updateBackendState config) backends results
|
||||
|
||||
-- Wait for next interval
|
||||
threadDelay (hcInterval config * 1000000)
|
||||
|
||||
-- Start health checker (automatically cleaned up)
|
||||
startHealthChecker :: HealthCheckConfig -> [Backend] -> IO (Async ())
|
||||
startHealthChecker config backends = do
|
||||
manager <- newTlsManager
|
||||
async $ healthCheckLoop manager config backends
|
||||
```
|
||||
|
||||
**Concurrency patterns**: Use `mapConcurrently` to check all backends in parallel, reducing total check time. Use `race` to implement timeout-based failure detection. Use `link` to propagate exceptions from health checker to main thread.
|
||||
|
||||
**Alternative: async-timer package** provides built-in periodic scheduling:
|
||||
|
||||
```haskell
|
||||
import Control.Concurrent.Async.Timer
|
||||
|
||||
let timerConf = setInterval 10000 $ -- 10 seconds
|
||||
setInitDelay 0 defaultConf
|
||||
|
||||
withAsyncTimer timerConf $ \timer -> do
|
||||
forever $ do
|
||||
timerWait timer
|
||||
checkAndUpdateBackends
|
||||
```
|
||||
|
||||
### State transitions with failure thresholds
|
||||
|
||||
Implement **hysteresis** to prevent flapping - require multiple consecutive failures before marking unhealthy, multiple successes before marking healthy:
|
||||
|
||||
```haskell
|
||||
data BackendState = Healthy | Unhealthy | Recovering
|
||||
deriving (Eq, Show)
|
||||
|
||||
data Backend = Backend
|
||||
{ backendState :: TVar BackendState
|
||||
, backendFailures :: TVar Int
|
||||
, backendSuccesses :: TVar Int
|
||||
}
|
||||
|
||||
updateBackendState :: HealthCheckConfig -> Backend -> Bool -> IO ()
|
||||
updateBackendState config backend healthy = atomically $ do
|
||||
state <- readTVar (backendState backend)
|
||||
failures <- readTVar (backendFailures backend)
|
||||
successes <- readTVar (backendSuccesses backend)
|
||||
|
||||
case (state, healthy) of
|
||||
(Healthy, False) -> do
|
||||
let newFailures = failures + 1
|
||||
writeTVar (backendFailures backend) newFailures
|
||||
when (newFailures >= hcMaxFailures config) $ do
|
||||
writeTVar (backendState backend) Unhealthy
|
||||
writeTVar (backendFailures backend) 0
|
||||
|
||||
(Unhealthy, True) -> do
|
||||
writeTVar (backendState backend) Recovering
|
||||
writeTVar (backendSuccesses backend) 1
|
||||
|
||||
(Recovering, True) -> do
|
||||
let newSuccesses = successes + 1
|
||||
writeTVar (backendSuccesses backend) newSuccesses
|
||||
when (newSuccesses >= hcRecoveryAttempts config) $ do
|
||||
writeTVar (backendState backend) Healthy
|
||||
writeTVar (backendSuccesses backend) 0
|
||||
|
||||
(Recovering, False) -> do
|
||||
writeTVar (backendState backend) Unhealthy
|
||||
writeTVar (backendSuccesses backend) 0
|
||||
|
||||
_ -> return ()
|
||||
```
|
||||
|
||||
This state machine **prevents transient failures from cascading**. A backend must fail `hcMaxFailures` consecutive checks (e.g., 3) before removal, and succeed `hcRecoveryAttempts` consecutive checks (e.g., 2) before returning to rotation.
|
||||
|
||||
### Circuit breaker integration
|
||||
|
||||
**Circuit breakers prevent cascading failures** by failing fast when a backend is degraded. The `circuit-breaker` package (Hackage) provides type-level configuration:
|
||||
|
||||
```haskell
|
||||
import System.CircuitBreaker
|
||||
|
||||
-- Define circuit breaker at type level
|
||||
-- 1000ms = error expiry time, 4 = threshold
|
||||
testBreaker :: CircuitBreaker "Test" 1000 4
|
||||
testBreaker = undefined
|
||||
|
||||
proxyWithCircuitBreaker :: Backend -> Request -> IO Response
|
||||
proxyWithCircuitBreaker backend req = do
|
||||
cbConf <- initialBreakerState
|
||||
|
||||
result <- flip runReaderT cbConf $
|
||||
withBreaker testBreaker $ liftIO $ forwardRequest backend req
|
||||
|
||||
case result of
|
||||
Left (CircuitBreakerClosed msg) ->
|
||||
-- Circuit open, return cached response or error
|
||||
return $ errorResponse 503 "Service Temporarily Unavailable"
|
||||
Right response ->
|
||||
return response
|
||||
```
|
||||
|
||||
Circuit breaker states: **Active** (closed, requests pass), **Testing** (half-open, testing recovery), **Waiting** (open, blocking requests). When error threshold (4) is reached within the time window (1000ms), the circuit opens and blocks requests until the window expires.
|
||||
|
||||
**Production pattern**: Combine circuit breakers with health checks for defense in depth:
|
||||
|
||||
```haskell
|
||||
selectBackendWithCircuitBreaker :: BackendPool -> IO (Maybe Backend)
|
||||
selectBackendWithCircuitBreaker pool = do
|
||||
healthy <- getHealthyBackends pool
|
||||
available <- filterM isCircuitBreakerClosed healthy
|
||||
case available of
|
||||
[] -> return Nothing
|
||||
backends -> Just <$> selectFromPool backends
|
||||
```
|
||||
|
||||
### Exponential backoff for failed backends
|
||||
|
||||
**Implement jittered exponential backoff** to avoid thundering herd when backends recover:
|
||||
|
||||
```haskell
|
||||
import System.Random (randomRIO)
|
||||
|
||||
data BackoffConfig = BackoffConfig
|
||||
{ initialDelay :: Int -- microseconds
|
||||
, maxDelay :: Int
|
||||
, maxRetries :: Int
|
||||
}
|
||||
|
||||
exponentialBackoffWithJitter :: BackoffConfig -> IO a -> IO (Maybe a)
|
||||
exponentialBackoffWithJitter config action = go 0 (initialDelay config)
|
||||
where
|
||||
go retries delay
|
||||
| retries >= maxRetries config = return Nothing
|
||||
| otherwise = do
|
||||
result <- try action
|
||||
case result of
|
||||
Right val -> return (Just val)
|
||||
Left (_ :: SomeException) -> do
|
||||
-- Add jitter: random value up to 50% of delay
|
||||
jitter <- randomRIO (0, delay `div` 2)
|
||||
threadDelay (delay + jitter)
|
||||
let nextDelay = min (delay * 2) (maxDelay config)
|
||||
go (retries + 1) nextDelay
|
||||
```
|
||||
|
||||
**Jitter is critical** - without it, all failed requests retry simultaneously, creating load spikes. With jitter, retries spread over time.
|
||||
|
||||
**Alternative: Use the `retry` package** (Hackage, widely adopted):
|
||||
|
||||
```haskell
|
||||
import Control.Retry
|
||||
|
||||
recovering
|
||||
(exponentialBackoff 50000 <> limitRetries 5)
|
||||
[const $ Handler $ \e -> return (isRetryable e)]
|
||||
(\_ -> performHealthCheck backend)
|
||||
```
|
||||
|
||||
The `retry` package uses **Monoid composition** for retry policies - combine `exponentialBackoff`, `limitRetries`, and `capDelay` to build complex policies declaratively.
|
||||
|
||||
## Integration with Warp and WAI
|
||||
|
||||
Warp is the **highest-performance Haskell web server**, achieving throughput comparable to nginx (~50,000-80,000 req/s single-threaded, scaling linearly to 8+ workers). Integration with load balancing leverages WAI middleware and reverse proxy libraries.
|
||||
|
||||
### Using http-reverse-proxy
|
||||
|
||||
**Two approaches**: raw socket (minimal overhead) and WAI-based (full feature set). For load balancing, **use the WAI approach** for request modification and middleware composition:
|
||||
|
||||
```haskell
|
||||
import Network.HTTP.Client.TLS
|
||||
import Network.HTTP.ReverseProxy
|
||||
import Network.Wai
|
||||
import Network.Wai.Handler.Warp (run)
|
||||
import Control.Concurrent.STM
|
||||
|
||||
data ProxyConfig = ProxyConfig
|
||||
{ pcBackends :: TVar [Backend]
|
||||
, pcManager :: Manager
|
||||
, pcBalancer :: LoadBalancer
|
||||
}
|
||||
|
||||
proxyApp :: ProxyConfig -> Application
|
||||
proxyApp config req respond = do
|
||||
mBackend <- selectHealthyBackend (pcBalancer config)
|
||||
case mBackend of
|
||||
Nothing ->
|
||||
respond $ responseLBS status503 [] "No healthy backends available"
|
||||
Just backend ->
|
||||
waiProxyTo
|
||||
(\_ -> return $ WPRProxyDest (ProxyDest
|
||||
(backendHost backend)
|
||||
(backendPort backend)))
|
||||
defaultOnExc
|
||||
(pcManager config)
|
||||
req
|
||||
respond
|
||||
```
|
||||
|
||||
**Key functions**:
|
||||
- `waiProxyTo`: Full request/response control
|
||||
- `WPRProxyDest`: Route to specific backend
|
||||
- `WPRModifiedRequest`: Modify request before proxying
|
||||
- `defaultOnExc`: Exception handler
|
||||
|
||||
**WebSocket support**: Use `waiProxyToSettings` with `wpsUpgradeToRaw = True` for WebSocket tunneling.
|
||||
|
||||
### Middleware for request routing
|
||||
|
||||
Middleware composes via function application. **Build a middleware stack** for logging, authentication, and routing:
|
||||
|
||||
```haskell
|
||||
import Network.Wai (Middleware, mapResponseHeaders)
|
||||
|
||||
-- Add load balancer info header
|
||||
addBackendHeader :: Backend -> Middleware
|
||||
addBackendHeader backend app req respond =
|
||||
app req $ respond . mapResponseHeaders
|
||||
((hBackend, encodeUtf8 $ backendId backend) :)
|
||||
|
||||
-- Connection tracking middleware
|
||||
trackingMiddleware :: ServerMetrics -> Middleware
|
||||
trackingMiddleware metrics app req respond = do
|
||||
atomically $ modifyTVar' (activeConnections metrics) (+1)
|
||||
atomically $ modifyTVar' (totalRequests metrics) (+1)
|
||||
|
||||
let respond' res = do
|
||||
atomically $ modifyTVar' (activeConnections metrics) (subtract 1)
|
||||
respond res
|
||||
|
||||
app req respond' `onException`
|
||||
atomically (modifyTVar' (errorCount metrics) (+1))
|
||||
|
||||
-- Compose middleware
|
||||
main = do
|
||||
config <- initProxyConfig
|
||||
let app = trackingMiddleware metrics
|
||||
$ proxyApp config
|
||||
run 8000 app
|
||||
```
|
||||
|
||||
### Performance optimization for 100k+ req/s
|
||||
|
||||
**Warp architecture** uses lightweight green threads (100,000+ possible) with one thread per connection. The GHC I/O manager provides non-blocking I/O via epoll/kqueue. Key optimizations:
|
||||
|
||||
1. **Minimize system calls**: Warp uses only `recv()`, `send()`, and `sendfile()`. Eliminate `open()`/`stat()`/`close()` via file descriptor caching.
|
||||
|
||||
2. **Specialize hot paths**: Use custom HTTP response composer instead of generic Builder. Cache date strings (regenerate once per second).
|
||||
|
||||
3. **Avoid locks**: Use lock-free atomic operations (`atomicModifyIORef`) instead of MVar spin locks. Warp's timeout manager uses double-IORef for lock-free status updates.
|
||||
|
||||
4. **Proper data structures**: ByteString for buffers (enables zero-copy splicing), Vector for backend lists (O(1) indexing).
|
||||
|
||||
**Compile-time optimizations**:
|
||||
|
||||
```cabal
|
||||
ghc-options: -Wall -O2 -threaded
|
||||
-rtsopts -with-rtsopts=-N
|
||||
-fspec-constr -fspecialise
|
||||
-funbox-strict-fields
|
||||
```
|
||||
|
||||
**Runtime settings**:
|
||||
|
||||
```bash
|
||||
./proxy +RTS -N -A64m -I0 -qg
|
||||
```
|
||||
|
||||
- `-N`: Use all CPU cores
|
||||
- `-A64m`: Large allocation area (fewer GCs)
|
||||
- `-I0`: Disable idle GC
|
||||
- `-qg`: Parallel GC
|
||||
|
||||
**Benchmarking**: Historical data shows **Mighty (Warp-based) achieved 50,000 req/s single-threaded**, scaling linearly to 8 workers. Modern Warp (2024-2025) includes further optimizations. Use `weighttp` or `wrk` for multi-threaded load testing.
|
||||
|
||||
### HTTP client connection pooling
|
||||
|
||||
**Manager configuration is critical** for backend connections:
|
||||
|
||||
```haskell
|
||||
import Network.HTTP.Client
|
||||
import Network.HTTP.Client.TLS
|
||||
|
||||
manager <- newManager $ defaultManagerSettings
|
||||
{ managerConnCount = 1000 -- Max connections per backend
|
||||
, managerIdleConnectionCount = 500 -- Idle connections to keep
|
||||
, managerResponseTimeout = responseTimeoutMicro 60000000
|
||||
}
|
||||
```
|
||||
|
||||
**Connection pooling impact**: Without pooling, each request establishes a new TCP connection (expensive handshake, especially for TLS). With pooling, **10-100x faster** for repeated requests to the same backend.
|
||||
|
||||
**Pool configuration guidelines**:
|
||||
- `managerConnCount`: Set to 500-5000 per backend based on capacity
|
||||
- `managerIdleConnectionCount`: Keep 50-80% of max connections idle
|
||||
- Share single Manager across application (thread-safe)
|
||||
|
||||
## Production implementation blueprint
|
||||
|
||||
Combining all patterns into a **production-ready architecture** for Milestone 1.3:
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE RecordWildCards #-}
|
||||
|
||||
module LoadBalancer where
|
||||
|
||||
import Control.Concurrent.Async
|
||||
import Control.Concurrent.STM
|
||||
import Network.HTTP.Client.TLS
|
||||
import Network.HTTP.ReverseProxy
|
||||
import Network.Wai
|
||||
import Network.Wai.Handler.Warp
|
||||
import qualified Data.Vector as V
|
||||
|
||||
-- Core data types
|
||||
data Backend = Backend
|
||||
{ backendId :: Int
|
||||
, backendHost :: ByteString
|
||||
, backendPort :: Int
|
||||
, backendWeight :: Int
|
||||
, currentWeight :: TVar Int
|
||||
, activeConns :: TVar Int
|
||||
, state :: TVar BackendState
|
||||
, failures :: TVar Int
|
||||
}
|
||||
|
||||
data BackendState = Healthy | Unhealthy | Recovering
|
||||
|
||||
data Strategy = RoundRobin | LeastConnections | SmoothWeightedRR
|
||||
|
||||
data ProxyConfig = ProxyConfig
|
||||
{ backends :: V.Vector Backend
|
||||
, strategy :: Strategy
|
||||
, healthChecker :: Async ()
|
||||
, httpManager :: Manager
|
||||
, rrCounter :: IORef Int
|
||||
}
|
||||
|
||||
-- Backend selection dispatcher
|
||||
selectBackend :: ProxyConfig -> IO (Maybe Backend)
|
||||
selectBackend config =
|
||||
case strategy config of
|
||||
RoundRobin -> selectRoundRobin config
|
||||
LeastConnections -> selectLeastConnections config
|
||||
SmoothWeightedRR -> selectWeightedRR config
|
||||
|
||||
-- Round-robin implementation
|
||||
selectRoundRobin :: ProxyConfig -> IO (Maybe Backend)
|
||||
selectRoundRobin config = do
|
||||
let backends' = backends config
|
||||
len = V.length backends'
|
||||
idx <- atomicModifyIORef' (rrCounter config) $ \i ->
|
||||
((i + 1) `mod` len, i)
|
||||
|
||||
-- Find next healthy backend
|
||||
findHealthy backends' idx len
|
||||
where
|
||||
findHealthy backends' start remaining
|
||||
| remaining <= 0 = return Nothing
|
||||
| otherwise = do
|
||||
let backend = backends' V.! start
|
||||
isHealthy <- atomically $ (== Healthy) <$> readTVar (state backend)
|
||||
if isHealthy
|
||||
then return (Just backend)
|
||||
else findHealthy backends'
|
||||
((start + 1) `mod` V.length backends')
|
||||
(remaining - 1)
|
||||
|
||||
-- Least connections implementation
|
||||
selectLeastConnections :: ProxyConfig -> IO (Maybe Backend)
|
||||
selectLeastConnections config = do
|
||||
let backends' = V.toList (backends config)
|
||||
healthy <- filterM isHealthy backends'
|
||||
case healthy of
|
||||
[] -> return Nothing
|
||||
bs -> do
|
||||
conns <- forM bs $ \b -> do
|
||||
count <- atomically $ readTVar (activeConns b)
|
||||
return (count, b)
|
||||
return $ Just $ snd $ minimum conns
|
||||
where
|
||||
isHealthy b = atomically $ (== Healthy) <$> readTVar (state b)
|
||||
|
||||
-- Smooth weighted round-robin
|
||||
selectWeightedRR :: ProxyConfig -> IO (Maybe Backend)
|
||||
selectWeightedRR config = atomically $ do
|
||||
let backends' = V.toList (backends config)
|
||||
|
||||
-- Increase current weights
|
||||
forM_ backends' $ \wb ->
|
||||
modifyTVar' (currentWeight wb) (+ backendWeight wb)
|
||||
|
||||
-- Select backend with max current weight
|
||||
weights <- mapM (readTVar . currentWeight) backends'
|
||||
let maxWeight = maximum weights
|
||||
selected = backends' !! fromJust (findIndex (== maxWeight) weights)
|
||||
|
||||
-- Check health
|
||||
isHealthy <- (== Healthy) <$> readTVar (state selected)
|
||||
guard isHealthy
|
||||
|
||||
-- Reduce selected backend's current weight
|
||||
let totalWeight = sum (map backendWeight backends')
|
||||
modifyTVar' (currentWeight selected) (subtract totalWeight)
|
||||
|
||||
return selected
|
||||
|
||||
-- Main proxy application
|
||||
proxyApp :: ProxyConfig -> Application
|
||||
proxyApp config req respond = do
|
||||
mBackend <- selectBackend config
|
||||
case mBackend of
|
||||
Nothing ->
|
||||
respond $ responseLBS status503 [] "No healthy backends"
|
||||
Just backend -> do
|
||||
-- Track connection
|
||||
atomically $ modifyTVar' (activeConns backend) (+1)
|
||||
|
||||
let dest = ProxyDest (backendHost backend) (backendPort backend)
|
||||
respond' res = do
|
||||
atomically $ modifyTVar' (activeConns backend) (subtract 1)
|
||||
respond res
|
||||
|
||||
waiProxyTo
|
||||
(\_ -> return $ WPRProxyDest dest)
|
||||
defaultOnExc
|
||||
(httpManager config)
|
||||
req
|
||||
respond'
|
||||
|
||||
-- Health checker (runs asynchronously)
|
||||
healthCheckLoop :: Manager -> [Backend] -> IO ()
|
||||
healthCheckLoop manager backends = forever $ do
|
||||
results <- mapConcurrently (checkHealth manager) backends
|
||||
zipWithM_ updateHealth backends results
|
||||
threadDelay 10000000 -- 10 seconds
|
||||
where
|
||||
checkHealth mgr backend = do
|
||||
let url = "http://" <> backendHost backend <> ":"
|
||||
<> show (backendPort backend) <> "/health"
|
||||
result <- timeout 2000000 $ do
|
||||
req <- parseRequest (unpack url)
|
||||
response <- httpLbs req mgr
|
||||
return $ statusCode (responseStatus response) == 200
|
||||
return $ fromMaybe False result
|
||||
|
||||
updateHealth backend healthy = atomically $ do
|
||||
currentState <- readTVar (state backend)
|
||||
failureCount <- readTVar (failures backend)
|
||||
case (currentState, healthy) of
|
||||
(Healthy, False) -> do
|
||||
let newFailures = failureCount + 1
|
||||
writeTVar (failures backend) newFailures
|
||||
when (newFailures >= 3) $ do
|
||||
writeTVar (state backend) Unhealthy
|
||||
writeTVar (failures backend) 0
|
||||
(Unhealthy, True) ->
|
||||
writeTVar (state backend) Recovering
|
||||
(Recovering, True) ->
|
||||
writeTVar (state backend) Healthy
|
||||
_ -> return ()
|
||||
|
||||
-- Initialization
|
||||
initProxyConfig :: [BackendSpec] -> Strategy -> IO ProxyConfig
|
||||
initProxyConfig specs strat = do
|
||||
backends <- V.fromList <$> mapM createBackend (zip [0..] specs)
|
||||
manager <- newTlsManager
|
||||
counter <- newIORef 0
|
||||
checker <- async $ healthCheckLoop manager (V.toList backends)
|
||||
|
||||
return ProxyConfig
|
||||
{ backends = backends
|
||||
, strategy = strat
|
||||
, healthChecker = checker
|
||||
, httpManager = manager
|
||||
, rrCounter = counter
|
||||
}
|
||||
where
|
||||
createBackend (idx, spec) = Backend idx
|
||||
<$> pure (bsHost spec)
|
||||
<*> pure (bsPort spec)
|
||||
<*> pure (bsWeight spec)
|
||||
<*> newTVarIO 0
|
||||
<*> newTVarIO 0
|
||||
<*> newTVarIO Healthy
|
||||
<*> newTVarIO 0
|
||||
|
||||
-- Main entry point
|
||||
main :: IO ()
|
||||
main = do
|
||||
let backendSpecs =
|
||||
[ BackendSpec "localhost" 8001 5
|
||||
, BackendSpec "localhost" 8002 1
|
||||
, BackendSpec "localhost" 8003 1
|
||||
]
|
||||
|
||||
config <- initProxyConfig backendSpecs SmoothWeightedRR
|
||||
|
||||
let settings = setPort 8000
|
||||
$ setTimeout 30
|
||||
$ defaultSettings
|
||||
|
||||
putStrLn "Load balancing proxy started on port 8000"
|
||||
runSettings settings (proxyApp config)
|
||||
```
|
||||
|
||||
## Library recommendations with versions
|
||||
|
||||
**Core infrastructure** (2024-2025 ecosystem):
|
||||
|
||||
- **warp** (3.3+): High-performance HTTP server - `ghc-options: -threaded`
|
||||
- **http-reverse-proxy** (0.6+): Reverse proxy primitives, WebSocket support
|
||||
- **http-client** (0.7+): HTTP client with connection pooling
|
||||
- **http-client-tls** (0.3+): TLS support via Haskell-native `tls` package
|
||||
|
||||
**Load balancing utilities**:
|
||||
|
||||
- **load-balancing** (1.0.1.1): Least-connections with round-robin tie-breaking
|
||||
- **roundRobin** (0.1.2.0): Simple round-robin selection
|
||||
- **resource-pool** (0.4+): Striped connection pooling (used by Yesod)
|
||||
|
||||
**Resilience libraries**:
|
||||
|
||||
- **retry** (0.9+): Exponential backoff and retry policies (Monoid-composable)
|
||||
- **circuit-breaker** (0.1+): Type-level circuit breakers with automatic backoff
|
||||
- **stamina** (0.2+): Modern "retries for humans" with Retry-After support
|
||||
|
||||
**Async \u0026 concurrency**:
|
||||
|
||||
- **async** (2.2+): Safe concurrent operations - use `withAsync` for automatic cleanup
|
||||
- **stm** (2.5+): Software Transactional Memory
|
||||
- **async-timer** (0.3+): Periodic timer execution
|
||||
|
||||
## Common pitfalls and solutions
|
||||
|
||||
**STM starvation**: Long transactions vulnerable to repeated aborts by short transactions. **Solution**: Keep transactions under 10 TVars, move pure computation outside `atomically`.
|
||||
|
||||
**Memory allocation bottlenecks**: GHC takes global lock for objects >409 bytes. **Solution**: Pool and reuse buffers. Configure larger allocation area (`+RTS -A64m`).
|
||||
|
||||
**Thundering herd**: All workers wake on new connection. **Solution**: Use prefork (multiple processes) instead of `-N` threading, or wait for parallel I/O manager integration.
|
||||
|
||||
**Health check storms**: All checks start simultaneously after deployment. **Solution**: Add random initial delay: `threadDelay =<< randomRIO (0, hcInterval config * 1000000)`.
|
||||
|
||||
**Circuit breaker cascades**: One slow backend causes all circuits to open. **Solution**: Implement per-backend circuit breakers with independent thresholds.
|
||||
|
||||
**Connection pool exhaustion**: Backends slow down, pool fills up. **Solution**: Set `managerIdleConnectionCount` conservatively (50-80% of max), implement connection timeout validation.
|
||||
|
||||
**WebSocket routing fails**: Standard proxy doesn't upgrade connections. **Solution**: Use `waiProxyToSettings` with `wpsUpgradeToRaw = True` for WebSocket support.
|
||||
|
||||
## Testing and validation strategies
|
||||
|
||||
**Property-based testing** with QuickCheck:
|
||||
|
||||
```haskell
|
||||
prop_roundRobinFairness :: [Backend] -> Property
|
||||
prop_roundRobinFairness backends =
|
||||
length backends > 0 ==> monadicIO $ do
|
||||
let selections = replicateM (length backends * 100) (selectBackend balancer)
|
||||
distribution <- run $ countSelections <$> selections
|
||||
assert $ all (\count -> count >= 90 && count <= 110) distribution
|
||||
```
|
||||
|
||||
**Concurrency testing** with dejafu:
|
||||
|
||||
```haskell
|
||||
import Test.DejaFu
|
||||
|
||||
testNoDeadlock :: IO ()
|
||||
testNoDeadlock = autocheck $ do
|
||||
balancer <- setup
|
||||
concurrently_
|
||||
(selectBackend balancer)
|
||||
(selectBackend balancer)
|
||||
```
|
||||
|
||||
**Load testing**: Use `wrk` for HTTP benchmarking:
|
||||
|
||||
```bash
|
||||
wrk -t12 -c400 -d30s http://localhost:8000/
|
||||
```
|
||||
|
||||
**Metrics collection**: Integrate `ekg` for real-time monitoring:
|
||||
|
||||
```haskell
|
||||
import System.Remote.Monitoring
|
||||
|
||||
main = do
|
||||
forkServer "localhost" 8081 -- Metrics dashboard
|
||||
store <- getStore
|
||||
registerGauge "active_connections" (readTVarIO activeConns) store
|
||||
```
|
||||
|
||||
This comprehensive implementation guide provides **production-ready patterns** for building a high-performance, type-safe reverse proxy in Haskell. The architecture balances functional purity with pragmatic performance optimization, leveraging Haskell's concurrency primitives to achieve 100k+ req/s throughput while maintaining composability and correctness guarantees.
|
||||
|
|
@ -1,177 +0,0 @@
|
|||
# ML-based bot detection for production reverse proxy with ultra-low latency
|
||||
|
||||
The challenge of implementing ML-based bot detection in a production reverse proxy with sub-millisecond latency requirements is achievable but demands careful architectural choices and aggressive optimization. For the Ᾰenebris project targeting <1ms added latency at 100k+ requests/second, the optimal approach combines in-process ONNX Runtime via Haskell FFI with quantized tree-based models, achieving 0.35ms inference time on cache misses and 0.12ms on cache hits while maintaining 90-95% detection accuracy. Network-based microservices add prohibitive overhead (0.5-2ms minimum), making them unsuitable despite their popularity. The bot detection landscape in 2024-2025 has evolved into sophisticated adversarial warfare where residential proxy networks evade 84% of traditional IP-based defenses, necessitating multi-layered detection combining TLS/HTTP/2 fingerprinting, behavioral analysis, and ensemble ML models with continuous adaptation.
|
||||
|
||||
This research synthesizes findings from Cloudflare's 46M req/sec architecture, academic papers through early 2025, and commercial bot detection vendors to provide actionable implementation guidance. Bot traffic now comprises 47-50% of internet traffic, with advanced bots showing increasing sophistication through AI-powered evasion, residential proxy networks with 30-100 million rotating IPs, and headless browser automation using tools like puppeteer-stealth that bypass most detection. Traditional defenses relying solely on IP reputation or simple fingerprinting are insufficient; success requires behavioral pattern analysis, graph neural networks for coordinated campaign detection, and adversarial training to build robust models. The core tension lies in balancing detection accuracy against false positive rates (target: <0.1%) while maintaining ultra-low latency—a constraint that eliminates many popular ML approaches including deep neural networks and standard microservice architectures.
|
||||
|
||||
## Feature engineering for sub-millisecond detection
|
||||
|
||||
Effective bot detection begins with extracting discriminative features during the TLS handshake and initial HTTP request without blocking the request path. TLS fingerprinting using JA3/JA4 provides the fastest and most reliable signal, computing MD5 hashes of ClientHello parameters (SSL version, ciphers, extensions, curves) in under 1ms with 93%+ detection rates for simple bots. The 2023 introduction of JA4 addresses Chrome's TLS extension randomization by sorting extensions before hashing, providing more stable fingerprints that work with HTTP/3/QUIC protocols. Implementation requires parsing the unencrypted ClientHello packet during handshake, concatenating specific fields with delimiters, and computing the hash—tools like Fingerproxy (Go) process 40M requests/day with this approach. Each browser and OS combination produces unique TLS stacks, making this feature highly discriminative with false positive rates under 0.001% when combined with other signals.
|
||||
|
||||
HTTP/2 fingerprinting extends TLS detection by analyzing the binary frame structure during connection initialization. The Akamai methodology from BlackHat 2017 examines SETTINGS frames (header table size, window size, concurrent streams), WINDOW_UPDATE increments, PRIORITY frame dependencies, and pseudo-header ordering to generate fingerprints like `1:65536;3:1000;4:6291456|15663105|0|m,a,s,p` for Chrome versus `1:65536;4:131072;5:16384|12517377|3:0:0:201,5:0:0:101|m,p,a,s` for Firefox. This extraction takes 1-2ms during handshake with negligible latency impact since it occurs during connection setup. Sophisticated bots using native browser libraries (Puppeteer, Playwright) produce perfect HTTP/2 fingerprints, but cross-verification with TLS fingerprints and User-Agent strings reveals inconsistencies—the "triangle of truth" approach where all three signals must align consistently.
|
||||
|
||||
Behavioral timing patterns provide the highest discriminative power but require session-state accumulation over multiple requests. Human users exhibit irregular inter-request timing (2-30+ seconds), exponential think-time distributions, and high variance in interaction patterns (coefficient of variation >0.5), while bots show mechanical precision with near-zero variance, too-fast sequences (<500ms between requests), or suspiciously uniform delays. Mouse movement analysis detects natural acceleration curves, jitter from hand tremor, and visual processing latency (100-300ms from stimulus to click), whereas bots produce smooth mechanical paths or no mouse events at all. Research shows behavioral detection achieves 87% accuracy standalone—outperforming reCAPTCHA v2 (69%) and Cloudflare Turnstile (33%)—but requires 10-50ms for ML inference on accumulated features, making it unsuitable for synchronous per-request decisions under 1ms constraints. The solution is asynchronous accumulation: track timing patterns in session state, update incrementally with each request, and perform ML inference out-of-band while applying cached risk scores in the critical path.
|
||||
|
||||
HTTP header analysis provides fast signals extractable in <0.5ms by examining header order, completeness, and unusual combinations. Legitimate browsers send headers in consistent orders—Chrome uses `Host, Connection, Upgrade-Insecure-Requests, User-Agent, Accept` while Firefox uses `Host, User-Agent, Accept, Accept-Language`—and bots often fail to replicate exact ordering or omit standard headers like Accept-Language. Missing an Accept header indicates 99% bot probability since all browsers include it. Inconsistencies between User-Agent claims and actual behavior (claiming Chrome but missing Sec-CH-UA client hints, or MSIE with modern Sec-Fetch headers) reveal spoofing attempts. The computational cost is minimal (simple string parsing and pattern matching), making header analysis ideal for the first-stage filter in a multi-tier detection pipeline. CloudFlare's detection system tracks specific header order violations with unique identifiers, treating order mismatches as high-confidence bot signals.
|
||||
|
||||
Session and cookie handling features detect stateless bots that don't maintain cookies or execute JavaScript. Setting a unique cookie on the first request and verifying its presence on subsequent requests catches simple scrapers that ignore state management—CloudFlare's cf_clearance cookie stores JavaScript challenge results for validation. JavaScript execution indicators range from simple challenge-response (compute a hash in browser, verify on server) to sophisticated browser API tests including Canvas fingerprinting, WebGL rendering checks, and navigator.webdriver property detection. Headless browser detection examines missing window.chrome objects, Selenium artifacts (_webdriver, __selenium_*), and Chrome DevTools Protocol traces. WebSocket behavior analysis verifies cookie passing during the upgrade handshake and monitors message frequency patterns. These session features add 2-5KB memory overhead per session with <0.1ms verification time, making them acceptable for the latency budget when cached efficiently.
|
||||
|
||||
Network-level TCP/IP fingerprinting using p0f-style techniques provides infrastructure-detection capabilities but requires low-level packet inspection (2-5ms overhead) with complexity that may exceed latency budgets. Analyzing TTL values, initial window sizes, TCP options ordering, and Maximum Segment Size can identify OS discrepancies (Linux server OS with Windows User-Agent = bot) and datacenter infrastructure (TTL values 63-65 typical for datacenters versus 7-20 hops for residential). While p0f v3 maintains 320+ signatures with 80-90% OS detection accuracy, implementing packet capture via eBPF or pcap in a Haskell reverse proxy adds significant engineering complexity. The recommended approach reserves TCP/IP fingerprinting for specialized scenarios or secondary validation rather than critical-path detection, focusing instead on application-layer signals that extract faster.
|
||||
|
||||
### Feature importance rankings for production deployment
|
||||
|
||||
Ranking features by discriminative power versus computational cost reveals clear priorities for sub-millisecond systems. The highest-value quick-extraction features are TLS fingerprinting (HIGH discrimination, <1ms), HTTP/2 fingerprinting (HIGH discrimination, 1-2ms), and header analysis (MEDIUM-HIGH discrimination, <0.5ms)—these form the mandatory Tier 1 checks that execute synchronously on every request. Behavioral timing and mouse dynamics provide the highest standalone accuracy (87%) but require session accumulation and 10-50ms ML inference, relegating them to asynchronous Tier 3 processing where risk scores update in background threads. JavaScript execution tests achieve 95%+ accuracy against simple bots but add latency through challenge injection (1-2ms) and validation (0.5-1ms), making them suitable for suspected bots rather than universal deployment.
|
||||
|
||||
The practical three-tier architecture allocates computational budgets strategically: Tier 1 (0.5-1ms total) performs immediate checks on TLS fingerprint extraction, HTTP header validation, User-Agent parsing, and cookie presence; Tier 2 (1-5ms budget) conducts TLS fingerprint database lookups, HTTP/2 fingerprint extraction, and cross-verification of the TLS-UserAgent-HTTP/2 consistency triangle; Tier 3 (async/next request) accumulates behavioral timing, analyzes path traversal patterns, performs ML inference on aggregated features, and validates JavaScript challenges. This tiered approach ensures 95%+ of requests complete Tier 1 checks within budget, with only suspicious traffic (flagged by Tier 1) proceeding to more expensive validation.
|
||||
|
||||
Implementation complexity varies significantly: User-Agent parsing, header analysis, and cookie tests qualify as LOW complexity with simple string operations; TLS fingerprinting (JA3/JA4) and HTTP/2 fingerprinting rate as MEDIUM complexity requiring binary protocol parsing but with established open-source implementations (Fingerproxy, JA4 reference); TCP/IP fingerprinting and behavioral ML models rank as HIGH complexity needing packet capture infrastructure or sophisticated ML pipelines. For Ᾰenebris targeting rapid deployment, the recommendation prioritizes LOW and MEDIUM complexity features initially, deferring HIGH complexity features to later iterations once core detection establishes baselines. Cloudflare's architecture achieving sub-100μs latency at 46M req/sec demonstrates this tiered approach works at extreme scale—they extract TLS and HTTP/2 fingerprints in the edge layer (0-2ms), perform database lookups in the decision engine (2-5ms), and run behavioral ML asynchronously.
|
||||
|
||||
## ML model architectures optimized for inference speed
|
||||
|
||||
The sub-millisecond latency constraint eliminates entire categories of ML models from consideration, narrowing viable options to tree-based methods and simple linear models. Decision trees achieve the fastest inference at 0.4-30 microseconds with 80-90% accuracy, making them ideal for first-stage filtering despite lower accuracy than ensemble methods. Single decision trees with max depth 8-10 compile to simple conditional branches executing in CPU cache with minimal memory footprint (5-15% of flash in embedded systems). Random Forests improve accuracy to 85-95% while maintaining acceptable latency: standard implementations require 10-200μs, but Intel oneDAL optimizations using AVX-512 SIMD instructions achieve 15.5x speedups, bringing inference to 5-100μs range. For 100-tree forests with depth 6-8, this translates to 50-150μs per prediction—well within the 500μs budget when combined with feature extraction overhead.
|
||||
|
||||
Gradient boosting models (XGBoost, LightGBM, CatBoost) provide the best accuracy-latency balance for bot detection, with LightGBM showing 1.5-2x faster inference than XGBoost at comparable accuracy (90-95%). Cloudflare's production deployment uses CatBoost achieving 309μs P50 and 813μs P99 latency through aggressive Rust optimization: zero-allocation code paths, custom CatBoost integration, buffer reuse for categorical features, and single-document evaluation APIs. Intel oneDAL compilation provides 24-36x speedups for XGBoost and 14.5x for LightGBM when deployed on Intel CPUs, bringing standard 100-500μs inference down to 10-150μs range. FPGA acceleration via Xelera Silva pushes extreme performance boundaries with 5-10μs inference (single-digit microseconds), though this requires specialized hardware primarily justified in high-frequency trading scenarios. For most deployments, CPU-based inference with Intel optimizations provides the best price-performance ratio.
|
||||
|
||||
Model configuration directly impacts latency: limiting tree count to 100-300 estimators, constraining max depth to 6-8 levels, and applying INT8 quantization achieves 2-4x speedup with <1% accuracy loss. Dynamic quantization converts FP32 models to INT8 automatically, reducing memory footprint 4x and enabling better CPU cache utilization. Model size matters critically—small models (50-100 trees, depth 6) occupying 1-5MB fit in L3 cache enabling 50-100μs inference; medium models (200-500 trees, depth 8) at 5-20MB may cause cache misses pushing latency to 100-500μs; large models (500+ trees) exceeding 20MB require DRAM access adding 50-100ns per cache miss. The recommendation for <1ms constraints: target 100-200 estimators at depth 6-8, apply INT8 quantization, and benchmark rigorously on production hardware before deployment.
|
||||
|
||||
LSTM and RNN models fail the latency requirement despite excellent sequence modeling capabilities, exhibiting 2-50ms inference times even with optimization. Standard TensorFlow Lite implementations require 10-50ms for simple LSTMs, while aggressive optimization (Plumerai) achieves 3-5ms at best—still 6-10x over budget. The architectural reasons are fundamental: recurrent layers process sequences step-by-step without parallelization, require 16-bit precision for accuracy (versus 8-bit for trees), and involve expensive matrix operations per time step. Sequence lengths directly multiply latency, making multi-request behavioral modeling prohibitively expensive. The only scenario justifying LSTMs is asynchronous behavioral analysis where 10-50ms latency is acceptable, processing accumulated session history in background threads while using cached risk scores for realtime decisions. For critical-path inference under 1ms, rule them out entirely.
|
||||
|
||||
### Comprehensive model comparison for ultra-low latency deployment
|
||||
|
||||
A detailed latency-accuracy-complexity analysis reveals clear tiers of viability. Models achieving <100μs inference qualify as TIER 1 (fully viable): Decision Trees (0.4-30μs, 80-90% accuracy, very low memory), Logistic Regression (1-10μs, 75-85% accuracy, negligible memory), and optimized Random Forests via Intel oneDAL (5-100μs, 85-95% accuracy, low memory). TIER 2 models (100-400μs, viable with optimization) include standard Random Forests (10-200μs), optimized XGBoost/LightGBM (10-150μs with Intel oneDAL), and optimized CatBoost (309μs P50 demonstrated by Cloudflare). TIER 3 models (400-1000μs, marginal cases) encompass standard gradient boosting implementations (100-500μs), Isolation Forest (50-200μs but lower accuracy at 85-93%), and multi-stage pipelines combining fast first-stage filtering with detailed second-stage analysis. Models exceeding 1ms and thus unsuitable include LSTM/RNN (2-50ms), unoptimized neural networks (1-10ms), and large ensembles without hardware acceleration (>2ms).
|
||||
|
||||
The latency-accuracy tradeoff curve shows diminishing returns beyond 500μs investment: at <100μs budget, Decision Trees provide 80-85% accuracy suitable for edge devices; at <500μs, optimized Random Forests or LightGBM achieve 90-94% accuracy ideal for real-time web requests; at <1ms, multi-stage pipelines with optimized GBDT reach 92-97% accuracy representing the practical maximum for sub-millisecond constraints. Pushing beyond 1ms to 1-10ms enables standard gradient boosting ensembles achieving 94-98% accuracy, while >10ms allows complex deep learning reaching 96-99%+ but violates latency requirements. The sweet spot for production bot detection lies at 300-800μs total budget: allocate 100-200μs for feature extraction, 200-500μs for ML inference (optimized gradient boosting), and 50-100μs for caching and decision enforcement.
|
||||
|
||||
Multi-stage detection pipelines maximize accuracy within constraints by filtering the majority of requests quickly and analyzing suspicious cases more deeply. A recommended two-stage architecture uses Stage 1 Decision Tree filtering (50μs) to classify 80-90% of obvious cases (clear bots or clear humans), passing only the suspicious 10-20% to Stage 2 Random Forest analysis (300-500μs) for final determination. Total latency calculates as: 90% * 50μs + 10% * 550μs = 45μs + 55μs = 100μs average, with P95 at ~550μs and P99 at ~650μs—comfortably under 1ms. This approach achieves 92-97% accuracy (better than single models) while maintaining aggressive latency targets. Implementation requires careful threshold tuning to avoid overwhelming Stage 2 with false positives; target 10-20% Stage 2 routing rate for optimal balance.
|
||||
|
||||
Combining supervised and unsupervised methods provides defense-in-depth: run supervised Random Forest (200μs) for known attack patterns in parallel with unsupervised Isolation Forest (150μs) for novel anomalies, combining scores with weighted ensemble (10μs overhead). The parallel architecture ensures total latency equals the maximum path (~200μs) rather than sum (~360μs), maintaining efficiency while catching both familiar and zero-day bots. Supervised models excel at precision for known patterns while unsupervised models provide recall against evolving threats. Weighted combination typically assigns 70-80% weight to supervised scores and 20-30% to unsupervised for bot detection workloads where false positives carry high cost.
|
||||
|
||||
## Training data sources and continuous learning infrastructure
|
||||
|
||||
Public bot detection datasets provide essential starting points but require careful selection based on domain relevance. The Bot-IoT dataset (69.3 GB full, 1.07 GB 5% subset) contains 72M+ records of IoT network traffic with botnet attacks including DDoS, reconnaissance, keylogging, and data exfiltration, available via IEEE DataPort and Kaggle with binary and multi-class labels. CTU-13 from Stratosphere Lab provides 13 scenarios of real botnet captures with manual labeling and NetFlow data (1.9 GB compressed), though complete pcap files remain unreleased for privacy. UNSW-NB15 offers 2.54M flows with nine attack categories and 100GB of pcap data, widely used in academic research with 95% benign and 5% attack ratio representing realistic imbalance. For web application bot detection specifically, the Bournemouth Web Bot Detection dataset includes web server logs with mouse movement behavioral biometrics across 28-61 pages, categorizing traffic as human, moderate bot, or advanced bot with behavioral ground truth.
|
||||
|
||||
NetFlow-based datasets provide preprocessed features ideal for ML training: NF-UNSW-NB15-v3 contains 2.36M flows with 53 NetFlow features and 5.4% attack rate; NF-BoT-IoT-v3 includes 16.99M flows with 99.7% attack concentration (extreme imbalance requiring careful sampling); NF-ToN-IoT-v3 offers 27.5M flows covering 10 attack classes including backdoor, injection, MITM, ransomware, and XSS with 38.98% attack prevalence. The University of Queensland maintains comprehensive ML-based NIDS dataset collections merging multiple sources with unified labeling—NF-UQ-NIDS combines 75.9M records while CIC-UQ-NIDS provides 27.2M records with CICFlowMeter features. These merged datasets solve the critical challenge of model generalization by providing diverse traffic sources, attack types, and network environments in unified formats that prevent overfitting to single-source characteristics.
|
||||
|
||||
Class imbalance poses fundamental challenges since production environments exhibit 30-50% bot traffic (per 2024 industry statistics) but datasets often show 95% benign traffic or conversely 99% attack concentration. SMOTE (Synthetic Minority Over-sampling Technique) addresses this by generating synthetic samples through k-nearest neighbor interpolation rather than simple duplication, creating new instances along line segments connecting minority class examples. Variants include Borderline-SMOTE focusing on decision boundary instances, ADASYN using adaptive density-based generation, and SMOTE-TOMEK combining oversampling with noise removal. For bot detection on Bot-IoT's extreme imbalance (99.7% attacks), researchers successfully applied SMOTE-DRNN achieving improved detection rates for minority legitimate traffic classes. Implementation requires tuning k-neighbors (typically 5) and sampling strategy parameters while monitoring for overfitting to synthetic distributions.
|
||||
|
||||
Cost-sensitive learning provides an alternative or complement to resampling by assigning asymmetric misclassification costs during training. For bot detection where false positives (blocking legitimate users) are costlier than false negatives (missing some bots), class weights like `{legitimate: 50, bot: 1}` make models 50x more sensitive to errors on legitimate traffic. This preserves original data distributions while directly encoding business priorities into optimization objectives. Sklearn's RandomForestClassifier supports `class_weight='balanced'` for automatic adjustment or manual cost assignment; XGBoost and LightGBM accept `scale_pos_weight` parameters. Cost-sensitive approaches often outperform pure resampling by avoiding distribution distortion while aligning model objectives with operational costs—a false positive may cost user trust and revenue while a false negative costs only marginal attack success.
|
||||
|
||||
### Continuous learning and drift detection for production ML
|
||||
|
||||
Online learning approaches enable continuous adaptation without full retraining by updating models incrementally as new data arrives. Hoeffding Trees build decision trees for streaming data using statistical guarantees about when sufficient samples justify splits, achieving O(1) update time per example. Semi-supervised learning leverages unlabeled data through pseudo-labeling: train on labeled examples, predict on unlabeled data, combine high-confidence predictions as pseudo-labels for retraining. This proves particularly effective in bot detection where verification latency (knowing if blocked traffic was truly a bot) creates label scarcity. The River Python library (formerly scikit-multiflow) provides production-ready online learning implementations including online Random Forests, online gradient boosting, and adaptive windowing for concept drift detection.
|
||||
|
||||
Model retraining cadences balance freshness against computational cost: daily retraining suits high-stakes applications like ad fraud or financial security where bot tactics evolve rapidly (Amazon's SLIDR system retrains daily); weekly retraining serves moderate drift scenarios typical in e-commerce; monthly retraining suffices for stable environments with slow-evolving threats. Cloudflare Bot Management uses continuous monitoring with gradual model deployment, A/B testing new models against champions before full rollout. Trigger-based retraining supplements scheduled updates: retrain when accuracy drops below threshold (e.g., 90%), when drift detection algorithms signal significant distribution changes, or when volume-based triggers accumulate sufficient new samples (every 100K verified examples). Hybrid approaches combine monthly full retraining with weekly incremental updates and emergency retraining triggered by anomalies.
|
||||
|
||||
Drift detection algorithms automatically identify when model performance degrades due to data distribution changes, enabling timely retraining interventions. The ADWIN (Adaptive Windowing) method dynamically adjusts window sizes by growing windows when distributions remain stable and shrinking when drift is detected, finding sub-windows with statistically distinct averages. Page-Hinckley test provides sequential change detection by calculating cumulative differences from mean values, triggering alarms when thresholds are exceeded with sensitivity dependent on parameter tuning. Statistical approaches include Kolmogorov-Smirnov tests comparing recent versus historical distributions, Population Stability Index (PSI) tracking feature distribution shifts (PSI <0.1 indicates no drift, 0.1-0.25 moderate drift, >0.25 significant drift requiring retraining), and Kullback-Leibler divergence measuring probabilistic distribution differences.
|
||||
|
||||
Performance-based drift detection monitors model accuracy, precision, recall, and F1-scores over sliding time windows, triggering alerts when sustained degradation occurs. Since ground truth labels often have verification latency (determining if challenged traffic was truly a bot requires human review or honeypot confirmation), proxy metrics like prediction confidence score distributions, feature distribution shifts, and error rate trends provide early warning signals. Tools like Evidently AI (25M+ downloads), NannyML (drift detection without ground truth), and Frouros (open-source drift detection library) provide production-ready monitoring dashboards and automated alerting. Best practices combine multiple detection methods—statistical tests for feature drift, performance metrics for model drift, and contextual approaches comparing lightweight recent models versus stable historical models to identify when fresh data requires new learning.
|
||||
|
||||
## Deployment architecture for Haskell integration with ONNX Runtime
|
||||
|
||||
Achieving <0.5ms added latency for ML inference in a Haskell-based reverse proxy eliminates network-based microservice architectures as viable options. Python microservices via FastAPI or Flask add inevitable network round-trip latency—even localhost gRPC calls incur 0.1-0.3ms minimum for network transit, with cross-network calls adding 1-5ms+ overhead. Research from Luis Sena demonstrates FastAPI's async model can actually degrade ML inference performance by 4x (from 5-7ms to 20-55ms) under concurrent workload due to event loop blocking on CPU-intensive operations. While gRPC shows 7-10x better performance than REST for large payloads and supports HTTP/2 multiplexing, the fundamental network bottleneck remains: PCIe transfer to GPUs adds 2-5μs, kernel launch overhead introduces microsecond delays, and even optimized connection pooling with keep-alive cannot reliably guarantee sub-500μs total latency including feature transmission and result retrieval.
|
||||
|
||||
The optimal architecture for Ᾰenebris uses in-process ONNX Runtime via Haskell FFI, eliminating network overhead entirely by embedding the ML inference engine directly in the proxy process. ONNX Runtime's C++ API achieves 5-15ms inference for typical models unoptimized, but with graph optimization (ORT_ENABLE_ALL), INT8 quantization, and proper threading configuration (intra-op threads: 2-4), simple models reach 0.2-0.3ms inference—within the 0.5ms budget when combined with feature extraction. Real-world examples include Cloudflare's Rust-based CatBoost integration achieving 309μs P50 latency, and research showing BERT-Large inference <1ms with TensorRT 8 optimization. The critical success factor is model simplicity: logistic regression (0.05-0.15ms), decision trees with depth ≤10 (0.10-0.25ms), small neural networks with 1-2 hidden layers (0.15-0.30ms), or gradient boosting with ≤20 trees (0.20-0.40ms).
|
||||
|
||||
Haskell FFI integration uses CApiFFI for robust C++ interoperation with minimal overhead. The CApiFFI extension generates intermediary C wrapper files providing stable interfaces despite ABI differences, with FFI call overhead of 1-5 microseconds per invocation. Using the `unsafe` FFI keyword eliminates runtime safety checks for non-callback functions, reducing overhead to sub-microsecond levels—critical for meeting aggressive latency targets. Memory management employs `ForeignPtr` with custom finalizers for automatic ONNX session cleanup, adding only 10-20 nanoseconds overhead (negligible). The pattern involves creating C wrapper functions that encapsulate ONNX Runtime C++ API complexity, exposing simple C-style interfaces that Haskell can call efficiently through FFI with `Ptr` types enabling zero-copy data passing for input features and output predictions.
|
||||
|
||||
The implementation architecture centers on in-memory prediction caching using Haskell's STM (Software Transactional Memory) to achieve cache hits in 0.05-0.15ms. Features are hashed to generate cache keys, with predictions stored alongside timestamps for TTL-based eviction (60-300 seconds typical). When cache hit rate exceeds 80-90%, average latency drops dramatically: most requests (cache hits) complete in ~0.12ms while cache misses require full inference at ~0.35ms, yielding P50 latency around 0.15ms and P95 around 0.40ms—comfortably within the 0.5ms budget. Cache warming strategies pre-compute predictions for common feature patterns during low-traffic periods, and LRU eviction maintains cache size bounds while maximizing hit rates. The Warp request handler pipeline flows: parse request → extract features (0.05ms) → check STM cache (0.05ms) → on miss call FFI (0.01ms) → ONNX inference (0.20ms) → store in cache (0.02ms) → format response (0.02ms).
|
||||
|
||||
### Detailed latency breakdown and optimization techniques
|
||||
|
||||
The target latency budget of <0.5ms allocates resources strategically across the request path. Feature extraction consumes 0.05ms for parsing TLS fingerprints, HTTP headers, and request metadata—this is synchronous and unavoidable but optimized through efficient Haskell parsing with strict evaluation and minimal allocations. Cache lookup via STM adds 0.05ms for in-memory hash table access with lock-free concurrency, achieving this through Software Transactional Memory's optimistic concurrency control where read-only transactions (cache hits) complete without coordination overhead. FFI call overhead contributes just 0.01ms when using `unsafe` FFI and `Ptr` types for zero-copy data transfer, avoiding marshaling by passing raw pointers to feature arrays. ONNX Runtime inference represents the largest component at 0.20ms, achievable through INT8 quantized models with 100-200 trees at depth 6-8, graph optimization enabled, and intra-op thread count set to 2-4 cores.
|
||||
|
||||
Model quantization provides 2-4x speedup with minimal accuracy loss: dynamic quantization converts FP32 models to INT8 post-training, reducing memory footprint 4x and enabling better cache utilization; static quantization requires calibration data but achieves best results; 4-bit quantization pushes further with 8x memory reduction though with 1-3% accuracy degradation. Graph optimization through ONNX Runtime's offline optimization pipeline fuses operators, eliminates redundant nodes, and reorders operations for cache-friendly memory access patterns. Threading configuration critically impacts latency—over-threading causes context switching overhead while under-threading leaves cores idle; optimal configuration sets intra-op threads to 2-4 (parallelizing within operators like matrix multiplication) and inter-op threads to 1 (no parallel operator execution needed for sub-millisecond inference).
|
||||
|
||||
CPU allocation separates Warp networking threads from ONNX inference threads to prevent interference: reserve 1-2 cores for Warp's lightweight green threads handling 100k+ req/sec concurrency, allocate remaining cores (6-14 on typical servers) to ONNX Runtime with thread pinning via Linux taskset to ensure cache locality. GHC runtime options like `+RTS -N8 -A32m -qg -I0` configure 8 OS threads, 32MB nursery for young generation (reducing GC frequency), parallel GC, and immediate scheduling without idle time. Memory bandwidth considerations matter at scale: typical CPUs provide 40-100 GB/sec bandwidth with 1-10 GB/sec consumed by high-throughput inference; L1/L2/L3 cache hierarchies (32KB/256KB/2-32MB) determine whether models fit in fast memory (sub-nanosecond access) or require DRAM (50-100ns latency per cache miss).
|
||||
|
||||
Model hot-swapping without downtime uses atomic TVar updates: load new ONNX models in background threads, validate inference latency meets requirements, then atomically swap the active model pointer visible to request handlers. Blue-green deployment maintains two model versions (blue=current, green=new) with load balancer switching traffic instantaneously upon validation. Canary deployment gradually routes 1%→5%→25%→50%→100% traffic to new models while monitoring false positive rates, inference latency, and user complaints, enabling automatic rollback if degradation occurs. The model registry maintains active and candidate model references with version tracking, allowing incremental rollout with per-request routing decisions based on hash(user_id) for consistent user experience during transitions.
|
||||
|
||||
## Production implementation patterns for false positive minimization
|
||||
|
||||
False positive mitigation represents the highest priority operational concern since blocking legitimate users destroys business value and user trust far more than missing some bots. Threshold tuning begins with ROC curve analysis visualizing true positive rate versus false positive rate across score thresholds (0.1-0.9), selecting operating points that prioritize precision over recall. Start conservatively with 0.7-0.8 thresholds yielding 95%+ precision, gradually lowering to 0.5-0.6 as confidence builds through production validation. Precision-recall curves guide this optimization by showing the tradeoff explicitly: bot detection typically targets precision >95% (fewer than 5% of blocks are mistakes) and recall >85% (catching 85%+ of bots), accepting 10-15% miss rate to avoid false positives. Cost-sensitive training assigns asymmetric weights like class_weight={legitimate: 50, bot: 1} making models 50x more sensitive to legitimate user errors.
|
||||
|
||||
Multi-threshold strategy implements gradual response based on prediction confidence scores rather than binary block/allow decisions. Scores 0.9-1.0 trigger hard blocking (403 response) for high-confidence bots; scores 0.7-0.9 serve CAPTCHA challenges testing JavaScript execution and human problem-solving; scores 0.5-0.7 apply rate limiting (reducing from 100 to 5-20 req/min) rather than blocking; scores 0.3-0.5 trigger increased monitoring and logging only; scores 0.0-0.3 receive normal processing. This progressive challenge approach minimizes user friction for uncertain cases—legitimate users encountering CAPTCHA can proceed after solving it, while bots lacking JavaScript execution or challenge-solving capability are filtered out. Adaptive thresholds tune per-endpoint based on criticality: login and payment endpoints use 0.8+ thresholds for blocking, standard browsing uses 0.7+, public APIs use 0.6+.
|
||||
|
||||
Whitelist management maintains trusted entity lists bypassing ML detection: verified API keys from partners, OAuth-authenticated users (reduced scrutiny after authentication), known good IP ranges (corporate offices, verified partners), and legitimate bots verified via IP and User-Agent (Googlebot, Bingbot confirmed against published IP ranges). Dynamic whitelisting automatically promotes IPs to trusted status after N successful sessions (e.g., 10 clean sessions in 7 days) with TTL-based expiration (30 days typical). This learns from user behavior patterns—if an IP consistently acts human-like over extended periods, reduce detection intensity to minimize false positive risk while maintaining monitoring for behavioral changes.
|
||||
|
||||
Feedback mechanisms collect ground truth through multiple channels: customer-reported false positives (blocked users contacting support), verified bot traffic from honeypots, post-purchase validation (users completing transactions prove legitimacy), and challenge success rates (CAPTCHAs consistently solved suggest legitimate). Cloudflare's approach uses verified bot databases and customer feedback to continuously refine models through monthly or quarterly retraining incorporating new ground truth. A/B testing frameworks compare champion versus challenger models on business metrics (false positive rate, user complaint volume, conversion rates) rather than solely accuracy, with statistical significance testing (p-value <0.05) over 1-2 week experiments and automatic rollback if key metrics degrade. Multi-stage verification applies challenge-response for uncertain cases (scores 0.5-0.8) rather than immediate blocking, learning from challenge outcomes to refine future predictions.
|
||||
|
||||
### Monitoring, drift detection, and operational resilience
|
||||
|
||||
Comprehensive monitoring tracks model performance, inference latency, error rates, and resource utilization with automated alerting thresholds. Model performance metrics include precision >95% (alert if <90%), recall >85%, F1 score >0.90, and AUC-ROC >0.95 calculated daily using verified ground truth from feedback mechanisms. Inference latency percentiles (P50, P95, P99, P99.9) are monitored per-request with alerts when P95 exceeds 25ms or P99 exceeds 50ms—sustained latency degradation indicates capacity issues, model complexity growth, or cache performance problems. False positive and false negative rates are tracked separately per endpoint and attack type, with FPR <0.1% (fewer than 1 in 1000 blocks are mistakes) considered acceptable and FNR <10-15% tolerated given the priority on avoiding false positives.
|
||||
|
||||
Feature drift detection uses statistical tests comparing recent versus historical feature distributions to identify when model assumptions become stale. Population Stability Index (PSI) calculates distribution divergence where PSI <0.1 indicates no drift, 0.1-0.25 signals moderate drift warranting investigation, and >0.25 demands immediate retraining. Kolmogorov-Smirnov tests compare feature distributions across time windows, triggering alerts when p-values indicate statistically significant differences. Prediction drift monitoring tracks model output distributions—if average bot scores shift from 0.3 to 0.5 over weeks without ground truth changes, either feature distributions have drifted or bot tactics have evolved. Tools like Evidently AI provide automated drift dashboards, WhyLabs specializes in embedding drift for unstructured data, and Arize AI offers commercial platforms with alerting integrations.
|
||||
|
||||
Circuit breaker patterns implement failure handling through three states: CLOSED (normal operation, requests flow to ML service with failure tracking), OPEN (ML service failing, immediately use fallback without calling service), and HALF-OPEN (testing recovery, allowing limited requests to probe service health). Configuration parameters include failure threshold (5 consecutive failures or 50% error rate over 1-minute window triggers OPEN state), timeout duration (30 seconds in OPEN before entering HALF-OPEN), and recovery threshold (3 consecutive successes in HALF-OPEN to return to CLOSED). Fallback strategies range from rule-based detection (IP reputation, rate limiting, User-Agent validation) to cached predictions (recent scores with 5-10 minute TTL) to allow-all versus deny-all policies based on endpoint criticality.
|
||||
|
||||
Graceful degradation defines multiple operational levels: Level 0 (full ML + all features), Level 1 (ML + cached features only, sacrificing real-time enrichment), Level 2 (simpler model like decision tree instead of gradient boosting), Level 3 (rule-based detection only), Level 4 (minimal protection using known-bad IPs and basic rate limiting). Auto-degradation triggers based on observed latency: if inference exceeds 100ms switch to Level 1, if exceeding 500ms switch to Level 2, if circuit breaker opens switch to Level 3. Health checking employs liveness probes (every 5 seconds: GET /health → 200 OK), readiness probes (every 10 seconds: verify model loaded and resources <90%), and deep health checks (every 60 seconds: run sample inference with known input, verify latency <50ms). Kubernetes integration provides automatic recovery through liveness/readiness probe-driven pod restarts when health checks fail consistently.
|
||||
|
||||
## State-of-the-art techniques and commercial approaches for 2024-2025
|
||||
|
||||
The bot detection arms race has intensified with AI-powered bots using Large Language Models achieving 29.6% detection evasion when trained adversarially against detection systems. Research published in February 2024 demonstrates mixture-of-heterogeneous-experts frameworks combining multimodal signals (user metadata, text content, network graphs) with LLM-based analysis improves detection by 9.1% over baselines, but this same technology enables sophisticated bots to manipulate features intelligently. Bayesian uncertainty-aware detection frameworks introduced in July 2024 separate epistemic uncertainty (model confidence) from aleatoric uncertainty (data noise), enabling confidence-based decision making where high-uncertainty predictions trigger challenges rather than blocks. Self-supervised contrastive learning (BotSSCL) achieves 67% cross-dataset generalization and only 4% adversarial success rate by learning robust representations resilient to distribution variations.
|
||||
|
||||
Graph Neural Networks capture coordinated bot campaign patterns through network structure analysis, with recent innovations including XG-BoT (explainable GNN for botnet forensics), BotHP (heterophily-aware detection for camouflaged interactions), and dynamic GNN with temporal transformers for evolving networks. These approaches achieve 2.4-3.1% accuracy improvements over traditional methods by modeling relationships between entities—a residential proxy network routing traffic from 100 bots appears benign at the IP level but shows suspicious patterns when graph analysis reveals concentrated ASN distribution, identical request sequences, or synchronized timing. Implementation complexity is high, requiring graph construction and distributed inference, but for applications with social network structure (forums, marketplaces) the detection accuracy gains justify the investment.
|
||||
|
||||
Commercial bot detection solutions demonstrate practical production architectures at massive scale. DataDome's two-step detection combines statistical/behavioral analysis with technical fingerprinting achieving <2ms decision time while processing trillions of signals across 30+ global PoPs, maintaining <0.01% CAPTCHA rate through aggressive true negative filtering. Their multi-layered AI adapts in <50ms to new attack patterns using TLS fingerprinting, Canvas/WebGL/Audio browser fingerprinting, behavioral analysis (mouse, typing, request patterns), and ML-based residential proxy detection. PerimeterX (HUMAN Security) focuses on behavioral fingerprinting with predictive analytics, analyzing mouse movements, keystroke dynamics, and request patterns through dynamic ML models that adapt in real-time to threats, deployed both cloud-native and on-premises for latency-sensitive applications.
|
||||
|
||||
Cloudflare Bot Management's v8 model (2024) specifically targets residential proxy networks with dedicated ML classifiers trained on distributed attack patterns, moving beyond IP reputation to examine request consistency, behavioral automation signals, and cross-account correlations. Their signature-based detection combined with ML algorithms generates bot scores 1-99 (1=bot, 99=human) with challenge mechanisms including rate limiting, CAPTCHA, and JavaScript proof-of-work. The architecture runs on Cloudflare's global edge network achieving sub-100μs latency impact at 46M+ req/sec scale through distributed model deployment with local inference and centralized training. AWS WAF Bot Control introduced residential proxy protection in 2023 using three-stage interaction (challenge → fingerprint → token) with silent proof-of-work and CAPTCHA actions, employing predictive ML to identify distributed attacks before they fully materialize.
|
||||
|
||||
### Residential proxy networks and sophisticated evasion techniques
|
||||
|
||||
Residential proxy networks pose the most challenging threat in 2024, evading 84% of traditional detection systems by routing bot traffic through legitimate residential IP addresses. These networks acquire 30-100 million IPs through mobile SDK bandwidth monetization (users trading bandwidth for free app services), browser extensions offering free VPN services while routing traffic, IoT device compromise (smart TVs, routers, cameras), and sometimes ISP partnerships for legitimate proxy services. The scale enables per-request IP rotation where each bot request originates from a different residential IP making IP reputation and rate limiting ineffective—traditional defenses see legitimate ISP addresses with normal request rates from each individual IP, missing the coordinated campaign.
|
||||
|
||||
Detection requires behavioral pattern analysis beyond IP: traffic diversity from single IP (multiple user sessions with distinct fingerprints suggesting proxy), geographic inconsistencies (IP geolocation not matching timezone/language headers), behavioral pattern sharing (multiple IPs showing identical request sequences or timing), and suspicious ISP/ASN concentrations (thousands of requests from a single mobile carrier AS in short timeframes suggesting SDK-based proxy). Graph analysis reveals coordination when individual nodes appear benign but network structure exposes campaigns—100 bots through residential proxies show benign IP patterns individually but graph clustering reveals they target the same endpoints with correlated timing. Single-request feature engineering focuses on signals present in first request: TLS fingerprint inconsistent with User-Agent claims, missing browser capabilities (no Canvas/WebGL support), or automation indicators (navigator.webdriver, headless browser artifacts).
|
||||
|
||||
Cloudflare's approach to residential proxy detection uses dedicated ML models trained on verified residential proxy traffic from threat intelligence feeds and honeypot validation, with features including request consistency across IP changes (same TLS fingerprint rotating IPs), behavioral automation signals (mechanical timing, missing mouse events), and statistical anomalies in ASN distributions. The model achieves lower false positive rates than IP reputation alone by focusing on behavior rather than network origin—a legitimate user behind residential proxy shows human behavioral patterns while a bot behind residential proxy exhibits automation signals. Multi-stage verification challenges suspected proxies with JavaScript execution tests, Canvas fingerprinting, and CAPTCHA rather than immediate blocking, learning from challenge outcomes to refine detection models continuously.
|
||||
|
||||
Headless browser detection has evolved as tools like puppeteer-stealth, undetected-chromedriver, and nodriver (2024) actively patch detection vectors. Traditional signals like navigator.webdriver === true, missing window.chrome object, and CDP artifact detection are increasingly ineffective as evasion libraries address these specific checks. Modern detection focuses on behavioral signals that are harder to replicate: missing mouse movements or mechanical linear paths, rapid form submission without natural interaction delays, uniform request timing patterns, and missing touch/scroll events. Server-side correlation combines multiple weak signals—any single indicator may be patched but the combination of TLS fingerprint, HTTP/2 fingerprint, User-Agent consistency, behavioral timing, and challenge solving creates a robust "fingerprint triangle" where sophisticated bots must spoof all dimensions consistently.
|
||||
|
||||
## Practical implementation roadmap and deployment strategy
|
||||
|
||||
The recommended implementation for Ᾰenebris follows a four-phase rollout balancing quick deployment against long-term sophistication. Phase 1 (Foundation, 2-4 weeks) establishes core detection with TLS fingerprinting (JA4), HTTP/2 fingerprinting (Akamai method), header analysis (order and completeness), and IP reputation baseline using commercial threat intelligence feeds. This tier implements simple ML models—decision trees or logistic regression—trained on public datasets (UNSW-NB15 or Bot-IoT) with basic rule-based fallback for circuit breaker failures. The infrastructure deploys ONNX Runtime via Haskell FFI with in-memory STM caching achieving 0.3-0.5ms latency, handling 80-85% detection accuracy sufficient to establish operational baselines and monitoring dashboards.
|
||||
|
||||
Phase 2 (Enhancement, 1-2 months) adds sophisticated ML with gradient boosting models (LightGBM or CatBoost) achieving 90-94% accuracy through training on domain-specific data collected from Phase 1 operations plus public datasets. Residential proxy detection begins through behavioral pattern analysis, request consistency tracking across IP changes, and ASN distribution anomaly detection trained on labeled residential proxy samples from threat intelligence. Adaptive challenge mechanisms implement multi-threshold scoring (hard block >0.9, CAPTCHA 0.7-0.9, rate limit 0.5-0.7) with challenge success feedback loops informing model retraining. Ensemble methods combine supervised gradient boosting with unsupervised Isolation Forest running in parallel for defense-in-depth against both known and novel attacks.
|
||||
|
||||
Phase 3 (Advanced, 3-6 months) deploys Graph Neural Networks if social network structure exists (forum posts, marketplace transactions, user relationships) to detect coordinated campaigns through network pattern analysis. Bayesian uncertainty quantification adds confidence scoring enabling nuanced decisions—high-uncertainty predictions trigger human review or additional challenges rather than automatic blocks. LLM-based detection experiments with mixture-of-experts frameworks on text-heavy features (form submissions, search queries) but with careful adversarial training to prevent LLM-guided evasion. Browser fingerprinting expands to Canvas, WebGL, and AudioContext with privacy compliance through fraud prevention legitimate interest under GDPR/CCPA, collecting only signals necessary for security with user transparency through privacy policies.
|
||||
|
||||
Phase 4 (Optimization, ongoing) establishes continuous learning infrastructure with drift detection (ADWIN, Page-Hinckley tests, PSI monitoring) triggering retraining cadence (weekly or monthly), A/B testing framework comparing champion versus challenger models on business metrics (FPR, conversion rate, latency), and threat intelligence integration subscribing to commercial feeds for emerging attack patterns. Model compression through quantization, pruning, and knowledge distillation optimizes inference latency, potentially incorporating hardware acceleration (Intel oneDAL, FPGA if justified by scale). Operational maturity grows through incident response playbooks, automated rollback procedures, comprehensive monitoring dashboards (Prometheus + Grafana), and privacy compliance audits ensuring GDPR/CCPA adherence with annual reviews.
|
||||
|
||||
### Latency optimization techniques and production considerations
|
||||
|
||||
Achieving sub-millisecond latency requires aggressive optimization across every component. Feature extraction optimization uses strict evaluation in Haskell to avoid lazy thunks accumulating in request handlers, unboxed types (Data.ByteString for binary data, Int for counters) to eliminate pointer indirection, and pre-allocated buffers for fingerprint computation avoiding garbage collection pressure. TLS and HTTP/2 fingerprint extraction occurs during connection setup outside the critical request path, with fingerprints cached per connection and reused for subsequent requests on the same TCP connection. Header parsing uses fast binary parsers (attoparsec) with zero-copy substring extraction via bytestring slicing rather than allocating new strings.
|
||||
|
||||
ONNX Runtime optimization applies graph optimization offline during model export (ORT_ENABLE_ALL level), dynamic quantization to INT8 using quantize_dynamic() post-training with calibration data for validation, and compiled model deployment where ONNX graphs are compiled to optimized machine code via TensorRT or OpenVINO for target hardware. Session configuration sets intra_op_num_threads to 2-4 (parallelizing within operators without over-threading), inter_op_num_threads to 1 (no parallel operator execution for simple models), dynamic_block_base to 4 (reducing latency variance), and memory_pattern optimization enabled for predictable memory access. Model selection constraints enforced: maximum 200 trees for gradient boosting, maximum depth 8, INT8 quantization applied, final model size <20MB to fit L3 cache.
|
||||
|
||||
Prediction caching via STM achieves lock-free concurrency through optimistic transaction execution, with cache keys generated by hashing feature vectors (Fast-murmur3 or xxHash providing 1-5μs hash computation), TTL-based eviction (60-300 seconds typical for bot scores), and LRU policy limiting cache size to 10-100MB (100k-1M entries). Cache warming pre-computes predictions for common feature patterns identified through request profiling, scheduled during low-traffic periods (overnight) or triggered on model deployment. Hit rate monitoring targets >80% with alerts when dropping below 70%, as sustained low hit rates indicate feature distribution shift or cache configuration problems. Cache invalidation on model updates ensures new models receive fresh inference data rather than stale predictions from previous versions.
|
||||
|
||||
CPU and memory optimization pins Warp worker threads to cores 0-1, ONNX inference threads to cores 2-7 (on 8-core system) via Linux taskset for cache locality, and uses GHC runtime options `+RTS -N8 -A32m -qg -I0 -qb0` configuring 8 OS threads, 32MB nursery (reducing minor GC frequency), parallel GC, immediate scheduling, and no allocation area limits. NUMA-aware deployment on multi-socket systems allocates inference workers to local memory banks avoiding cross-socket memory access latency (40-60ns penalty). Memory bandwidth monitoring ensures inference doesn't exceed 70% of available bandwidth (leaving headroom for traffic spikes), with model quantization and compression reducing bandwidth pressure by 4x compared to FP32 models.
|
||||
|
||||
## Critical success factors and deployment readiness
|
||||
|
||||
Success in production ML bot detection hinges on prioritizing false positive minimization above detection rate—blocking one legitimate user causes more business damage than missing several bots. The operational principle "90% detection with 0.1% false positives beats 95% detection with 1% false positives" guides all threshold tuning, model selection, and architecture decisions. Implement multi-stage challenges rather than immediate blocking: only scores >0.9 warrant hard blocks, 0.7-0.9 receive CAPTCHAs allowing legitimate users to proceed, and 0.5-0.7 get rate limiting rather than denial. Track false positive reports obsessively through customer support channels, conversion funnel drop-offs at challenge points, and feedback mechanisms, using this ground truth to continuously refine models and thresholds.
|
||||
|
||||
Latency optimization for <1ms requires eliminating network hops (use in-process ONNX rather than microservices), aggressive model simplification (100-200 trees at depth 6-8), INT8 quantization, and cache hit rates >80%. The architectural decision tree flows: can the model inference achieve <200μs? Yes → in-process deployment viable; No → simplify model or use multi-stage pipeline with fast first-stage filter. Can cache hit rate reach 80%? Yes → average latency <0.2ms achieved; No → investigate feature cardinality, consider feature bucketing to reduce cache key diversity. Does the Haskell FFI overhead exceed 10μs? Yes → review unsafe FFI usage and eliminate marshaling; No → acceptable overhead for integration. Monitor P95 and P99 latencies continuously, alerting when P95 exceeds 0.5ms as sustained high-percentile latency indicates capacity issues or model complexity growth.
|
||||
|
||||
Operational resilience requires circuit breakers with tested fallback strategies (rule-based detection as minimum viable protection), graceful degradation across five operational levels (full ML → cached features → simpler model → rules-only → minimal protection), and automated health checking with Kubernetes liveness/readiness probes enabling automatic recovery. Incident response playbooks document procedures for false positive spikes (immediate whitelist addition, threshold relaxation, root cause analysis within 1 hour), false negative spikes from novel attacks (lower thresholds immediately, add attack-specific rules, retrain within 24 hours), and ML service outages (circuit breaker fallback, on-call alert, restore or full fallback within 15 minutes). Conduct quarterly disaster recovery drills testing circuit breaker activation, model rollback procedures, and fallback performance under production load.
|
||||
|
||||
Privacy compliance with GDPR and CCPA is achievable through fraud prevention legitimate interest exemption but requires transparency and data minimization. Document explicitly that fingerprinting and behavioral analysis serve security purposes (detecting automated abuse), provide clear privacy policy disclosures describing what signals are collected and why, implement opt-out mechanisms for non-essential collection (beyond security-required signals), and enforce regional data residency (EU data stays in EU, California residents' data in CCPA-compliant storage). Use anonymous visitor IDs rather than cross-site tracking, limit data retention to security-necessary periods (7-90 days for most signals), and conduct annual privacy audits verifying compliance as regulations evolve. The fraud prevention exception explicitly permits fingerprinting for detecting automated abuse without consent, but best practices include transparent disclosure and minimal collection.
|
||||
|
||||
### Deployment checklist and model validation criteria
|
||||
|
||||
Pre-deployment validation requires model performance meeting accuracy thresholds (AUC >0.95, precision >95%, recall >85%), latency benchmarks (P50 <0.15ms, P95 <0.4ms, P99 <0.5ms tested on production hardware), and feature extraction testing (all fingerprinting code handling edge cases, malformed inputs, and adversarial inputs without crashes or excessive latency). Test fallback strategies by simulating ML service failures and verifying rule-based detection activates within 100ms, processes requests successfully, and maintains acceptable security posture. Configure circuit breakers with failure thresholds (5 consecutive failures or 50% error rate triggers OPEN state), timeout durations (30 seconds before HALF-OPEN), and recovery thresholds (3 successes to CLOSE), validating state transitions through fault injection testing.
|
||||
|
||||
Implementation monitoring establishes dashboards tracking model accuracy (daily calculation using verified ground truth), inference latency (P50/P95/P99/P99.9 percentiles), false positive rate (<0.1% target), false negative rate (<15% acceptable), cache hit rate (>80% target), error rate (<0.1% target), and resource utilization (CPU <80%, memory <85%, network bandwidth <70%). Alert configurations trigger on latency degradation (P95 >0.5ms for 5 minutes), accuracy drops (precision <90% or recall <80%), false positive spikes (>0.2% sustained), drift detection (PSI >0.25 for any feature), and service health failures (3 consecutive health check failures). Integrate alerts with on-call rotation and incident management systems ensuring 24/7 response capability for production issues.
|
||||
|
||||
Shadow deployment validates new models by running them in parallel with production models but not affecting user traffic for 1-7 days. During shadow mode, compare challenger model predictions against champion model and ground truth (when available), calculating relative performance metrics: if challenger shows >2% accuracy improvement and <10% latency increase and no increase in false positives, promote to canary deployment. Canary rollout gradually increases traffic to new model: 5% for 24 hours → 25% for 48 hours → 50% for 72 hours → 100% permanent, with automatic rollback if key metrics degrade at any stage. Monitor false positive reports, conversion rates, and user complaints during rollout as business metrics provide ground truth that technical metrics may miss.
|
||||
|
||||
A/B testing framework implements statistical rigor for model comparison through randomized traffic assignment (hash(user_id) % 100 determines variant), defines success metrics (primary: false positive rate, secondary: detection rate and latency, guardrail: conversion rate ≥95% of baseline), calculates required sample size (10k-100k requests per variant for 95% confidence detecting 10% relative difference), and runs experiments for sufficient duration (1-2 weeks capturing weekly seasonality). Analyze results using two-sample t-tests for latency comparisons, chi-squared tests for categorical outcomes (block/allow decisions), and relative risk ratios for false positive rates. Maintain A/B testing infrastructure permanently to enable continuous model improvement through experimental validation before full production rollout.
|
||||
|
||||
## Final recommendations for Ᾰenebris project
|
||||
|
||||
For a production-grade Haskell-based reverse proxy targeting <1ms added latency at 100k+ req/sec, implement in-process ONNX Runtime via unsafe Haskell FFI with quantized gradient boosting models (LightGBM or CatBoost) limited to 100-200 trees at depth 6-8. Deploy aggressive in-memory caching using STM with feature hashing and TTL-based eviction targeting >80% hit rate, achieving 0.12ms average latency on cache hits and 0.35ms on cache misses. Extract TLS fingerprints (JA4), HTTP/2 fingerprints (Akamai method), and header analysis in the critical path with behavioral timing accumulation in background threads, implementing multi-threshold scoring (>0.9 block, 0.7-0.9 CAPTCHA, 0.5-0.7 rate limit) to minimize false positives while maintaining security. Train initial models on UNSW-NB15 or Bot-IoT datasets with continuous learning infrastructure collecting domain-specific ground truth through feedback mechanisms (false positive reports, challenge outcomes, honeypot validation), retraining monthly with drift detection triggering emergency updates when PSI exceeds 0.25.
|
||||
|
||||
Prioritize operational resilience through circuit breakers with rule-based fallback, graceful degradation across five operational levels, comprehensive monitoring with automated alerting (latency, accuracy, false positives, drift), and incident response playbooks tested quarterly. For residential proxy detection, implement behavioral pattern analysis examining request consistency across IP changes, ASN distribution anomalies, and automation signals rather than relying on IP reputation alone. Maintain GDPR/CCPA compliance through fraud prevention legitimate interest, transparent privacy disclosures, data minimization (collect only security-necessary signals), and regional data residency. The measured approach balances rapid deployment (Phase 1 foundation in 2-4 weeks) against long-term sophistication (Phase 4 advanced GNN and LLM detection in 3-6 months), allowing iterative validation and operational learning before committing to complex architectures.
|
||||
|
||||
Avoid common pitfalls: don't use Python microservices for <1ms latency (network overhead makes this impossible); don't deploy ML without circuit breakers and fallback (outages will occur); don't use single 0.5 threshold for all endpoints (tune per criticality); don't block immediately on first suspicious signal (progressive challenges reduce false positives); don't ignore model drift (distributions shift, requiring monthly retraining); don't trust accuracy metrics alone (validate with user feedback and business metrics measuring actual impact). The production readiness criterion is not achieving 99% accuracy but rather maintaining <0.1% false positive rate while detecting 85-90% of bots with <0.5ms latency—user experience and system responsiveness matter as much as raw detection performance for a reverse proxy competing with nginx.
|
||||
|
||||
The state-of-the-art in 2024-2025 demonstrates sophisticated adversaries using LLMs for intelligent evasion, residential proxy networks routing through 30-100M legitimate IPs, and headless browser automation with detection-bypass libraries. Defense requires continuous adaptation through monthly retraining, threat intelligence integration, and A/B testing new detection techniques before production deployment. Success is measured not by detection rate alone but by the ratio of bots blocked to legitimate users impacted—prioritize false positive minimization obsessively, implement progressive challenges rather than immediate blocking, collect ground truth through multiple channels, and maintain operational humility recognizing that perfect detection is impossible but 90% detection with 0.1% false positives creates substantial business value for protecting the Ᾰenebris reverse proxy infrastructure.
|
||||
|
|
@ -1,695 +0,0 @@
|
|||
# Zero-copy proxying unlocks gigabit+ throughput in Haskell
|
||||
|
||||
Building a high-performance proxy in Haskell requires understanding zero-copy techniques, compiler optimizations, and profiling methodologies. Zero-copy operations using splice() and sendfile() eliminate CPU copies between kernel buffers, reducing latency by 20-40% and enabling proxy servers to forward 60+ Gbps on modest hardware. Combined with GHC's advanced optimization capabilities and proper benchmarking, Haskell can achieve performance comparable to nginx while maintaining type safety and maintainability. The key insight: HAProxy demonstrates 1 Gbps forwarding on a 3-watt device using splice(), while Warp reaches 50,000 requests/second matching nginx performance through careful optimization of ByteString usage, zero-copy file serving, and GC tuning.
|
||||
|
||||
This report provides implementation-ready guidance for building production-grade proxies in Haskell, covering syscall-level optimizations through Haskell's Foreign Function Interface, compiler flag tuning for maximum performance, and comprehensive profiling workflows to identify bottlenecks. The techniques here enable developers to leverage Linux kernel optimizations while working in a high-level functional language.
|
||||
|
||||
## Zero-copy fundamentals eliminate redundant data movement
|
||||
|
||||
Traditional I/O operations copy data four times: disk to kernel buffer, kernel to user space, user space to socket buffer, and socket buffer to network. Each copy consumes CPU cycles and memory bandwidth. Zero-copy techniques reduce this to two DMA transfers with zero CPU copies, keeping data in kernel space throughout the transfer.
|
||||
|
||||
**splice() enables socket-to-socket forwarding**. This syscall moves data between file descriptors using kernel pipe buffers without user-space copies. For proxy servers forwarding between client and backend sockets, splice() is essential. The syscall signature requires one descriptor to be a pipe, creating a two-step process: splice data from source socket to pipe, then from pipe to destination socket. Kernel implementation uses reference-counted page pointers rather than copying bytes—only metadata changes, not actual data.
|
||||
|
||||
Performance characteristics show dramatic improvements. At 10 Gbps with 16KB buffers, copy overhead represents only 6.25% of processing time on modern Xeon processors achieving 20 GB/s memory bandwidth. However, eliminating this overhead alongside reduced context switches (from 4 to 2) and minimal cache pollution enables **HAProxy to achieve 60 Gbps forwarding on 4-core machines**. The key limitation: at least one file descriptor must be a pipe, and NIC must support scatter-gather DMA for optimal performance.
|
||||
|
||||
**sendfile() optimizes file-to-socket transfers**. Designed for serving static files, sendfile() transfers data directly from file to socket without user-space intervention. Modern Linux implementations (5.12+) actually implement sendfile() as a wrapper around splice() internally. The API is simpler than splice(), requiring no intermediate pipe, making it ideal for serving cached content or static files in reverse proxy scenarios.
|
||||
|
||||
Performance benchmarks reveal significant gains. Netflix achieved 6.7x throughput improvement (6 Gbps to 40 Gbps) on FreeBSD using sendfile() optimizations. Java zero-copy implementations showed 26% faster file copies with 56% less CPU time and 65% fewer cache misses compared to traditional I/O. For production proxy workloads, Google's MSG_ZEROCOPY research demonstrated 5-8% improvements in real deployments, though simple benchmarks showed 39% gains—the difference attributable to zero-copy setup costs for smaller transfers.
|
||||
|
||||
**When zero-copy provides maximum benefit**: Large file transfers (>10KB), high-frequency forwarding operations, memory bandwidth-constrained systems, and static content serving all benefit substantially. Conversely, small messages (<4KB), SSL/TLS connections requiring user-space processing, and dynamic content generation see limited or no benefit from zero-copy techniques.
|
||||
|
||||
## Haskell FFI bridges to zero-copy syscalls
|
||||
|
||||
Haskell's Foreign Function Interface enables direct access to Linux syscalls while maintaining type safety. The key challenge lies in marshalling between Haskell's high-level types and C's low-level representations, particularly for file descriptors, buffers, and error handling.
|
||||
|
||||
**Basic FFI patterns establish syscall bindings**. Foreign imports declare C function signatures with appropriate type mappings: `CInt` for integers, `CSsize` for signed size types, `Ptr a` for pointers. The `unsafe` keyword speeds calls that cannot callback to Haskell, while `safe` allows blocking operations without freezing other Haskell threads. For zero-copy syscalls, unsafe imports suffice as they're simple kernel calls.
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE ForeignFunctionInterface #-}
|
||||
import Foreign.C.Types
|
||||
import System.Posix.Types (Fd(..))
|
||||
|
||||
foreign import ccall unsafe "splice"
|
||||
c_splice :: CInt -> Ptr CLong -> CInt -> Ptr CLong
|
||||
-> CSize -> CUInt -> IO CSsize
|
||||
|
||||
splice :: Fd -> Fd -> Int -> [SpliceFlag] -> IO Int
|
||||
splice (Fd fdIn) (Fd fdOut) len flags = do
|
||||
let cflags = foldr (.|.) 0 [f | SpliceFlag f <- flags]
|
||||
result <- c_splice fdIn nullPtr fdOut nullPtr
|
||||
(fromIntegral len) cflags
|
||||
if result == -1
|
||||
then throwErrno "splice"
|
||||
else return (fromIntegral result)
|
||||
```
|
||||
|
||||
**Existing libraries provide production-ready implementations**. The `splice` package offers cross-platform zero-copy transfers, automatically using Linux splice() on GNU/Linux and falling back to portable Haskell implementations elsewhere. Its API handles bidirectional forwarding for proxy scenarios:
|
||||
|
||||
```haskell
|
||||
import Network.Socket.Splice
|
||||
import Control.Concurrent (forkIO)
|
||||
|
||||
-- Bidirectional zero-copy proxy
|
||||
forkIO $ splice 4096 (clientSocket, Nothing) (backendSocket, Nothing)
|
||||
forkIO $ splice 4096 (backendSocket, Nothing) (clientSocket, Nothing)
|
||||
```
|
||||
|
||||
The `simple-sendfile` package powers Warp's high-performance static file serving. Used internally by Warp for ResponseFile handlers, it automatically selects optimal implementations: Linux sendfile(), FreeBSD/macOS native sendfile(), Windows TransmitFile(), or portable fallback. The API supports sending with headers in a single operation using the MSG_MORE flag:
|
||||
|
||||
```haskell
|
||||
import Network.Sendfile
|
||||
|
||||
sendfileWithHeader :: Socket -> FilePath -> FileRange
|
||||
-> IO () -> [ByteString] -> IO ()
|
||||
-- Sends headers and file data efficiently
|
||||
sendfileWithHeader sock path (PartOfFile offset len)
|
||||
tickle headers
|
||||
```
|
||||
|
||||
**WAI/Warp integration demonstrates production patterns**. Warp's `ResponseFile` constructor triggers zero-copy serving automatically. When serving static files, Warp uses sendfile() with header coalescing—sending HTTP headers via send() with MSG_MORE flag, then immediately calling sendfile() for the body. This optimization proved 100x faster for sequential requests by ensuring headers and body transmit in a single TCP packet.
|
||||
|
||||
Warp also implements file descriptor caching, controlled by `settingsFdCacheDuration`. Setting this to 10-30 seconds for static content eliminates repeated open() syscalls, though it requires caution in development environments where files change frequently. The default zero seconds prioritizes correctness over performance.
|
||||
|
||||
**Error handling requires careful EINTR and EAGAIN management**. Network syscalls can return EINTR (interrupted) or EAGAIN (would block) errors that require retry logic:
|
||||
|
||||
```haskell
|
||||
spliceWithRetry :: Fd -> Fd -> Int -> IO ()
|
||||
spliceWithRetry fdIn fdOut chunkSize = loop
|
||||
where
|
||||
loop = do
|
||||
result <- try $ splice fdIn fdOut chunkSize
|
||||
[spliceNonBlock, spliceMore]
|
||||
case result of
|
||||
Right 0 -> return () -- EOF
|
||||
Right n | n < chunkSize -> do
|
||||
threadWaitRead fdIn
|
||||
threadWaitWrite fdOut
|
||||
loop
|
||||
Right _ -> loop
|
||||
Left e | ioeGetErrorType e == eAGAIN -> do
|
||||
threadWaitWrite fdOut
|
||||
loop
|
||||
Left e -> throwIO e
|
||||
```
|
||||
|
||||
Integration with GHC's I/O manager via `threadWaitRead` and `threadWaitWrite` enables non-blocking operation without busy-waiting, crucial for handling thousands of concurrent connections efficiently.
|
||||
|
||||
## ByteString optimization reduces allocation pressure
|
||||
|
||||
Haskell's ByteString types provide efficient binary data handling essential for network protocols. Understanding internal representations and choosing appropriate variants dramatically impacts proxy performance.
|
||||
|
||||
**Strict ByteString uses contiguous memory with minimal overhead**. Internally represented as a ForeignPtr with offset and length, strict ByteStrings enable zero-copy slicing—multiple ByteStrings can reference the same underlying buffer with different offsets. This **splicing capability eliminates copying during HTTP header parsing**: parsing "GET /path HTTP/1.1" can produce three ByteStrings (method, path, version) by adjusting offsets without copying bytes.
|
||||
|
||||
Memory overhead measures approximately 48 bytes per ByteString (ForeignPtr metadata), but the actual byte data contains no pointers, meaning GC doesn't scan it—only the metadata structures. For large allocations (>409 bytes on 64-bit), ByteStrings use pinned memory requiring a global lock, potentially causing contention on systems with 16+ cores. However, pinned memory prevents GC from moving data, enabling safe FFI calls to C functions expecting stable pointers.
|
||||
|
||||
**Lazy ByteString implements streaming via chunk lists**. Represented as a lazy list of strict ByteString chunks (default 32KB each), lazy ByteStrings handle arbitrarily large data without loading everything into memory. The chunk list spine adds some GC overhead, but allows processing gigabyte files with constant memory usage. Critical insight from Warp's implementation: "Lazy ByteStrings manipulate large or unbounded streams without requiring the entire sequence resident in memory."
|
||||
|
||||
Conversion costs between variants matter significantly. `toStrict` forces entire lazy ByteString evaluation then copies all data (O(n) time and space). Conversely, `fromStrict` merely wraps a strict ByteString in a single-chunk lazy ByteString (O(1)). The Hackage documentation warns: **"Avoid converting back and forth between strict and lazy bytestrings"** as repeated conversions waste CPU and memory.
|
||||
|
||||
**Builder patterns enable efficient construction**. The ByteString.Builder monoid supports O(1) concatenation, assembling responses from multiple parts without intermediate allocations:
|
||||
|
||||
```haskell
|
||||
import Data.ByteString.Builder
|
||||
|
||||
buildHttpResponse :: Int -> [(ByteString, ByteString)]
|
||||
-> LazyByteString -> LazyByteString
|
||||
buildHttpResponse status headers body = toLazyByteString builder
|
||||
where
|
||||
builder = statusLine <> headerLines
|
||||
<> byteString "\r\n" <> lazyByteString body
|
||||
statusLine = byteString "HTTP/1.1 " <> intDec status
|
||||
<> byteString " OK\r\n"
|
||||
headerLines = mconcat
|
||||
[ byteString k <> byteString ": " <> byteString v <> byteString "\r\n"
|
||||
| (k, v) <- headers ]
|
||||
```
|
||||
|
||||
Warp discovered Builder too slow for hot paths like HTTP header composition, implementing custom memcpy()-based composers instead. For application-level code, Builder provides excellent performance while maintaining readability.
|
||||
|
||||
**Connection pooling prevents resource exhaustion**. The `resource-pool` library manages reusable connections efficiently. Key configuration parameters include stripe count (independent sub-pools reducing lock contention), resources per stripe (total capacity), and idle timeout (automatic cleanup).
|
||||
|
||||
```haskell
|
||||
import Data.Pool
|
||||
|
||||
createBackendPool :: HostName -> PortNumber -> IO (Pool Socket)
|
||||
createBackendPool host port = do
|
||||
capabilities <- getNumCapabilities
|
||||
newPool $
|
||||
defaultPoolConfig
|
||||
(connectBackend host port) -- Create function
|
||||
close -- Destroy function
|
||||
30.0 -- 30 sec idle timeout
|
||||
(10 * capabilities) -- 10 connections per core
|
||||
& setNumStripes (Just capabilities)
|
||||
```
|
||||
|
||||
Stripe count should match capabilities for optimal load distribution. The `withResource` function ensures exception-safe usage: if the action throws any exception, the resource gets destroyed rather than returned to the pool, preventing poisoned connections from circulating. For applications where backend connections may die unexpectedly, implement health checking before use.
|
||||
|
||||
Network I/O integration with ByteString achieves maximum efficiency using the `Network.Socket.ByteString` module. The `sendAll` function ensures complete transmission, looping until all bytes transmit. The `sendMany` function implements vectored I/O (scatter-gather), transmitting multiple ByteStrings in a single syscall—critical for sending HTTP headers and body efficiently.
|
||||
|
||||
## GHC compiler flags unlock native performance
|
||||
|
||||
Glasgow Haskell Compiler offers extensive optimization controls affecting runtime performance by orders of magnitude. Understanding flag interactions and profiling-driven tuning separates adequate from exceptional performance.
|
||||
|
||||
**Optimization levels provide base performance tiers**. The `-O` flag enables safe optimizations balancing compile time with runtime performance, typically achieving 5% better performance than the native code generator baseline. The `-O2` flag applies aggressive optimizations including spec-constr (recursive function specialization based on argument shapes) and liberate-case (unrolling recursive functions once in their RHS). While `-O2` significantly increases compile time, recent GHC versions show diminishing returns—it rarely produces substantially better code than `-O` for most programs.
|
||||
|
||||
Specific optimizations merit individual attention. **Strictness analysis** (`-fstrictness`, enabled by default with `-O`) determines which function arguments are strict, enabling call-by-value and unboxing. The worker/wrapper transformation (`-fworker-wrapper`) exploits this information by creating specialized worker functions with unboxed arguments. These optimizations fundamentally change evaluation strategy, eliminating thunk allocation in hot paths.
|
||||
|
||||
**Common subexpression elimination** (`-fcse`) eliminates redundant computations but can interfere with streaming libraries. The key issue: **full laziness** (`-ffull-laziness`) floats let-bindings outside lambdas to reduce repeated computation, but increases memory residency through additional sharing. For streaming applications using conduit or pipes, `-fno-full-laziness` may prevent space leaks caused by over-sharing.
|
||||
|
||||
Inlining controls determine function call overhead. The `-funfolding-use-threshold` flag (default 80) governs when functions inline at call sites—the "most useful knob" for controlling inlining according to GHC developers. Lower values reduce code size at performance cost, higher values increase inlining aggressiveness. Cross-module optimization requires `-fcross-module-specialise`, allowing INLINABLE functions to specialize across module boundaries.
|
||||
|
||||
**LLVM backend trades compilation speed for runtime performance**. Activated with `-fllvm`, it leverages LLVM's advanced optimization passes including partial redundancy elimination, sophisticated loop optimizations, and superior register allocation. Numeric-intensive code sees 10-30% improvements, with some cases showing 2x speedups. The `lens` library compiles 22% faster wall-clock time with proper parallelization using LLVM. However, LLVM requires external installation and roughly doubles compilation time compared to GHC's native code generator.
|
||||
|
||||
**Manual pragmas direct compiler optimization**. The INLINE pragma aggressively inlines functions by making their "cost" effectively zero, critical for functions that enable downstream optimizations through inlining. However, overuse causes code bloat. The INLINABLE pragma exports function unfoldings for cross-module optimization without forcing inlining—ideal for polymorphic library functions enabling specialization at call sites:
|
||||
|
||||
```haskell
|
||||
{-# INLINABLE genericSort #-}
|
||||
genericSort :: Ord a => [a] -> [a]
|
||||
genericSort = ... -- Will specialize for each type
|
||||
|
||||
{-# SPECIALIZE genericSort :: [Int] -> [Int] #-}
|
||||
-- Explicitly requests specialized version
|
||||
```
|
||||
|
||||
Runtime system tuning complements compile-time optimization. **Parallel GC configuration** balances throughput and latency. The `-N` flag sets capability count (typically number of cores), while `-qg1` restricts parallel GC to old generation only, improving cache locality for young generation collections. For parallel programs, consider `-qb` (disable load balancing) to reduce GC overhead.
|
||||
|
||||
Allocation area sizing (`-A`) critically impacts GC frequency. Default 4MB works well for sequential programs, but parallel applications benefit from 64MB or larger. The `-n` flag divides allocation area into chunks, enabling better parallel utilization: `-A64m -n4m` creates 16 chunks allowing cores allocating faster to grab more allocation area. This configuration particularly benefits programs with 8+ cores and high allocation rates.
|
||||
|
||||
Heap size management via `-H` (suggested heap) and `-M` (maximum heap) prevents memory exhaustion while allowing GC to optimize collection timing. Setting `-H2G` hints at expected heap size, allowing GC to size generations appropriately. Setting `-M4G` caps maximum heap, throwing exceptions when exceeded—essential for production servers preventing OOM kills.
|
||||
|
||||
**Production build configuration combines multiple techniques**:
|
||||
|
||||
```bash
|
||||
# CPU-intensive application
|
||||
ghc -O2 -fllvm -threaded -rtsopts -with-rtsopts=-N Main.hs
|
||||
|
||||
# Runtime execution
|
||||
./Main +RTS -N -A64m -n8m -qg1 -H2G -M4G -RTS
|
||||
```
|
||||
|
||||
For development, disable optimization (`-O0`) for fastest compilation, using `-O` or `-O2` only for performance testing. The cabal.project file provides package-level control:
|
||||
|
||||
```
|
||||
optimization: 2 -- Use -O2 for local packages
|
||||
|
||||
package myproxy
|
||||
ghc-options: -threaded -rtsopts -with-rtsopts=-N
|
||||
```
|
||||
|
||||
Never use `ghc-options: -O2` in .cabal files—use the `optimization` field instead to properly integrate with Cabal's build system.
|
||||
|
||||
## Benchmarking methodology validates optimization impact
|
||||
|
||||
Accurate performance measurement requires understanding tool capabilities, avoiding common pitfalls, and statistical rigor. Different tools serve different purposes: criterion for micro-benchmarks, wrk for HTTP/1.1 load testing, h2load for HTTP/2.
|
||||
|
||||
**wrk excels at HTTP/1.1 load generation**. Its multi-threaded architecture generates high request rates, with LuaJIT scripting enabling complex request patterns. Basic usage requires thread count (`-t`), connection count (`-c`), and duration (`-d`):
|
||||
|
||||
```bash
|
||||
wrk -t4 -c100 -d60s --latency http://localhost:8080/api/endpoint
|
||||
```
|
||||
|
||||
Thread count should typically match CPU cores on the load generator machine. Connection count should exceed thread count significantly—common ratios range from 10:1 to 100:1 depending on expected production concurrency. Duration minimum should be 30 seconds, with 60+ seconds preferred for stable statistics accounting for JIT warmup and cache effects.
|
||||
|
||||
Lua scripting enables realistic traffic patterns. The `request()` function executes per-request, enabling dynamic request generation:
|
||||
|
||||
```lua
|
||||
names = {"Alice", "Bob", "Charlie"}
|
||||
request = function()
|
||||
headers = {}
|
||||
headers["Content-Type"] = "application/json"
|
||||
body = '{"name": "' .. names[math.random(#names)] .. '"}'
|
||||
return wrk.format("POST", "/api/users", headers, body)
|
||||
end
|
||||
```
|
||||
|
||||
Interpreting results requires understanding latency distribution. The `--latency` flag provides percentile breakdown:
|
||||
|
||||
```
|
||||
Latency Distribution
|
||||
50% 635.91us
|
||||
75% 712.34us
|
||||
90% 1.04ms
|
||||
99% 2.87ms
|
||||
```
|
||||
|
||||
The 50th percentile (median) represents typical performance, while 99th percentile reveals tail latency crucial for user experience. **Standard deviation percentage above 90% indicates consistent performance**—lower values suggest high variance requiring investigation.
|
||||
|
||||
**h2load specializes in HTTP/2 testing**. Unlike wrk, h2load supports HTTP/2 multiplexing via the `-m` flag (max concurrent streams per client). This tests server HTTP/2 implementation efficiency:
|
||||
|
||||
```bash
|
||||
h2load -n100000 -c100 -m100 https://localhost:8443
|
||||
```
|
||||
|
||||
With 100 clients each maintaining 100 concurrent streams, the server handles 10,000 concurrent requests—testing multiplexing and priority handling. The output reports header compression statistics showing HPACK efficiency, typically 90%+ compression for repeated headers. Comparing HTTP/1.1 vs HTTP/2 requires running h2load with `--h1` flag:
|
||||
|
||||
```bash
|
||||
h2load -n50000 -c100 -m1 --h1 http://localhost:8080 # HTTP/1.1
|
||||
h2load -n50000 -c100 -m100 https://localhost:8443 # HTTP/2
|
||||
```
|
||||
|
||||
**criterion provides statistically rigorous micro-benchmarking**. Designed for Haskell-specific challenges like lazy evaluation, criterion runs benchmarks multiple times, applies linear regression to filter noise, and reports confidence intervals:
|
||||
|
||||
```haskell
|
||||
import Criterion.Main
|
||||
|
||||
main = defaultMain
|
||||
[ bgroup "parsing"
|
||||
[ bench "parseHeaders" $ nf parseHeaders sampleInput
|
||||
, bench "parseBody" $ nf parseBody sampleBody
|
||||
]
|
||||
]
|
||||
```
|
||||
|
||||
The critical distinction: `nf` (normal form) versus `whnf` (weak head normal form). For strict data like `Int` or `Bool`, `whnf` suffices. For structures like lists or ByteStrings where you want to ensure full evaluation, **use `nf` to avoid measuring only thunk creation**:
|
||||
|
||||
```haskell
|
||||
-- WRONG: Only evaluates list constructor
|
||||
bench "sum" $ whnf sum [1..1000000]
|
||||
|
||||
-- RIGHT: Forces complete evaluation
|
||||
bench "sum" $ nf sum [1..1000000]
|
||||
```
|
||||
|
||||
Environment setup prevents measurement artifacts from file I/O or initialization:
|
||||
|
||||
```haskell
|
||||
main = defaultMain
|
||||
[ env setupEnv $ \testData ->
|
||||
bgroup "processing"
|
||||
[ bench "process" $ nf processData testData ]
|
||||
]
|
||||
where setupEnv = BS.readFile "input.dat"
|
||||
```
|
||||
|
||||
**System preparation ensures reproducible results**. CPU frequency scaling causes variance—set CPU governor to performance mode:
|
||||
|
||||
```bash
|
||||
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
```
|
||||
|
||||
Background processes introduce noise. Stop unnecessary services before benchmarking. For network benchmarks, **use separate machines for load generator and server**—localhost testing eliminates network stack traversal, producing unrealistic results.
|
||||
|
||||
Warmup periods matter significantly. First runs encounter cold caches (CPU, disk), uninitialized JIT state, and fresh memory allocation. h2load provides explicit warmup via `--warm-up-time=5`, running 5 seconds before starting measurement. For wrk, run a short test first, then the main benchmark.
|
||||
|
||||
Statistical rigor requires multiple runs. Run each benchmark 3-5 times, report median performance. Criterion handles this automatically, but for wrk/h2load, script multiple executions:
|
||||
|
||||
```bash
|
||||
for i in {1..5}; do
|
||||
wrk -t4 -c100 -d60s http://localhost:8080 >> results-run-$i.txt
|
||||
done
|
||||
```
|
||||
|
||||
## Profiling reveals hidden performance bottlenecks
|
||||
|
||||
Systematic profiling identifies actual bottlenecks rather than assumed hotspots. GHC's profiling infrastructure spans CPU time, memory allocation, garbage collection, and heap composition.
|
||||
|
||||
**Cost-center profiling measures time and allocation**. Compiling with `-prof -fprof-late` instruments code with cost centers while minimizing optimization interference. The `fprof-late` flag inserts cost centers after optimization passes, reducing profiling overhead compared to traditional `-fprof-auto`:
|
||||
|
||||
```bash
|
||||
ghc -O2 -prof -fprof-late -rtsopts MyProgram.hs
|
||||
./MyProgram +RTS -p -RTS
|
||||
```
|
||||
|
||||
The resulting `.prof` file shows time and allocation percentages:
|
||||
|
||||
```
|
||||
COST CENTRE %time %alloc
|
||||
parseRequest 23.4 18.2
|
||||
routeMatching 12.7 8.1
|
||||
buildResponse 34.8 42.3
|
||||
```
|
||||
|
||||
Interpreting these results: `buildResponse` consumes most time (34.8%) and allocations (42.3%), making it the primary optimization target. The `entries` column reveals invocation count—high entry count with low per-call cost may indicate inappropriate inlining.
|
||||
|
||||
**Flame graphs visualize profiling data** effectively. The `ghc-prof-flamegraph` tool converts `.prof` files to interactive SVG visualizations showing call stacks hierarchically:
|
||||
|
||||
```bash
|
||||
ghc-prof-flamegraph MyProgram.prof
|
||||
# Generates MyProgram.svg
|
||||
|
||||
ghc-prof-flamegraph --alloc MyProgram.prof # Allocation flamegraph
|
||||
```
|
||||
|
||||
Flame graph width represents time/allocation percentage, height shows call stack depth. Wide flat sections indicate optimization opportunities. Clicking sections zooms into subtrees for detailed analysis.
|
||||
|
||||
**Heap profiling diagnoses memory issues**. Different profiling modes reveal distinct information. The `-hc` flag profiles by cost center (who allocated), `-hy` by type (what was allocated), `-hd` by constructor (specific data constructors), and `-hr` by retainer (what keeps objects alive):
|
||||
|
||||
```bash
|
||||
ghc -O2 -prof -fprof-auto -rtsopts -eventlog MyProgram.hs
|
||||
./MyProgram +RTS -hy -l -i0.1 -RTS
|
||||
eventlog2html MyProgram.eventlog
|
||||
```
|
||||
|
||||
The `-i0.1` flag samples every 0.1 seconds for detailed temporal resolution. Generated HTML provides interactive charts showing memory composition over time. Rising memory suggests space leaks—**look for THUNK accumulation indicating lazy evaluation building unevaluated expressions**.
|
||||
|
||||
**Info table profiling eliminates profiling overhead**. This newer technique requires no `-prof` compilation, instead using debug information from `-finfo-table-map`:
|
||||
|
||||
```bash
|
||||
ghc -O2 -finfo-table-map -fdistinct-constructor-tables -eventlog MyProgram.hs
|
||||
./MyProgram +RTS -hi -l -RTS
|
||||
eventlog2html MyProgram.eventlog
|
||||
```
|
||||
|
||||
This approach provides heap profiles without profiling's 20-100% runtime overhead. The detailed HTML report includes exact source locations for allocations, crucial for identifying leak sources. Constructor tables enable pinpointing which module and line created specific heap objects.
|
||||
|
||||
**Garbage collection statistics reveal GC pressure**. The `-s` flag outputs summary statistics after execution:
|
||||
|
||||
```
|
||||
MUT time 0.63s ( 0.64s elapsed)
|
||||
GC time 19.60s ( 19.62s elapsed)
|
||||
Total time 20.23s ( 20.26s elapsed)
|
||||
Productivity 3.1%
|
||||
```
|
||||
|
||||
Productivity below 80% indicates excessive GC overhead. The detailed breakdown shows:
|
||||
|
||||
```
|
||||
Gen 0: 3222 collections, parallel
|
||||
Gen 1: 10 collections, parallel
|
||||
Alloc rate: 1,823 bytes per MUT second
|
||||
```
|
||||
|
||||
High Gen 0 collections with large allocation rate suggests **increasing `-A` (allocation area)**. Frequent Gen 1 collections indicate insufficient heap size—try larger `-H` or `-M` values. High "bytes copied during GC" suggests living data exceeds allocation area, requiring larger nursery.
|
||||
|
||||
**EventLog and ThreadScope visualize parallel execution**. For threaded programs, eventlog captures detailed execution traces:
|
||||
|
||||
```bash
|
||||
ghc -O2 -threaded -eventlog -rtsopts MyProgram.hs
|
||||
./MyProgram +RTS -N4 -ls -RTS
|
||||
threadscope MyProgram.eventlog
|
||||
```
|
||||
|
||||
ThreadScope displays CPU activity across cores, spark creation/conversion (for parallel strategies), GC activity, and thread migration. Effective parallel programs show sustained CPU activity across all cores with minimal GC pauses. Gaps indicate load imbalance or excessive synchronization.
|
||||
|
||||
**Profiling workflow progresses systematically**:
|
||||
|
||||
1. **Baseline measurement** with `-O2 +RTS -s` establishes initial performance
|
||||
2. **Time profiling** with `-prof -fprof-late +RTS -p` identifies CPU hotspots
|
||||
3. **Memory profiling** with `-hd -l` and eventlog2html reveals allocation patterns
|
||||
4. **Detailed investigation** using info table profiling for exact source locations
|
||||
5. **Iterative optimization** applying fixes and re-profiling to verify improvements
|
||||
|
||||
Common patterns emerge from profiling. **Space leaks from lazy accumulation** manifest as rising THUNK count in heap profiles. Fix with strict foldl' and bang patterns. **CAF retention** appears as constant memory baseline—convert top-level values to functions accepting unit argument. **List fusion failures** show intermediate list allocation—switch to Vector with fusion or streaming libraries.
|
||||
|
||||
## Optimization checklist ensures systematic improvement
|
||||
|
||||
Successful optimization follows priority order: algorithms trump micro-optimizations, measure before optimizing, and validate improvements with profiling.
|
||||
|
||||
**Algorithmic improvements provide largest gains**. Changing from O(n²) to O(n log n) complexity dwarfs low-level optimizations. Before tuning GHC flags or adding strictness, evaluate data structures and algorithms. Replace lists with vectors for random access, Map with HashMap for integer keys, and sort algorithms with appropriate complexity for data characteristics.
|
||||
|
||||
**Compilation optimization checklist**:
|
||||
- [ ] Use `-O` or `-O2` for production builds
|
||||
- [ ] Add `-fllvm` for numeric-intensive code after benchmarking
|
||||
- [ ] Enable `-threaded` for concurrent programs
|
||||
- [ ] Include `-rtsopts` to allow runtime tuning
|
||||
- [ ] Set `-with-rtsopts=-N` for automatic parallelism
|
||||
- [ ] Use `optimization: 2` in cabal.project, not `ghc-options`
|
||||
|
||||
**Code-level optimization checklist**:
|
||||
- [ ] Add `INLINABLE` to polymorphic library exports
|
||||
- [ ] Add `SPECIALIZE` pragmas for frequently-used type instances
|
||||
- [ ] Use strict evaluation on hot path arguments (bang patterns)
|
||||
- [ ] Mark strict record fields with `!` or use `UNPACK` for small fields
|
||||
- [ ] Replace `foldl` with `foldl'` for strict accumulation
|
||||
- [ ] Use `ByteString` throughout, avoiding `String` in I/O paths
|
||||
- [ ] Prefer `Builder` for constructing `ByteString` responses
|
||||
|
||||
**Runtime tuning checklist**:
|
||||
- [ ] Set `-N` to number of CPU cores for parallel programs
|
||||
- [ ] Tune `-A` based on allocation rate (start with 64MB for parallel)
|
||||
- [ ] Use `-n` for chunk allocation on 8+ core systems
|
||||
- [ ] Set `-H` to hint expected heap size
|
||||
- [ ] Set `-M` to cap maximum memory usage
|
||||
- [ ] Monitor GC with `+RTS -s` to verify tuning effectiveness
|
||||
|
||||
**Zero-copy implementation checklist**:
|
||||
- [ ] Use `splice` package for socket-to-socket forwarding
|
||||
- [ ] Use `simple-sendfile` or Warp's `ResponseFile` for static content
|
||||
- [ ] Implement connection pooling with `resource-pool` (stripes = cores)
|
||||
- [ ] Configure appropriate idle timeouts (10-30 seconds)
|
||||
- [ ] Add health checking for long-lived backend connections
|
||||
- [ ] Use `sendAll` to ensure complete transmission
|
||||
- [ ] Enable `NoDelay` socket option to disable Nagle algorithm
|
||||
|
||||
**Benchmarking validation checklist**:
|
||||
- [ ] Set CPU governor to performance mode
|
||||
- [ ] Stop unnecessary background services
|
||||
- [ ] Use separate machines for load generator and server
|
||||
- [ ] Include 5-10 second warmup period
|
||||
- [ ] Run benchmarks for 60+ seconds duration
|
||||
- [ ] Execute 3-5 runs and report median
|
||||
- [ ] Document hardware, software versions, and configuration
|
||||
- [ ] Store benchmark results in version control
|
||||
|
||||
**Profiling workflow checklist**:
|
||||
- [ ] Establish baseline with `+RTS -s` statistics
|
||||
- [ ] Profile time with `-prof -fprof-late +RTS -p`
|
||||
- [ ] Generate flamegraphs for visual analysis
|
||||
- [ ] Profile memory with `-hd -l` and eventlog2html
|
||||
- [ ] Use info table profiling for exact source locations
|
||||
- [ ] Verify productivity \u003e 80% (not GC-bound)
|
||||
- [ ] Check allocation rate and GC frequency
|
||||
- [ ] Re-profile after each optimization to confirm improvement
|
||||
|
||||
**Common pitfalls to avoid**:
|
||||
- Don't optimize without profiling data
|
||||
- Don't use `ghc-options: -O2` in .cabal files
|
||||
- Don't over-inline (causes code bloat)
|
||||
- Don't use `whnf` when `nf` is needed in criterion
|
||||
- Don't benchmark on localhost (unrealistic network stack)
|
||||
- Don't forget warmup periods
|
||||
- Don't assume more `-A` is always better
|
||||
- Don't apply `-funbox-strict-fields` globally without testing
|
||||
|
||||
## Zero-copy implementation guide provides practical patterns
|
||||
|
||||
Implementing zero-copy proxying in Haskell combines FFI syscall bindings, existing library usage, and careful error handling. This section provides production-ready code patterns.
|
||||
|
||||
**Complete proxy server with connection pooling**:
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
module Main where
|
||||
|
||||
import Network.Socket hiding (recv, send)
|
||||
import Network.Socket.ByteString
|
||||
import Network.Socket.Splice
|
||||
import Data.Pool
|
||||
import Control.Concurrent
|
||||
import Control.Monad
|
||||
import Control.Exception
|
||||
import System.IO
|
||||
|
||||
data ProxyConfig = ProxyConfig
|
||||
{ listenPort :: PortNumber
|
||||
, targetHost :: HostName
|
||||
, targetPort :: PortNumber
|
||||
, poolStripes :: Int
|
||||
, poolPerStripe :: Int
|
||||
, poolIdleTime :: Double
|
||||
}
|
||||
|
||||
-- Create backend connection pool
|
||||
createBackendPool :: ProxyConfig -> IO (Pool Socket)
|
||||
createBackendPool config =
|
||||
newPool $
|
||||
defaultPoolConfig
|
||||
(connectBackend (targetHost config) (targetPort config))
|
||||
close
|
||||
(poolIdleTime config)
|
||||
(poolPerStripe config)
|
||||
& setNumStripes (Just $ poolStripes config)
|
||||
|
||||
connectBackend :: HostName -> PortNumber -> IO Socket
|
||||
connectBackend host port = do
|
||||
addr:_ <- getAddrInfo
|
||||
(Just defaultHints { addrSocketType = Stream })
|
||||
(Just host) (Just $ show port)
|
||||
sock <- socket (addrFamily addr) Stream defaultProtocol
|
||||
setSocketOption sock NoDelay 1
|
||||
setSocketOption sock ReuseAddr 1
|
||||
connect sock (addrAddress addr)
|
||||
return sock
|
||||
|
||||
-- Main proxy server
|
||||
runProxy :: ProxyConfig -> IO ()
|
||||
runProxy config = do
|
||||
pool <- createBackendPool config
|
||||
addr:_ <- getAddrInfo
|
||||
(Just defaultHints
|
||||
{ addrSocketType = Stream
|
||||
, addrFlags = [AI_PASSIVE] })
|
||||
Nothing (Just $ show $ listenPort config)
|
||||
|
||||
sock <- socket (addrFamily addr) Stream defaultProtocol
|
||||
setSocketOption sock ReuseAddr 1
|
||||
bind sock (addrAddress addr)
|
||||
listen sock 128
|
||||
|
||||
putStrLn $ "Proxy listening on port " ++ show (listenPort config)
|
||||
|
||||
forever $ do
|
||||
(client, clientAddr) <- accept sock
|
||||
forkIO $ handleClient pool client
|
||||
`finally` gracefulClose client 5000
|
||||
|
||||
-- Handle individual connection with zero-copy
|
||||
handleClient :: Pool Socket -> Socket -> IO ()
|
||||
handleClient pool client = do
|
||||
done <- newEmptyMVar
|
||||
|
||||
withResource pool $ \backend -> do
|
||||
-- Bidirectional zero-copy forwarding
|
||||
let chunkSize = 65536 -- 64KB chunks
|
||||
|
||||
forkIO $ do
|
||||
result <- try $ splice chunkSize (client, Nothing)
|
||||
(backend, Nothing)
|
||||
case result of
|
||||
Left (e :: SomeException) ->
|
||||
putStrLn $ "Client->Backend error: " ++ show e
|
||||
Right _ -> return ()
|
||||
putMVar done ()
|
||||
|
||||
result <- try $ splice chunkSize (backend, Nothing)
|
||||
(client, Nothing)
|
||||
case result of
|
||||
Left (e :: SomeException) ->
|
||||
putStrLn $ "Backend->Client error: " ++ show e
|
||||
Right _ -> return ()
|
||||
|
||||
takeMVar done
|
||||
|
||||
main :: IO ()
|
||||
main = do
|
||||
capabilities <- getNumCapabilities
|
||||
let config = ProxyConfig
|
||||
{ listenPort = 8080
|
||||
, targetHost = "backend.example.com"
|
||||
, targetPort = 8080
|
||||
, poolStripes = capabilities
|
||||
, poolPerStripe = 10
|
||||
, poolIdleTime = 30.0
|
||||
}
|
||||
runProxy config
|
||||
```
|
||||
|
||||
**Efficient HTTP response builder**:
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
import Data.ByteString.Builder
|
||||
import qualified Data.ByteString.Lazy as BL
|
||||
|
||||
buildHttpResponse :: Int -> [(ByteString, ByteString)]
|
||||
-> BL.ByteString -> BL.ByteString
|
||||
buildHttpResponse status headers body = toLazyByteString $
|
||||
mconcat
|
||||
[ byteString "HTTP/1.1 "
|
||||
, intDec status
|
||||
, byteString " "
|
||||
, statusText status
|
||||
, byteString "\r\n"
|
||||
, mconcat [ byteString k <> byteString ": "
|
||||
<> byteString v <> byteString "\r\n"
|
||||
| (k, v) <- headers ]
|
||||
, byteString "\r\n"
|
||||
, lazyByteString body
|
||||
]
|
||||
where
|
||||
statusText 200 = byteString "OK"
|
||||
statusText 404 = byteString "Not Found"
|
||||
statusText 500 = byteString "Internal Server Error"
|
||||
statusText _ = byteString "Unknown"
|
||||
```
|
||||
|
||||
**Zero-copy HTTP header parser**:
|
||||
|
||||
```haskell
|
||||
import qualified Data.ByteString as BS
|
||||
import qualified Data.ByteString.Char8 as BC
|
||||
import Data.Word
|
||||
|
||||
-- Parse request line without copying
|
||||
parseRequestLine :: ByteString -> Maybe (ByteString, ByteString, ByteString)
|
||||
parseRequestLine bs = do
|
||||
let (method, rest1) = BS.break (== space) bs
|
||||
guard (not $ BS.null rest1)
|
||||
|
||||
let rest2 = BS.drop 1 rest1
|
||||
(path, rest3) = BS.break (== space) rest2
|
||||
guard (not $ BS.null rest3)
|
||||
|
||||
let rest4 = BS.drop 1 rest3
|
||||
(version, _) = BS.break (== cr) rest4
|
||||
|
||||
return (method, path, version)
|
||||
where
|
||||
space = 32; cr = 13
|
||||
|
||||
-- Parse headers using splicing
|
||||
parseHeaders :: ByteString -> [(ByteString, ByteString)]
|
||||
parseHeaders = go . BC.lines
|
||||
where
|
||||
go [] = []
|
||||
go (line:rest)
|
||||
| BS.null line = []
|
||||
| otherwise =
|
||||
case BC.break (== ':') line of
|
||||
(key, value)
|
||||
| BS.null value -> go rest
|
||||
| otherwise ->
|
||||
let val = BS.dropWhile (== 32) (BS.drop 1 value)
|
||||
in (key, val) : go rest
|
||||
```
|
||||
|
||||
**Cabal configuration for production**:
|
||||
|
||||
```cabal
|
||||
-- myproxy.cabal
|
||||
name: myproxy
|
||||
version: 0.1.0.0
|
||||
build-type: Simple
|
||||
cabal-version: 2.0
|
||||
|
||||
executable myproxy
|
||||
main-is: Main.hs
|
||||
build-depends: base >= 4.14 && < 5
|
||||
, network >= 3.1
|
||||
, bytestring >= 0.11
|
||||
, resource-pool >= 0.3
|
||||
, splice >= 0.4
|
||||
ghc-options: -O2 -threaded -rtsopts -with-rtsopts=-N
|
||||
default-language: Haskell2010
|
||||
|
||||
-- For profiling builds
|
||||
-- cabal build --enable-profiling --ghc-options="-fprof-late"
|
||||
```
|
||||
|
||||
**Deployment script with optimal RTS settings**:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# deploy.sh
|
||||
|
||||
# Build optimized binary
|
||||
cabal build --enable-optimization=2
|
||||
|
||||
# Set CPU governor
|
||||
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
|
||||
# Run with optimized RTS settings
|
||||
./myproxy +RTS \
|
||||
-N `# Use all cores` \
|
||||
-A64m `# 64MB allocation area` \
|
||||
-n4m `# 4MB chunks` \
|
||||
-qg1 `# Parallel GC for old gen only` \
|
||||
-H2G `# Hint 2GB heap` \
|
||||
-M4G `# Cap at 4GB` \
|
||||
-I0 `# Disable idle GC` \
|
||||
-T `# Collect statistics` \
|
||||
-RTS
|
||||
```
|
||||
|
||||
This implementation guide provides production-ready patterns combining zero-copy techniques, efficient ByteString usage, connection pooling, and optimal compiler/runtime configuration for building high-performance proxies in Haskell.
|
||||
|
|
@ -1,259 +0,0 @@
|
|||
# Rate Limiting Algorithms and Distributed Systems for API Security
|
||||
|
||||
Modern API security demands sophisticated rate limiting to prevent abuse, ensure fair resource allocation, and maintain system stability under attack. Production systems at Cloudflare process 46 million requests per second with sub-100 microsecond detection latency, while Stripe's Redis-based implementation handles millions of requests monthly. The sliding window counter algorithm has emerged as the industry standard, achieving 94% accuracy with O(1) complexity and 16MB memory footprint per million users—a balance proven at billion-request scale.
|
||||
|
||||
This comprehensive technical guide covers algorithm selection, distributed implementation patterns, adaptive and ML-based approaches, and production-ready code examples from systems handling global-scale traffic. The research synthesizes implementations from GitHub, AWS, Stripe, Cloudflare, and academic foundations, providing decision frameworks for selecting algorithms, Redis schemas with atomic Lua scripts, and security patterns for defending against sophisticated attacks.
|
||||
|
||||
## Core algorithm comparison reveals critical tradeoffs
|
||||
|
||||
Rate limiting algorithms differ fundamentally in their approach to traffic management, with each optimized for specific use cases. The sliding window counter represents the convergence point of accuracy and performance that has driven its adoption across high-traffic production systems.
|
||||
|
||||
**Token bucket** dominates where burst capacity is essential. AWS API Gateway and Stripe both implement this approach, allowing clients to accumulate tokens at a fixed refill rate while permitting instant bursts up to bucket capacity. A bucket configured with 100 token capacity and 10 tokens/second refill rate allows an immediate burst of 100 requests followed by sustained throughput of 10 requests/second. The algorithm requires only 20 bytes per user (storing token count and last refill timestamp), achieving 500 nanosecond latency with 94% accuracy. Implementation involves simple arithmetic: elapsed time multiplied by refill rate determines new tokens, with consumption checked against available balance. The primary weakness emerges at boundaries where clients can game the system by timing requests to burst periods, and greedy clients may monopolize resources by constantly draining tokens.
|
||||
|
||||
**Leaky bucket** enforces perfectly smooth output rates, making it ideal for protecting backend systems requiring constant load. NGINX implements this as its default algorithm, processing requests from a FIFO queue at fixed intervals. Unlike token bucket's variable output, leaky bucket guarantees predictable backend load—critical for VoIP systems, real-time streaming, and network traffic shaping. The algorithm maintains a queue of pending requests that "leak" at constant rate, introducing ~5 microsecond latency with greater than 99% accuracy. Memory consumption scales with queue size at approximately 800MB per million users with 100-request capacity, significantly higher than alternatives. The fatal flaw is inability to handle legitimate bursts: a mobile app syncing after extended offline period faces request starvation despite low average rate. Shopify's GraphQL API implements a sophisticated points-based variant where query complexity determines "marble" cost, with buckets leaking at 50-500 points/second depending on subscription tier.
|
||||
|
||||
**Sliding window log** achieves the highest accuracy of any algorithm at 99.997% based on Cloudflare's analysis of 400 million requests, with zero false positives and only 3 false negatives (all under 15% above threshold). The algorithm maintains a sorted set of every request timestamp, removing expired entries and counting remaining requests on each check. This perfect accuracy comes at severe cost: O(n) time complexity for processing, 800MB to 8GB memory per million users depending on traffic volume, and 50 microsecond latency. Implementation requires careful memory management with aggressive cleanup to prevent unbounded growth. The algorithm suits low-volume APIs where precision matters more than scalability, or regulatory environments requiring perfect audit trails.
|
||||
|
||||
**Sliding window counter** has emerged as the recommended algorithm for production systems, used by Cloudflare to handle billions of requests daily. This hybrid approach maintains counters for current and previous time windows, calculating a weighted estimate: `count = prev_count × (1 - elapsed%) + current_count`. With only 16 bytes per user, O(1) complexity, and 1 microsecond latency, the algorithm achieves 94% accuracy—a 6% average variance acceptable for nearly all use cases. The boundary approximation creates edge cases: a client making 94 requests at 00:00:59 and 94 at 00:01:01 might pass when the true sliding window would reject. However, this 6% error rate represents an optimal engineering tradeoff: sliding window log's 99.997% accuracy costs 50x more memory and 50x higher latency for marginal improvement.
|
||||
|
||||
**Fixed window** serves only non-critical scenarios due to catastrophic burst problems. Time divided into fixed intervals with counters resetting at boundaries creates the infamous "double rate" vulnerability: 10 requests at 00:00:59 plus 10 at 00:01:01 yields 20 requests in 2 seconds despite a 10/minute limit. The algorithm offers 100 nanosecond latency and 12MB per million users, but 50-200% accuracy variance disqualifies it for production APIs. Use cases are limited to internal development, prototyping, or highly tolerant systems where boundary bursts pose no risk.
|
||||
|
||||
### Algorithm selection matrix
|
||||
|
||||
| Algorithm | Time | Space/User | Latency | Accuracy | Memory (1M users) | Burst Support | Best Use Case |
|
||||
|-----------|------|------------|---------|----------|-------------------|---------------|---------------|
|
||||
| **Fixed Window** | O(1) | 12B | 100ns | 50-200% | 12 MB | Poor | Development only |
|
||||
| **Token Bucket** | O(1) | 20B | 500ns | ~94% | 20 MB | Excellent | APIs with variable traffic |
|
||||
| **Sliding Window Counter** | O(1) | 16B | 1μs | ~94% | 16 MB | Good | **Recommended for production** |
|
||||
| **Sliding Window Log** | O(n) | 800-8000B | 50μs | 99.997% | 800MB-8GB | Good | High-precision/low-volume |
|
||||
| **Leaky Bucket** | O(1) | 800B | 5μs | >99% | 800 MB | None | Constant rate required |
|
||||
|
||||
The decision framework is straightforward: **choose sliding window counter for 95% of production systems**. Its performance characteristics match modern distributed architectures while accuracy suffices for security and fairness. Select token bucket when burst handling is critical and you're following industry standards (AWS, Stripe patterns). Choose leaky bucket only when backend systems absolutely require constant rate input (NGINX integration, legacy systems). Reserve sliding window log for regulatory compliance, high-security environments, or low-traffic APIs where memory cost is irrelevant.
|
||||
|
||||
## Distributed rate limiting demands atomic operations
|
||||
|
||||
Rate limiting in distributed systems introduces race conditions, consistency challenges, and synchronization overhead that can undermine algorithm guarantees. Redis-based implementations with atomic Lua scripts solve these problems while maintaining sub-millisecond latency at scale.
|
||||
|
||||
**Redis sorted sets** provide the foundation for sliding window log implementation. The atomic Lua script handles three operations in a single round-trip: remove expired timestamps with `ZREMRANGEBYSCORE`, count remaining entries with `ZCARD`, and conditionally add new timestamp with `ZADD`. The key design pattern uses the timestamp as both member and score, enabling efficient range queries. TTL set via `EXPIRE` prevents memory leaks from abandoned keys. This pattern scales to millions of users with proper key naming: `rate_limit:{user_id}:{endpoint}` enables per-user, per-endpoint limits with independent quotas.
|
||||
|
||||
```lua
|
||||
-- sliding_window.lua
|
||||
local key = KEYS[1]
|
||||
local limit = tonumber(ARGV[1])
|
||||
local window = tonumber(ARGV[2])
|
||||
local current_time = tonumber(ARGV[3])
|
||||
|
||||
redis.call('ZREMRANGEBYSCORE', key, '-inf', current_time - window)
|
||||
local count = redis.call('ZCARD', key)
|
||||
|
||||
if count < limit then
|
||||
redis.call('ZADD', key, current_time, current_time)
|
||||
redis.call('EXPIRE', key, window)
|
||||
return {1, limit - count - 1}
|
||||
else
|
||||
return {0, 0}
|
||||
end
|
||||
```
|
||||
|
||||
**Token bucket in Redis** uses hash structures to store mutable state atomically. The hash contains `tokens` (float) and `last_refill` (timestamp) fields updated together. The refill algorithm calculates elapsed time since last refill, computes new tokens as `min(capacity, current + elapsed × rate)`, and conditionally decrements if sufficient tokens exist. The pattern avoids race conditions by performing all calculations within the Lua script's atomic context. TTL set to 3600 seconds (one hour) auto-expires inactive users while allowing legitimate users to maintain state across requests.
|
||||
|
||||
```lua
|
||||
-- token_bucket.lua
|
||||
local key = KEYS[1]
|
||||
local capacity = tonumber(ARGV[1])
|
||||
local refill_rate = tonumber(ARGV[2])
|
||||
local current_time = tonumber(ARGV[3])
|
||||
local requested = tonumber(ARGV[4]) or 1
|
||||
|
||||
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
|
||||
local tokens = tonumber(bucket[1]) or capacity
|
||||
local last_refill = tonumber(bucket[2]) or current_time
|
||||
|
||||
local elapsed = current_time - last_refill
|
||||
tokens = math.min(capacity, tokens + elapsed * refill_rate)
|
||||
|
||||
if tokens >= requested then
|
||||
tokens = tokens - requested
|
||||
redis.call('HMSET', key, 'tokens', tokens, 'last_refill', current_time)
|
||||
redis.call('EXPIRE', key, 3600)
|
||||
return {1, math.floor(tokens)}
|
||||
else
|
||||
return {0, math.floor(tokens)}
|
||||
end
|
||||
```
|
||||
|
||||
**Sliding window counter** achieves optimal performance by storing only two integer counters rather than full request logs. The implementation requires two keys: `{user}:{current_minute}` and `{user}:{previous_minute}`. The weighted calculation `previous_count × (1 - elapsed_percent) + current_count` approximates the true sliding window. The critical insight: this approximation delivers 94% accuracy while consuming 94% less memory than sorted sets. Production deployments should handle key rotation carefully, potentially storing both keys in a hash structure to ensure atomic updates across window boundaries.
|
||||
|
||||
**Consistent hashing** distributes rate limit state across Redis cluster nodes while minimizing key redistribution during scaling. Cloudflare's implementation uses Twemproxy with consistent hashing to shard rate limit data across memcache clusters. When adding nodes, only K/n keys redistribute (K = total keys, n = nodes), preserving most rate limit counters. This pattern enables horizontal scaling without reset-all disruption. The tradeoff: distributed counts become "never completely accurate" as AWS documentation states—network latency between nodes introduces timing windows where concurrent requests may both succeed despite exceeding limits. Production systems accept this 1-3% variance as cost of distribution.
|
||||
|
||||
**High availability patterns** prevent rate limiter failures from cascading. Redis Sentinel provides automatic failover with 3-5 sentinel nodes monitoring master health. Upon master failure, sentinels promote a replica within seconds, with applications reconnecting automatically via sentinel-aware clients. Redis Cluster offers alternative architecture with hash slots sharded across nodes, providing both HA and horizontal scaling. The critical decision: **fail open or fail closed** during Redis outage. Stripe fails open (allows requests) to prioritize availability; financial systems often fail closed (deny requests) for security. Implementing circuit breakers wraps Redis calls, tracking failure rates and automatically entering degraded mode when thresholds exceed limits.
|
||||
|
||||
```javascript
|
||||
async function checkRateLimit(userId) {
|
||||
try {
|
||||
return await redis.eval(luaScript, [key], [limit, window, Date.now()]);
|
||||
} catch (error) {
|
||||
logger.warn('Rate limiter degraded', { error, userId });
|
||||
metrics.increment('rate_limiter.errors');
|
||||
return { allowed: true, degraded: true }; // Fail open
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Schema design best practices** center on key naming conventions that enable efficient queries and avoid collisions. The pattern `{prefix}:{identifier}:{scope}:{timestamp}` provides hierarchy: `rate_limit:api:user:12345:endpoint:/api/data:window:1672531200`. This structure supports querying by user, endpoint, or time window. TTL strategies should align with window duration: set expiry to 2× window duration to prevent premature deletion during edge cases. For token bucket, longer TTL (1 hour) maintains state for intermittent users while auto-expiring inactive accounts. Memory optimization: use Redis hashes for small objects (under 100 fields) as they consume less memory than separate keys due to ziplist encoding.
|
||||
|
||||
**Alternative distributed stores** offer different tradeoffs. Memcached provides simpler protocol with potentially lower latency but lacks Lua scripting, forcing less efficient read-modify-write patterns. Hazelcast offers in-process data grids eliminating network latency entirely, ideal for rate limiting within microservices. Etcd suits systems already using it for configuration, though write throughput lags Redis. The verdict: **Redis dominates production rate limiting** due to Lua atomicity, proven scale (Cloudflare, Stripe, GitHub), and operational maturity. Consider alternatives only when architectural constraints prevent Redis adoption.
|
||||
|
||||
## Adaptive rate limiting responds to real-time conditions
|
||||
|
||||
Static rate limits fail when traffic patterns vary or system capacity fluctuates. Adaptive algorithms adjust limits dynamically based on server health metrics, user reputation, and traffic analysis, achieving optimal throughput while preventing overload.
|
||||
|
||||
**Netflix's adaptive concurrency limits** apply TCP congestion control principles to API rate limiting. The algorithm calculates `gradient = RTT_no_load / RTT_actual`, where gradient of 1 indicates no queuing delay, while values less than 1 signal congestion. The formula `newLimit = currentLimit × gradient + sqrt(currentLimit)` adjusts concurrency dynamically, with square root queue size enabling fast growth at low limits while providing stability at scale. Production results show convergence within seconds to optimal concurrency, near 100% retry success rate, and elimination of manual tuning. The approach prevents cascading failures by automatically backing off when backend latency increases, then gradually restoring capacity as performance recovers.
|
||||
|
||||
**AIMD (Additive Increase, Multiplicative Decrease)** provides simpler alternative inspired by TCP congestion algorithms. During normal operation, gradually increase rate limits (e.g., +10 requests/minute every 5 minutes). Upon detecting congestion—CPU above 80%, error rate exceeding 5%, or P99 latency doubling—multiplicatively decrease limits by 50%. This asymmetric approach provides stability: slow increases prevent oscillation while rapid decreases protect against overload. Implementation tracks moving averages of key metrics with circuit breaker pattern triggering limit adjustments.
|
||||
|
||||
**Server load monitoring** drives adjustments based on real-time capacity. CPU utilization above 80% triggers linear reduction of rate limits: `adjusted_limit = base_limit × (1 - cpu_load)`. Memory pressure follows similar pattern with heap usage monitoring. Response latency provides early warning: P99 latency exceeding baseline by 2× suggests saturation before resource metrics spike. Queue depth offers immediate signal: pending request count above threshold indicates insufficient capacity. Bitbucket Data Center combines physical memory evaluation (at startup) with CPU load monitoring (periodic) to dynamically allocate operation tickets, with formula `safe_bound = (total_RAM - JVM_heap - overhead) / avg_operation_memory` determining memory-constrained limits.
|
||||
|
||||
**User reputation scoring** enables trust-based differentiation. IP reputation combines multiple signals: threat score (0-100 probability of malicious intent based on historical attacks), VPN/proxy detection (likelihood of anonymization), blocklist presence (checking 100+ databases), and behavioral patterns (request rate consistency, navigation flow, session characteristics). High-reputation users receive elevated limits while suspicious actors face restrictions. GitHub demonstrates tiered approach: unauthenticated requests limited to 60/hour, authenticated to 5,000/hour, enterprise to 15,000/hour. Stripe adjusts limits per customer tier with automatic promotion as usage grows, balancing security and user experience.
|
||||
|
||||
**Cloudflare's volumetric abuse detection** uses unsupervised learning to establish per-endpoint baselines automatically. The system analyzes P99, P90, and P50 request rate distributions over time, identifying anomalies that indicate attacks rather than legitimate traffic surges. Per-session limits (via authorization tokens rather than IP addresses) minimize false positives from CGNAT shared IPs. The approach adapts to traffic changes automatically—distinguishing viral marketing campaigns from DDoS attacks by analyzing request patterns across endpoints. Integration with WAF machine learning scores, bot management scores, and TLS fingerprinting provides multi-dimensional threat assessment.
|
||||
|
||||
**Automatic scaling strategies** adjust both rate limits and infrastructure capacity. Kubernetes-based deployments monitor cluster metrics (CPU, memory per pod) via Prometheus at 10-second intervals, feeding decisions to adaptive policy engines. When average CPU exceeds 70%, the system both reduces per-user rate limits by 20% and triggers horizontal pod autoscaling. This dual response—reducing demand while increasing supply—prevents cascading failures during traffic spikes. AWS Shield Advanced employs 24-hour to 30-day baseline learning periods, automatically creating WAF rules when traffic exceeds learned patterns, with mitigation rules deployed in count or block mode based on confidence levels.
|
||||
|
||||
## Machine learning detects sophisticated attacks
|
||||
|
||||
Traditional rate limiting fails against coordinated distributed attacks, low-and-slow techniques, and adversarial evasion. Machine learning models trained on billions of requests identify attack patterns invisible to rule-based systems, achieving detection rates above 99% while maintaining sub-millisecond inference latency.
|
||||
|
||||
**Cloudflare's production ML pipeline** processes 46+ million HTTP requests per second in real-time, using CatBoost gradient boosting models with sub-50 microsecond inference per model. The architecture runs multiple models in shadow mode (logging only) with one active model influencing firewall decisions, enabling safe validation before promotion. Training data comes from trillions of requests across 26+ million internet properties, with high-confidence labels generated by heuristics engine (classifying ~15% of traffic) and customer-reported incidents. CatBoost was selected for native categorical feature support, reduced overfitting through novel gradient boosting scheme, and fast inference via C and Rust APIs. The bot score output ranges 0-100 (0=bot, 100=human), integrating with firewall rules for action decisions (allow, challenge, block).
|
||||
|
||||
**Feature engineering** determines model effectiveness. Network-level features include IP geolocation, ASN (Autonomous System Number), reputation scores, and JA3 TLS fingerprints capturing client SSL/TLS implementation. Header analysis examines User-Agent parsing and validation, Accept-Language patterns, header order and capitalization, and presence of custom headers. Inter-request features from Cloudflare's Gagarin platform track request rate over time windows, session duration and consistency, navigation patterns with referrer chains, and time-between-requests distributions. Behavioral features capture mouse movements, click patterns, keystroke dynamics, and maximum sustained click rate via sliding window analysis. Research on Twitter bot detection identified 49 profile features spanning message-based metrics (URL count, retweet frequency), part-of-speech patterns, special character usage, word frequency distributions, and sentiment analysis.
|
||||
|
||||
**Akamai's Behavioral DDoS Engine** combines multiple AI components into integrated defense. The baseline generator processes clean data over 2-week periods to create traffic profiles. The detection engine maintains multidimensional traffic views leveraging baseline intelligence. The mitigation engine identifies attackers using dimension combinations (IP + User-Agent + geolocation). Platform DDoS Intelligence provides threat signals from historical attack data. The baseline validator employs AI-based tuning, evaluating hundreds of attacks monthly to reduce false positives. Protection levels adjust sensitivity: strict mode responds rapidly to slight anomalies (high-security environments), moderate balances protection versus false positives (recommended), while conservative tolerates substantial deviations. Production case studies show 99.95% detection rate across 1.4 billion requests from 7,000+ IPs and 99.50% detection across 185 million requests from 5,000+ IPs.
|
||||
|
||||
**Anomaly detection algorithms** identify novel attack patterns without labeled training data. Isolation Forest effectively detects outliers in high-dimensional feature spaces by measuring how quickly observations can be isolated via random partitioning. K-Nearest Neighbors Conformal Anomaly Detection uses Mahalanobis distance and non-conformity measures (sum of distances to k-nearest neighbors) for contextual anomaly detection. Relative entropy (Kullback-Leibler divergence) compares current request distributions to baseline distributions via hypothesis testing. Deep learning approaches include LSTM recurrent neural networks for temporal pattern recognition in request sequences and autoencoders learning normal traffic patterns in unsupervised fashion, with reconstruction error indicating anomalies.
|
||||
|
||||
**Real-time versus batch prediction** presents fundamental architectural tradeoff. Real-time inference generates predictions on-demand at request time with sub-millisecond to low-millisecond latency requirements. Cloudflare's edge deployment runs models on every edge server with <100 microsecond overhead, using CatBoost via LuaJIT FFI with no network calls. The approach handles single observation processing with continuous availability at millions of requests/second throughput. Batch prediction processes large datasets offline on scheduled intervals (hourly, daily, weekly), enabling complex model architectures and extensive feature computation via big data frameworks (Spark, Hadoop) with cost-optimized compute. Use cases include historical traffic analysis, model retraining data generation, and reputation score updates. The hybrid approach: real-time inference for immediate blocking decisions, batch processing for reputation updates and model retraining.
|
||||
|
||||
**Integration with traditional rate limiting** creates layered defense. Layer 1 employs traditional algorithms (token bucket, leaky bucket) providing fast, deterministic response in <1ms. Layer 2 adds heuristics engine with simple rule-based detection executing in ~20 microseconds, classifying ~15% of traffic. Layer 3 incorporates ML models with multi-feature analysis and ~50 microsecond inference handling sophisticated attacks. Layer 4 applies behavioral analysis with unsupervised anomaly detection and long-term pattern recognition. Layer 5 reserves human verification (CAPTCHA, JavaScript challenge) as fallback for uncertain cases. Score combination methods include weighted ensemble (`FinalScore = w1×Heuristic + w2×ML + w3×Behavior`), decision trees with confidence-based routing, and uncertainty thresholds triggering additional verification.
|
||||
|
||||
**AWS Shield Advanced demonstrates production integration** with automatic ML-based mitigation. The system monitors traffic baselines over 24 hours to 30 days, detecting deviations using ML models combined with heuristics. Upon detection, Shield automatically creates WAF rules deployed in Shield-managed rule group (consuming 150 WCU capacity), with customers choosing count or block mode. Rules automatically remove when attacks subside, providing adaptive defense without manual intervention. Integration with CloudWatch enables alerting and 24/7 DRT (DDoS Response Team) support for Enterprise customers.
|
||||
|
||||
## Proof-of-work and advanced strategies add defense layers
|
||||
|
||||
Rate limiting alone cannot stop determined attackers with distributed resources. Proof-of-work challenges, geographic filtering, hierarchical limits, and context-aware strategies create defense-in-depth against sophisticated threats.
|
||||
|
||||
**Cloudflare Turnstile** represents modern CAPTCHA alternative, running non-interactive JavaScript challenges including proof-of-work, proof-of-space, Web API probing, and browser quirk detection. Three widget modes provide flexibility: managed mode (adaptive checkbox), non-interactive (visible but no interaction), and invisible (completely hidden). Tokens expire after 300 seconds with single-use only validation, requiring server-side verification via Siteverify API. Implementation requires simple HTML div with sitekey and included JavaScript. Security considerations demand never exposing secret keys client-side, rotating keys regularly, restricting hostnames to controlled domains, and monitoring via Turnstile Analytics. Production use cases include login form protection, API endpoint protection via WAF integration, and form submission validation.
|
||||
|
||||
**Computational puzzles** create asymmetric costs: hard to solve but easy to verify. Hash-based puzzles require finding nonce such that `sha256(challenge + nonce)` has N leading zeros, with difficulty adjusted by required zero count. Client-side implementation solves transparently without user awareness, while server validates solution in microseconds. Performance metrics show 85% false positive reduction versus CAPTCHA-only, 95% completion rates (versus 70% for visual challenges), and 80% reduction in successful bot attacks. The approach suits account registration (medium difficulty), login verification (low difficulty), and anti-scraping measures (variable difficulty), with difficulty dynamically adjusted based on threat level.
|
||||
|
||||
**Geographic-based rate limiting** applies region-specific limits optimizing for threat landscape and resource costs. MaxMind GeoIP databases provide 99% country accuracy, 75% city accuracy via IP-based geographic determination. Tiered regional limits example: US-OR (Oregon) receives 1000 requests/minute as high-trust region, rest of US gets 500 requests/minute, while default regions limited to 100 requests/minute. AWS WAF geo match implementation supports country codes with per-region rate limits. Use cases include fighting spam from specific regions, prioritizing resources for key markets, compliance with regional regulations, and cost optimization. Critical security consideration: **never rely solely on geography** as VPNs easily spoof location. Layer geographic limits with IP reputation, behavioral analysis, and proof-of-work challenges.
|
||||
|
||||
**Hierarchical rate limiting** implements multiple cascading layers preventing resource starvation. Four-layer architecture: Layer 1 global infrastructure limits (100,000 requests/second across entire infrastructure prevents system overload), Layer 2 category/service limits (authentication 10,000/minute, data API 50,000/minute separates traffic classes), Layer 3 user/client limits (1,000 requests/hour per user ensures fairness), Layer 4 endpoint-specific limits (POST /expensive 10/minute protects costly operations). Redis implementation checks all layers hierarchically, incrementing all counters only when all checks pass, providing atomic multi-level enforcement. Slack's notification system demonstrates pattern: global limit 100 notifications/30 minutes, with category limits for errors (10), warnings (10), info (10) that sum above global limit, demonstrating how global acts as final constraint.
|
||||
|
||||
**HTTP method-specific rate limiting** recognizes different resource impacts by method. GET requests typically allow 100-1000/minute as read-only and less expensive. POST requests restricted to 10-50/minute for resource creation with higher cost. PUT/PATCH receives 20-100/minute for updates with moderate cost. DELETE most restrictive at 5-20/minute given security sensitivity. NGINX implementation maps request methods to different limit zones, applying write operation limits (10/minute) to POST/PUT/DELETE while read operations (100/minute) apply to GET. Login endpoints warrant special treatment: `POST /api/login` limited to 5 attempts/minute per IP and 10 attempts/minute per username, with exceeded limits triggering security alerts and incrementing threat scores.
|
||||
|
||||
**Burst handling techniques** separate algorithms' core differentiator. Token bucket allows bursts up to bucket capacity: 100 token capacity with 10 tokens/second refill permits 100 request instant burst followed by sustained 10/second. Configuration flexibility: capacity determines maximum burst size, refill rate sets sustained throughput, tokens per request enables weighted costs. Leaky bucket smooths bursts into constant output rate, processing requests from queue at fixed intervals. The bucket accepts bursts into queue (up to capacity), but backend receives perfectly steady stream. Comparison reveals token bucket best for bursty traffic and variable traffic patterns, while leaky bucket optimal when backend requires constant rate (VoIP, streaming, real-time systems). Combined approach uses local token bucket (capacity 100, refill 50/second) absorbing local bursts with global leaky bucket (capacity 1000, leak 100/second) smoothing global traffic.
|
||||
|
||||
**Time-based adjustments** adapt to known traffic patterns. Peak hours handling (8am-5pm business hours) allows higher limits (100 requests/minute) during expected high traffic, with off-peak hours (6pm-7am) reduced limits (50 requests/minute). Adaptive strategies monitor server load, reducing limits when CPU exceeds 80% regardless of time. Maintenance windows employ severe restrictions (10 requests/minute) with whitelist for admin IPs and monitoring systems during scheduled downtime. Critical implementation detail: **prevent thundering herd at window boundaries** by adding jitter to retry timing (±20% randomization) so clients don't all retry simultaneously at exact boundary.
|
||||
|
||||
**Strategy selection** depends on traffic characteristics and security requirements. Per-IP rate limiting suits anonymous traffic, DDoS prevention, and brute force mitigation as first defense layer, but fails against shared IPs (NAT, proxies) and VPN bypass. Per-user limiting enables precise control for authenticated users, supports subscription tiers, and provides better UX, requiring authentication mechanism. Per-endpoint limiting protects resource-intensive operations with different costs: POST /login limited to 5/minute, GET /health unlimited, POST /api/data 100/hour. **Recommended approach combines all three**: global infrastructure limit (10,000/minute), per-IP limit (100/minute), per-user tier limits (1,000/hour free, 10,000/hour premium), and per-endpoint limits for sensitive operations.
|
||||
|
||||
## Real-world implementations provide production patterns
|
||||
|
||||
Major platforms have converged on proven patterns through years of evolution handling billions of requests. Their implementations reveal practical tradeoffs between theoretical purity and operational reality.
|
||||
|
||||
**GitHub API** implements token bucket with sophisticated point-based secondary limits. Primary limits: unauthenticated 60/hour per IP, authenticated 5,000/hour, Enterprise Cloud 15,000/hour for GitHub Apps. Secondary limits prevent abuse: max 100 concurrent requests, 900 points/minute for REST (2,000 for GraphQL), 90 seconds CPU time per 60 seconds real time, 80 content-creating requests/minute. Point costs vary by operation: GET/HEAD/OPTIONS cost 1 point, POST/PATCH/PUT/DELETE cost 5 points, GraphQL mutations 5 points. Headers returned include `x-ratelimit-limit`, `x-ratelimit-remaining`, `x-ratelimit-used`, `x-ratelimit-reset` (Unix epoch), and `x-ratelimit-resource`. Error responses use 403 or 429 status with `x-ratelimit-remaining` at 0. Best practices include conditional requests (ETags, If-None-Match), response caching, and GraphQL to reduce calls.
|
||||
|
||||
**Stripe API** employs Redis-based token bucket with four limiter types: request rate limiter (100/second live mode, 25/second sandbox), concurrent request limiter, fleet usage load shedder (critical vs non-critical requests), and worker utilization load shedder. Resource-specific limits include 1,000 PaymentIntent updates/hour per intent, Files API 20 read + 20 write/second, Search API 20 read/second, meter events 1,000 calls/second. The `Stripe-Rate-Limited-Reason` header indicates which limit triggered (global-concurrency, global-rate, endpoint-concurrency, endpoint-rate, resource-specific). Engineering blog reveals Redis provides low-latency distributed state, with exponential backoff recommended (randomization prevents thundering herd), and client-side token bucket for sophisticated applications. The system "constantly triggered," rejecting millions of test mode requests monthly, demonstrating production hardening.
|
||||
|
||||
**AWS API Gateway** uses token bucket with multi-level throttling: account-level default 10,000 requests/second per region with 5,000 burst capacity (lower regions 2,500 RPS with 1,250 burst). Throttling order: per-client → per-method → per-stage → account-level → regional. Usage plans enable custom limits per API key/client. Status code 429 returned with no specific rate limit headers by default, relying on CloudWatch metrics for monitoring. Documentation acknowledges distributed architecture means rate limiting "never completely accurate," with brief request acceptance after quota reached acceptable. Token bucket refills at rate limit with capacity equal to burst limit, allowing burst traffic followed by sustained throughput.
|
||||
|
||||
**Cloudflare Advanced Rate Limiting** supports counting by IP address, country, ASN, headers (custom, User-Agent), cookies, query parameters, session IDs, JA3 fingerprints (TLS client fingerprinting), bot scores, and request/response body fields. Dynamic fields include WAF machine learning scores, bot management scores, response status codes, and JSON body values (GraphQL operations). Use cases: count by session ID for authenticated APIs, track suspicious login patterns (failed 401/403 responses), rate limit GraphQL mutations via body inspection, separate counting from mitigation expressions. Integration with Bot Management provides scores consumed by firewall rules: `if (cf.bot_management.score < 30 and http.request.uri.path eq "/login") { action: challenge }`. Actions available include log, bypass, allow, challenge (CAPTCHA), JS challenge (browser validation), and block.
|
||||
|
||||
**Shopify API** implements leaky bucket with calculated query cost for GraphQL. REST Admin API limits: standard plan 40 requests bucket with 2/second leak, Shopify Plus 400 requests bucket with 20/second leak, private apps can request up to 200/second. GraphQL points system: scalar field 0 points, object field 1 point, connection (first N) costs N points, mutation 10 points default. Standard plan 50 points/second, Advanced 100 points/second, Plus up to 500 points/second, with 1,000 point bucket capacity. Header `X-Shopify-Shop-Api-Call-Limit` shows "current/max" (e.g., "32/40"). The leaky bucket metaphor: bucket holds "marbles" (requests) that leak at constant rate, with each REST request 1 marble and GraphQL requests variable marbles based on complexity.
|
||||
|
||||
**Performance benchmarks** reveal production characteristics. Stripe's Redis-based system handles millions of requests monthly with sub-millisecond latency. Cloudflare's distributed rate limiting across global edge network uses Twemproxy cluster with consistent hashing, processing 46+ million requests/second. AWS acknowledges variance between configured and actual limits depending on request volume, backend latency, and distributed gateway architecture. Algorithm throughput comparison: Fixed Window highest (minimal overhead), Token Bucket very high, Sliding Window Counter high, Leaky Bucket medium-high, Sliding Window Log medium. Accuracy ranking: Sliding Window Log 99.997%, Leaky Bucket >99%, Sliding Window Counter ~94%, Token Bucket ~94%, Fixed Window lowest (boundary issues). Memory efficiency: Fixed Window/Token Bucket/Sliding Window Counter all O(1) at 12-20MB per million users, Leaky Bucket O(n) at 800MB, Sliding Window Log O(n) at 800MB-8GB.
|
||||
|
||||
## Implementation requires standardized headers and error handling
|
||||
|
||||
Production rate limiters must communicate limits, remaining quota, and reset timing to clients reliably. IETF standardization efforts and RFC specifications provide battle-tested patterns.
|
||||
|
||||
**IETF draft-ietf-httpapi-ratelimit-headers-10** defines modern standard replacing legacy X-RateLimit headers. `RateLimit-Policy` header advertises quota policies: `"default";q=100;w=60` (100 requests per 60 second window). Parameters include `q` (REQUIRED quota limit), `w` (OPTIONAL window seconds), `qu` (OPTIONAL quota unit like "requests", "content-bytes", "concurrent-requests"), and `pk` (OPTIONAL partition key for multi-tenant). Multiple policies can coexist: `"burst";q=100;w=60,"daily";q=1000;w=86400`. The `RateLimit` header indicates current service limits: `"default";r=50;t=30` (50 remaining with 30 seconds until reset). Parameters: `r` (REQUIRED remaining quota), `t` (OPTIONAL delta-seconds until reset). Critical design: **use delta-seconds not timestamps** to avoid clock synchronization issues, prevent clock skew problems, and eliminate thundering herd when all clients reset simultaneously.
|
||||
|
||||
**Legacy headers** remain common during transition period. Standard pattern: `X-RateLimit-Limit: 100` (maximum allowed), `X-RateLimit-Remaining: 50` (quota remaining), `X-RateLimit-Reset: 60` (seconds until reset), `Retry-After: 60` (when rate limited). Legacy headers should be maintained alongside IETF standard headers during migration period, with documentation clarifying which to prefer.
|
||||
|
||||
**HTTP 429 Too Many Requests** provides standardized error response. RFC 9457 Problem Details format:
|
||||
|
||||
```http
|
||||
HTTP/1.1 429 Too Many Requests
|
||||
Content-Type: application/problem+json
|
||||
Retry-After: 60
|
||||
RateLimit: "default";r=0;t=60
|
||||
|
||||
{
|
||||
"type": "https://iana.org/assignments/http-problem-types#quota-exceeded",
|
||||
"title": "Too Many Requests",
|
||||
"detail": "You have exceeded the maximum number of requests",
|
||||
"instance": "/api/users/123",
|
||||
"violated-policies": ["default"],
|
||||
"trace": {
|
||||
"requestId": "uuid-here"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**HTTP 503 Service Unavailable** signals temporary capacity reduction distinct from quota exhaustion. Use when system is degraded but client hasn't exceeded personal quota: `Retry-After: 120` with problem details type `temporary-reduced-capacity`. This distinction enables clients to understand whether they should back off permanently (429) or retry after system recovery (503).
|
||||
|
||||
**Exponential backoff with jitter** prevents thundering herd. Client implementation should parse `Retry-After` header, falling back to exponential calculation: initial delay 1-2 seconds, doubling on each retry, max 3-5 attempts. Jitter critical: add ±20% randomization to delay so clients don't synchronize retries. Production implementation:
|
||||
|
||||
```javascript
|
||||
async function fetchWithRetry(url, maxRetries = 3) {
|
||||
for (let attempt = 0; attempt < maxRetries; attempt++) {
|
||||
const res = await fetch(url);
|
||||
|
||||
if (res.status === 429) {
|
||||
const retryAfter = res.headers.get('retry-after');
|
||||
const delay = parseInt(retryAfter) * 1000 || (1000 * Math.pow(2, attempt));
|
||||
const jitter = delay * 0.2 * (Math.random() - 0.5);
|
||||
|
||||
await sleep(delay + jitter);
|
||||
continue;
|
||||
}
|
||||
|
||||
return res;
|
||||
}
|
||||
throw new Error('Max retries exceeded');
|
||||
}
|
||||
```
|
||||
|
||||
**Data structure design** determines algorithm performance. Fixed window uses simple counter with TTL: `rate_limit:{user_id}:{window_id}` storing integer count with O(1) time and space. Sliding window log employs Redis sorted set: `rate_limit:{user_id}` with timestamps as both members and scores, O(N) space, O(log N) operations. Sliding window counter stores two counters `{user}:{current_minute}` and `{user}:{previous_minute}` with weighted formula. Token bucket uses hash with fields `{tokens: float, last_refill: timestamp}`. Key design principle: all TTL must be set to prevent memory leaks, typically 2× window duration to handle edge cases, with token bucket using longer TTL (3600 seconds) for intermittent users.
|
||||
|
||||
**Testing strategies** validate implementation correctness. Unit testing covers basic limit enforcement (verify N requests allowed, N+1 rejected), window reset (confirm new requests allowed after expiry), and atomicity under concurrency (verify exactly N allowed from 2N concurrent requests). Load testing uses k6 or JMeter: 100 virtual users, 30 second duration, thresholds ensuring 429 rate below 10%, validation that rate limit headers present. Production testing should verify: Lua scripts atomic across operations, TTL set on all keys, server-side time used exclusively, headers RFC-compliant, 429/503 status codes appropriate, exponential backoff with jitter, monitoring tracks health metrics, failover tested with Redis Sentinel, degradation plan for outages.
|
||||
|
||||
**Edge cases** require careful handling. Clock skew solved by server-side monotonic time only—never trust client timestamps. Bounded tolerance can sanitize suspicious timestamps: `if (abs(client_time - server_time) > max_skew) return server_time`. Race conditions prevented by atomic Lua scripts executing all operations in single transaction. Distributed locks (Redlock) provide alternative but add latency. Failover scenarios handled via Redis Sentinel with 3-5 sentinel nodes monitoring master health, applications reconnecting automatically. Graceful degradation decides fail-open (allow requests, prioritize availability) versus fail-closed (deny requests, prioritize security). Network partitions use max-wins conflict resolution: `resolved = max(countA, countB)` taking most restrictive count after heal. Thundering herd prevented by jitter in retry timing: ±20% randomization in Retry-After. Memory leaks avoided by aggressive cleanup: `ZREMRANGEBYSCORE` removes old entries plus `EXPIRE` for auto-deletion.
|
||||
|
||||
**Production deployment checklist** ensures operational readiness: atomicity via Lua scripts for all operations, TTL set on all Redis keys, time source server-side monotonic only, headers return RFC-compliant RateLimit format, status codes use 429/503 appropriately, retry logic implements exponential backoff plus jitter, monitoring tracks limiter health metrics, load testing under expected peak load, failover via Redis Sentinel or Cluster mode, documentation provides clear rate limit policies, degradation plan handles graceful failure, logging captures violations for security analysis. Performance benchmarks on AWS ElastiCache r6g.large: Fixed Window ~50,000 ops/sec, Sliding Window Log ~10,000 ops/sec, Sliding Window Counter ~40,000 ops/sec, Token Bucket ~35,000 ops/sec.
|
||||
|
||||
## Security considerations must address sophisticated attacks
|
||||
|
||||
Rate limiting serves as critical security control, but attackers continuously evolve evasion techniques. Defense-in-depth architectures, proper monitoring, and attack-specific mitigations protect against sophisticated threats.
|
||||
|
||||
**DDoS protection** requires multi-layer defense. Layer 3/4 network rate limiting at CDN edge blocks volumetric floods before reaching application infrastructure. Layer 7 application rate limiting inspects HTTP requests, blocking application-layer attacks (Slowloris, HTTP floods). Distributed global rate limiting with shared state across edge nodes prevents circumvention via geographic distribution. Strict limits on expensive endpoints protect resource-intensive operations: database queries, external API calls, complex computations warrant 10-100× lower limits than read operations.
|
||||
|
||||
**Brute force protection** demands endpoint-specific strategies. Login protection implements multiple limit layers: 5 attempts per 5 minutes per IP, 10 attempts per hour per username, 10,000 attempts per hour globally. Actions escalate: return 429 after limit, exponential backoff on repeated failures, CAPTCHA after 3 failures, account lock after 10 failures, security team alert on patterns. Credential stuffing detection tracks failed login attempts over 1 hour window: threshold of >100 401 responses from single IP or >50 403 responses triggers rate limit reduction (1 request/minute), CAPTCHA requirement, and security review.
|
||||
|
||||
**API scraping prevention** detects automated data collection. Indicators include high volume (>1000 requests/minute), high 404 rate (>50% of requests suggesting enumeration), suspicious or missing User-Agent headers, and sequential resource ID access patterns. Actions reduce limit to 10 requests/minute, require authentication, or challenge with proof-of-work (difficulty 5). More sophisticated scrapers warrant behavioral analysis: mouse movement patterns, JavaScript execution validation, and browser fingerprint consistency checks.
|
||||
|
||||
**Defense in depth** layers multiple security controls. Layer 1 CDN rate limiting provides first defense at edge. Layer 2 API Gateway rate limiting adds second checkpoint. Layer 3 application rate limiting enforces business logic limits. Layer 4 database connection limits prevent resource exhaustion. Layer 5 circuit breakers detect cascade failures and fail gracefully. Each layer defends against different attack vectors with different granularity.
|
||||
|
||||
**Monitoring and alerting** detect attacks in progress. Metrics to track: rate limit hits (requests denied), 429 response count, requests per second (detect spikes), average burst size. Alerts trigger on: spike in 429s (>1000/minute suggests attack), single user abuse (>90% of limit repeatedly), distributed attack (>100 IPs simultaneously hitting limits). Log violations for security analysis: timestamp, user identifier, endpoint, limit exceeded, IP address, User-Agent, and any additional context. Integration with SIEM systems enables correlation with other security events.
|
||||
|
||||
**Key protection** prevents limit bypass via compromised credentials. Never expose secrets client-side—API keys, tokens, or credentials must remain server-side only. Rotate API keys regularly (quarterly or after suspected compromise). Use different keys per environment (development, staging, production). Implement key revocation capability with immediate effect. Monitor for compromised keys via anomaly detection: sudden usage spike, requests from unusual geographies, or access pattern changes suggest compromise.
|
||||
|
||||
**IP spoofing prevention** validates request origin. Trust X-Forwarded-For only from trusted proxies (Cloudflare IPs, internal load balancers), falling back to direct connection IP for untrusted sources. Validation logic: `if (source_ip in trusted_proxies) use_x_forwarded_for else use_source_ip`. Attackers cannot spoof source IP in TCP connections (requires completing handshake), but HTTP headers easily forged. Additional validation: check for multiple X-Forwarded-For values (chain of proxies), validate IP format, and compare against geolocation data for consistency.
|
||||
|
||||
**Hierarchical rate limiting** prevents resource starvation. Global infrastructure limit (100,000/second) protects total capacity. Category limits (authentication 10,000/minute, data API 50,000/minute) prevent single category monopolizing resources. Per-user limits (1,000/hour) ensure fairness. Per-endpoint limits protect expensive operations. All layers checked hierarchically with atomic counter increments ensuring consistency. Slack's approach: global 100 notifications/30 minutes with category sublimits (errors 10, warnings 10, info 10) that sum above global, demonstrating global as final constraint.
|
||||
|
||||
**Common pitfalls** undermine security if not avoided. Non-atomic read-modify-write creates race conditions: `const count = await redis.get(key); if (count < limit) await redis.set(key, count + 1)` allows concurrent requests both succeeding. Solution: atomic Lua scripts. Missing TTL causes memory leaks: `await redis.incr(key)` lives forever. Solution: `redis.multi().incr(key).expire(key, 60).exec()`. Trusting client time enables manipulation: `const time = req.body.timestamp` attacker-controlled. Solution: `const time = Date.now()` server authority. No error handling causes cascading failures. Solution: try-catch with fail-closed default and comprehensive logging.
|
||||
|
||||
The security landscape continuously evolves. **Adaptive, ML-based rate limiting with defense-in-depth** provides robust protection against current threats while remaining flexible enough to address emerging attack patterns. Regular security audits, penetration testing, and incident response planning ensure rate limiting effectiveness over time.
|
||||
|
|
@ -1,426 +0,0 @@
|
|||
# TLS/SSL & Let's Encrypt ACME Protocol: Complete Implementation Guide
|
||||
|
||||
Transport Layer Security (TLS) implementation remains critical for production systems in 2025, with TLS 1.3 now mandatory for federal systems and industry standards pushing toward stronger security defaults. This guide provides comprehensive technical documentation for implementing secure TLS/SSL infrastructure with focus on Haskell libraries and ACME automation.
|
||||
|
||||
## TLS protocol evolution: From 2-RTT to 1-RTT handshakes
|
||||
|
||||
**TLS 1.3 achieves 50% faster handshakes** through fundamental protocol redesign. The TLS 1.2 handshake requires two full round-trip times before application data flows, taking 200-400ms on typical networks. TLS 1.3 reduces this to a single round-trip by having clients send speculative key shares in the ClientHello message, enabling servers to derive shared secrets immediately. The simplified protocol removes 20+ years of accumulated vulnerabilities by eliminating CBC mode ciphers, static RSA key exchange, and compression—each responsible for major security incidents.
|
||||
|
||||
The handshake differences reveal security improvements. TLS 1.2 transmits the entire handshake in cleartext except the final Finished messages, exposing certificate chains and negotiated parameters to network observers. TLS 1.3 encrypts all handshake messages after ServerHello, protecting certificate information and reducing surveillance capabilities. The protocol mandates forward secrecy through ephemeral key exchange (ECDHE/DHE only), meaning session keys remain secure even if long-term private keys are later compromised—a crucial property TLS 1.2's RSA key exchange lacked.
|
||||
|
||||
Cipher suite configuration drastically simplifies in TLS 1.3. Where TLS 1.2 requires specifying key exchange, authentication, encryption, and MAC algorithms separately (like `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`), **TLS 1.3 reduces this to just bulk cipher and hash** (`TLS_AES_128_GCM_SHA256`). Only five cipher suites exist, all using Authenticated Encryption with Associated Data (AEAD). The protocol removes all vulnerable legacy options: RC4, 3DES, CBC mode, MD5, SHA-1 MACs, and anonymous/NULL ciphers all vanish. This eliminates entire attack classes including BEAST, CRIME, Lucky13, and POODLE.
|
||||
|
||||
### Modern cipher suite selection for 2025
|
||||
|
||||
For production deployments supporting both TLS 1.2 and 1.3, **configure cipher suites in strict order**: `TLS_AES_128_GCM_SHA256`, `TLS_CHACHA20_POLY1305_SHA256`, `TLS_AES_256_GCM_SHA384` for TLS 1.3, followed by `ECDHE-ECDSA-AES128-GCM-SHA256`, `ECDHE-RSA-AES128-GCM-SHA256`, `ECDHE-ECDSA-CHACHA20-POLY1305`, and `ECDHE-RSA-CHACHA20-POLY1305` for TLS 1.2 compatibility. Every cipher must provide Perfect Forward Secrecy through ephemeral key exchange. Disable SSLv2, SSLv3, TLS 1.0, and TLS 1.1 completely—they're deprecated since 2020 and vulnerable to protocol downgrade attacks.
|
||||
|
||||
The most critical configuration: **enable server cipher order preference**. Without this, malicious clients can force weak cipher selection through downgrade attacks. Set `ssl_prefer_server_ciphers on` in Nginx or `SSLHonorCipherOrder On` in Apache. Prioritize AEAD ciphers (GCM, ChaCha20-Poly1305), prefer ECDHE over DHE for performance, and exclude any cipher suite lacking ephemeral key exchange. Never use ciphers with RSA key exchange (`TLS_RSA_*`), which remain vulnerable to ROBOT attacks and provide no forward secrecy.
|
||||
|
||||
For Diffie-Hellman parameters, **use minimum 2048-bit groups**, preferably 3072-bit for enhanced security. TLS 1.3 standardizes predefined FFDHE groups from RFC 7919 (ffdhe2048, ffdhe3072, ffdhe4096) and elliptic curve groups (X25519, X448, P-256, P-384). X25519 provides the best performance-security balance and should be your first choice. Generate custom DH parameters for TLS 1.2: `openssl dhparam -out dhparam.pem 2048`.
|
||||
|
||||
### Certificate validation and chain building
|
||||
|
||||
Certificate validation requires building a trust chain from the end-entity certificate to a trusted root CA. The process involves cryptographic signature verification of each certificate by its issuer, validity period checking (notBefore ≤ current_time < notAfter), hostname verification against the Subject Alternative Name extension, and revocation checking via CRL or OCSP. Common misconfigurations include serving incomplete certificate chains—the server MUST send the complete chain excluding only the root CA, which clients already trust.
|
||||
|
||||
**OCSP stapling eliminates 30%+ latency overhead** from traditional revocation checking. Without stapling, browsers make separate connections to CA OCSP responders, adding round-trips and creating privacy concerns as CAs observe which sites users visit. With stapling enabled, the server periodically fetches signed OCSP responses and includes them in TLS handshakes. This improves privacy by eliminating CA surveillance, reduces latency by removing extra connections, and increases reliability by caching responses server-side.
|
||||
|
||||
Certificate Transparency provides detection of mis-issued certificates. Since April 2018, Chrome requires all certificates to appear in public CT logs with Signed Certificate Timestamps (SCTs). Browsers verify SCTs during handshakes, rejecting certificates lacking transparency proof. This prevents rogue CAs from secretly issuing certificates for domains they don't control, as all certificates become publicly auditable at crt.sh and similar services.
|
||||
|
||||
## Server Name Indication: Virtual hosting at scale
|
||||
|
||||
SNI solves a fundamental TLS limitation: **servers must present certificates before knowing which domain clients request**. Without SNI, hosting multiple HTTPS sites on a single IP address becomes impossible—each domain requires dedicated IP space or error-prone wildcard certificates. SNI extends the TLS protocol by adding a server_name field to the ClientHello message, transmitted in plaintext before encryption begins. This allows servers to select the appropriate certificate based on the requested hostname.
|
||||
|
||||
The protocol operates at the TLS handshake level. During ClientHello, the client includes an SNI extension (type code 0) containing the fully qualified DNS hostname in ASCII encoding. The server reads this value before presenting its certificate, enabling selection from multiple certificates bound to the same IP address. This unlocks cost-effective virtual hosting at massive scale—cloud providers like Cloudflare serve millions of domains from shared IP addresses using SNI routing.
|
||||
|
||||
**Privacy represents SNI's critical weakness**: the hostname transmits in cleartext during handshakes, visible to network observers. This enables censorship (China, Iran, Turkey filter based on SNI values), corporate surveillance, and ISP tracking. Encrypted Client Hello (ECH) addresses this by encrypting sensitive handshake parameters including SNI. ECH uses a dual-ClientHello architecture: ClientHelloOuter contains public information and a public server name (like cloudflare.com), while ClientHelloInner (encrypted via HPKE) contains the real SNI and sensitive extensions. Chrome and Firefox enabled ECH by default in 2023, requiring DNS-over-HTTPS for public key distribution via HTTPS/SVCB DNS records.
|
||||
|
||||
Legacy compatibility remains a consideration. Clients without SNI support (Windows XP, Android pre-2.3, ancient embedded devices) cannot specify hostnames during handshakes. Servers should configure fallback default certificates for these clients, though their market share approaches zero in 2025. More relevant: direct IP address connections lack hostnames, requiring separate handling. Wildcard certificates (*.example.com) work with SNI but only match single subdomain levels—*.example.com matches api.example.com but not deep.api.example.com.
|
||||
|
||||
## Certificate management lifecycle
|
||||
|
||||
**Modern certificate lifecycles default to 90 days**, driven by Let's Encrypt's automation-first philosophy. Short lifetimes reduce compromise windows and force proper automation, preventing the manual renewal chaos that plagued annual certificates. The CA/Browser Forum now mandates maximum 398-day validity (13 months) for publicly trusted certificates, down from multi-year certificates common before 2020. This shift fundamentally changes operational practices—manual certificate management becomes untenable at 90-day renewal cycles.
|
||||
|
||||
Certificate storage security requires strict permissions. Private keys must be readable only by the service account: `chmod 400` or `chmod 600` with `chown root:root` on Linux systems. Store keys in dedicated directories with `chmod 700` permissions, separate from world-readable certificate directories. For high-value keys, Hardware Security Modules (HSMs) provide tamper-resistant storage with FIPS 140-2 Level 3/4 compliance. Cloud providers offer managed HSMs: AWS CloudHSM, Azure Key Vault Premium, Google Cloud HSM. These prevent private key extraction even by administrators with root access.
|
||||
|
||||
**Automated rotation at 30 days before expiry** provides 30-day buffer for failure recovery with 90-day certificates. This timing allows two renewal attempts before expiration: initial attempt at 60 days into the 90-day lifetime, with daily retries if necessary. ACME Renewal Info (ARI) further optimizes this by providing server-specified renewal windows exempt from rate limits. Zero-downtime rotation techniques include load balancer certificate overlap (add new certificate while old remains valid), hot reload (Nginx `nginx -s reload`, Apache `apachectl graceful`), and canary deployments (5% → 25% → 50% → 100% server rollout).
|
||||
|
||||
Certificate chains require complete intermediate certificates. Servers MUST send end-entity certificate, all intermediates, but NOT the root CA (clients already trust roots). Common misconfiguration: serving only the leaf certificate, causing "unable to get local issuer certificate" errors in OpenSSL-based clients. While Chrome fetches missing intermediates via Authority Information Access (AIA) extensions, curl and many API clients do not. Verify chains: `openssl s_client -connect example.com:443 -showcerts` should show multiple certificates with "Verify return code: 0 (ok)".
|
||||
|
||||
Key generation algorithms balance compatibility and performance. **RSA 2048-bit remains the compatibility standard**, supported universally but requiring larger certificates and slower operations. RSA 3072-bit provides post-2030 security but increases computational overhead. **ECDSA P-256 offers equivalent security with 50% smaller certificates** and faster signing, ideal for mobile and IoT. Ed25519 provides best performance but lower legacy compatibility. For general web use, deploy dual certificate configuration: ECDSA P-256 as primary with RSA 2048 fallback for legacy clients. Generate keys: `openssl ecparam -genkey -name prime256v1 -out private.key` for ECDSA, `openssl genrsa -out private.key 2048` for RSA.
|
||||
|
||||
## ACME protocol: Automated certificate management
|
||||
|
||||
The ACME protocol (RFC 8555) automates the complete certificate lifecycle: account creation, domain authorization, challenge validation, certificate issuance, and renewal. Let's Encrypt issues over 340,000 certificates per hour using ACME, making it the largest CA by certificate count. The protocol uses JSON Web Signature (JWS) for all requests, ensuring authenticity and integrity. Every request includes a replay-prevention nonce, the exact request URL for integrity protection, and account identification via "kid" (key ID) field.
|
||||
|
||||
**The complete ACME workflow flows through seven phases**. First, create an account by generating an ES256 or EdDSA key pair and POSTing to /acme/new-account with contact information and terms of service agreement. Second, submit an order to /acme/new-order specifying up to 100 DNS names or IP addresses. Third, fetch authorizations from the order, each providing multiple challenge options. Fourth, fulfill one challenge per authorization—HTTP-01, DNS-01, or TLS-ALPN-01. Fifth, notify the server by POSTing empty JSON to the challenge URL. Sixth, after all authorizations become valid, finalize by submitting a Certificate Signing Request (CSR) to the order's finalize URL. Seventh, download the issued certificate chain from the certificate URL.
|
||||
|
||||
### Challenge types and selection strategy
|
||||
|
||||
HTTP-01 challenges require serving a specific file at `http://DOMAIN/.well-known/acme-challenge/TOKEN` containing token concatenated with base64url-encoded SHA256 hash of the account public key. The ACME server fetches this file from multiple network vantage points over port 80 (mandatory, no alternatives). **HTTP-01 works for standard websites but cannot issue wildcard certificates** and requires port 80 accessible from the internet. Best for single-server deployments with public web servers.
|
||||
|
||||
DNS-01 challenges require creating TXT records at `_acme-challenge.DOMAIN` containing base64url-encoded SHA256 hash of the key authorization. This challenge type enables wildcard certificate issuance (*.example.com), works without public web servers, functions behind firewalls, and supports multi-server environments easily. **DNS-01 remains the only method for wildcards**. Challenges: requires DNS provider API access, subject to DNS propagation delays (seconds to hours), and exposes sensitive DNS API credentials. Use DNS-01 for wildcard certificates, CDN/proxy scenarios, internal domains, and when port 80 is unavailable.
|
||||
|
||||
TLS-ALPN-01 operates at the TLS layer by requiring servers to present self-signed certificates containing specific acmeIdentifier extensions when clients negotiate the "acme-tls/1" ALPN protocol on port 443. This challenge enables validation when port 80 is blocked but 443 remains accessible, suitable for TLS-terminating proxies. Limited adoption due to implementation complexity and lack of client library support—HTTP-01 or DNS-01 preferred in most scenarios.
|
||||
|
||||
### Rate limits and operational considerations
|
||||
|
||||
Let's Encrypt enforces multiple rate limit categories. **Certificates per Registered Domain**: 50 per 7 days (refills 1 per 202 minutes), overridable for hosting providers. **New Orders per Account**: 300 per 3 hours (refills 1 per 36 seconds), overridable. **Duplicate Certificates** (identical identifier set): 5 per 7 days (refills 1 per 34 hours), NOT overridable. **Authorization Failures per Identifier**: 5 per hour, NOT overridable. **Consecutive Authorization Failures**: 1,152 maximum (refills 1 per day, resets on success), NOT overridable.
|
||||
|
||||
**ACME Renewal Info (ARI) exempts renewal requests from ALL rate limits**, making it the preferred renewal method. Query the /acme/renewal-info/{certID} endpoint at least twice daily to receive optimal renewal windows. Renewals using the same identifier set also bypass New Orders and Certificates per Domain limits (but remain subject to Duplicate Certificates and Authorization Failures). This enables high-volume hosting providers to renew millions of certificates without hitting limits.
|
||||
|
||||
Staging environment testing proves critical: use https://acme-staging-v02.api.letsencrypt.org/directory for all development and testing. Staging provides 30,000 certificates per week vs 50 production, 1,500 new orders per 3 hours vs 300 production. Certificates from staging are NOT browser-trusted (use for automated testing only). Common pitfall: testing against production accidentally, consuming precious rate limit quota and potentially triggering authorization failure lockouts.
|
||||
|
||||
Account key management requires careful handling. Generate ES256 (ECDSA P-256) or EdDSA keys for ACME accounts—RSA 2048+ works but is not recommended. Store account keys separately from certificate keys. The account key identifies your account in all ACME operations via "kid" in JWS headers. Key rollover uses a clever double-JWS structure: inner JWS signed by NEW key (containing account URL and old public key), outer JWS signed by OLD key (containing inner JWS). This atomic operation prevents any service interruption during key rotation.
|
||||
|
||||
## Haskell TLS implementation guide
|
||||
|
||||
The Haskell TLS ecosystem provides pure Haskell implementations avoiding OpenSSL dependencies. The core `tls` library (version 2.1.13) supports TLS 1.2 and 1.3, implements modern cipher suites (AES-GCM, ChaCha20-Poly1305), and provides SNI, ALPN, session resumption, and Encrypted Client Hello support. The library underwent breaking changes in version 2.x, switching from cryptonite to crypton and removing data-default dependency. All applications should use TLS 1.2 as minimum with TLS 1.3 preferred.
|
||||
|
||||
### Basic TLS client implementation
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
import Network.TLS
|
||||
import Network.TLS.Extra.Cipher
|
||||
import Network.Socket
|
||||
import Data.Default.Class
|
||||
|
||||
tlsClient :: HostName -> ServiceName -> IO ()
|
||||
tlsClient hostname port = do
|
||||
-- Create TCP socket
|
||||
addr:_ <- getAddrInfo Nothing (Just hostname) (Just port)
|
||||
sock <- socket (addrFamily addr) Stream defaultProtocol
|
||||
connect sock (addrAddress addr)
|
||||
|
||||
-- Configure TLS parameters
|
||||
let params = (defaultParamsClient hostname "")
|
||||
{ clientSupported = def
|
||||
{ supportedCiphers = ciphersuite_default
|
||||
, supportedVersions = [TLS13, TLS12]
|
||||
, supportedGroups = [X25519, P256]
|
||||
}
|
||||
, clientShared = def
|
||||
{ sharedCAStore = systemStore
|
||||
}
|
||||
}
|
||||
|
||||
-- Perform handshake
|
||||
ctx <- contextNew sock params
|
||||
handshake ctx
|
||||
|
||||
-- Send HTTP request
|
||||
sendData ctx "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
|
||||
response <- recvData ctx
|
||||
print response
|
||||
|
||||
-- Clean shutdown
|
||||
bye ctx
|
||||
close sock
|
||||
```
|
||||
|
||||
The `tls` library provides predefined cipher suite configurations: `ciphersuite_default` includes recommended strong ciphers, `ciphersuite_strong` restricts to only strongest (PFS + AEAD + SHA2), and `ciphersuite_all` includes legacy ciphers (avoid in production). **Always specify supportedVersions explicitly** to prevent TLS 1.0/1.1 usage. System certificate stores integrate via `sharedCAStore = systemStore`, using the operating system's trusted root certificates.
|
||||
|
||||
### HTTPS server with warp-tls
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
import Network.Wai
|
||||
import Network.Wai.Handler.Warp
|
||||
import Network.Wai.Handler.WarpTLS
|
||||
import qualified Network.TLS as TLS
|
||||
import Network.TLS.Extra.Cipher
|
||||
|
||||
app :: Application
|
||||
app _ respond =
|
||||
respond $ responseLBS status200
|
||||
[("Content-Type", "text/plain")
|
||||
,("Strict-Transport-Security", "max-age=31536000; includeSubDomains")]
|
||||
"Secure HTTPS Server"
|
||||
|
||||
main :: IO ()
|
||||
main = do
|
||||
let tlsConfig = (tlsSettings "certificate.pem" "key.pem")
|
||||
{ tlsAllowedVersions = [TLS.TLS13, TLS.TLS12]
|
||||
, tlsCiphers = ciphersuite_strong
|
||||
, onInsecure = DenyInsecure "HTTPS required"
|
||||
}
|
||||
|
||||
warpConfig = defaultSettings
|
||||
& setPort 443
|
||||
& setHost "0.0.0.0"
|
||||
|
||||
putStrLn "HTTPS server on :443"
|
||||
runTLS tlsConfig warpConfig app
|
||||
```
|
||||
|
||||
Warp-TLS (version 3.4.13) provides production-ready HTTPS servers with HTTP/2 support via ALPN negotiation. The `tlsSettings` function loads certificate and key files, while `tlsAllowedVersions` restricts protocol versions. Setting `onInsecure = DenyInsecure "message"` rejects plain HTTP connections with a clear error message. For SNI multi-domain hosting, use `tlsSettingsSni` with a function returning appropriate credentials per hostname.
|
||||
|
||||
### Certificate validation with x509
|
||||
|
||||
```haskell
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
import Data.X509
|
||||
import Data.X509.Validation
|
||||
import Data.X509.CertificateStore
|
||||
import System.X509
|
||||
|
||||
validateCertificate :: FilePath -> HostName -> IO Bool
|
||||
validateCertificate certFile hostname = do
|
||||
-- Load certificate chain
|
||||
certs <- readSignedObject certFile
|
||||
let chain = CertificateChain certs
|
||||
|
||||
-- Get system CA store
|
||||
store <- getSystemCertificateStore
|
||||
|
||||
-- Validate with default checks
|
||||
let cache = exceptionValidationCache []
|
||||
failures <- validateDefault store cache (hostname, ":443") chain
|
||||
|
||||
case failures of
|
||||
[] -> putStrLn "✓ Certificate valid" >> return True
|
||||
errs -> do
|
||||
putStrLn "✗ Certificate validation failed:"
|
||||
mapM_ (putStrLn . (" " ++) . show) errs
|
||||
return False
|
||||
```
|
||||
|
||||
The `x509` library (version 1.7.7) parses X.509 certificates, while `x509-validation` (1.6.12) performs chain validation. The `validateDefault` function implements complete RFC 5280 validation: cryptographic signature verification, validity period checks, hostname matching via Subject Alternative Names, CA constraints, and key usage verification. Custom validation hooks enable certificate pinning or specialized trust models.
|
||||
|
||||
### ACME certificate automation workflow
|
||||
|
||||
```haskell
|
||||
import System.Process
|
||||
import System.Directory
|
||||
import Control.Monad
|
||||
|
||||
data CertConfig = CertConfig
|
||||
{ domains :: [String]
|
||||
, webroot :: FilePath
|
||||
, accountKey :: FilePath
|
||||
, domainKey :: FilePath
|
||||
, certFile :: FilePath
|
||||
}
|
||||
|
||||
initializeKeys :: CertConfig -> IO ()
|
||||
initializeKeys cfg = do
|
||||
accountExists <- doesFileExist (accountKey cfg)
|
||||
domainExists <- doesFileExist (domainKey cfg)
|
||||
|
||||
unless accountExists $
|
||||
callCommand $ "openssl genrsa 4096 > " ++ accountKey cfg
|
||||
|
||||
unless domainExists $
|
||||
callCommand $ "openssl genrsa 4096 > " ++ domainKey cfg
|
||||
|
||||
requestCert :: CertConfig -> IO ()
|
||||
requestCert cfg = do
|
||||
let cmd = unwords
|
||||
[ "hasencrypt -D"
|
||||
, "-w", webroot cfg
|
||||
, "-a", accountKey cfg
|
||||
, "-d", domainKey cfg
|
||||
, unwords (domains cfg)
|
||||
, ">", certFile cfg
|
||||
]
|
||||
callCommand cmd
|
||||
|
||||
renewCert :: CertConfig -> IO Bool
|
||||
renewCert cfg = do
|
||||
exists <- doesFileExist (certFile cfg)
|
||||
if exists
|
||||
then do
|
||||
let cmd = unwords
|
||||
[ "hasencrypt -D"
|
||||
, "-w", webroot cfg
|
||||
, "-a", accountKey cfg
|
||||
, "-d", domainKey cfg
|
||||
, "-r", certFile cfg
|
||||
, unwords (domains cfg)
|
||||
]
|
||||
exitCode <- system cmd
|
||||
return $ exitCode == ExitSuccess
|
||||
else do
|
||||
requestCert cfg
|
||||
return True
|
||||
```
|
||||
|
||||
**Hasencrypt provides the most mature Haskell ACME client**, supporting HTTP-01 challenges with automatic renewal. The `-D` flag selects Let's Encrypt production, empty `-D` uses production, omitting `-D` uses staging. The `-r` flag enables smart renewal (only renews if certificate expires soon). Schedule renewal checks with cron: `0 2 * * * hasencrypt -D -w /var/www -a account.pem -d domain.pem -r cert.pem example.com`. After successful renewal, reload web servers: `nginx -s reload` or `systemctl reload nginx`.
|
||||
|
||||
For DNS-01 challenges and wildcard certificates, integrate with DNS provider APIs. Popular providers with Haskell support include Cloudflare (via cloudflare-api package), Route53 (via amazonka), and DigitalOcean (via digitalocean package). Create TXT records at _acme-challenge.domain.com with the challenge value, wait for DNS propagation, then notify the ACME server. Clean up old TXT records to prevent response size issues.
|
||||
|
||||
## Security best practices for production
|
||||
|
||||
**HSTS deployment requires staged rollout to prevent lockout scenarios**. Start with short max-age (300 seconds) for testing, monitoring all resources load over HTTPS. Increase to 1 week (604800), then 1 month (2592000) while monitoring logs. Finally deploy long-term policy: `Strict-Transport-Security: max-age=63072000; includeSubDomains; preload`. The 2-year max-age provides strong security while includeSubDomains applies policy to all subdomains (ensure ALL subdomains support HTTPS first). The preload directive signals consent for inclusion in browser hardcoded lists, providing protection on first visit but nearly irreversible—test thoroughly for 3-6 months before preload submission.
|
||||
|
||||
OCSP stapling configuration eliminates 30%+ handshake latency by caching revocation responses server-side. **Enable in Nginx**: `ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /path/to/chain.pem; resolver 8.8.8.8;`. **Enable in Apache**: `SSLUseStapling on; SSLStaplingCache "shmcb:/var/run/ocsp(128000)"`. Verify with `openssl s_client -connect example.com:443 -status -tlsextdebug`, looking for "OCSP Response Status: successful". The server fetches signed OCSP responses periodically and includes them in TLS handshakes, improving privacy (CA no longer observes site visits), reducing latency (no client→OCSP connection), and increasing reliability (cached responses insulate from OCSP server outages).
|
||||
|
||||
Modern security headers provide defense-in-depth. **Content-Security-Policy** prevents XSS attacks: `default-src 'self'; script-src 'self' 'unsafe-inline' cdn.example.com`. **X-Frame-Options** prevents clickjacking: `DENY` or `SAMEORIGIN`. **X-Content-Type-Options** prevents MIME sniffing: `nosniff`. **Referrer-Policy** controls referrer information: `strict-origin-when-cross-origin`. **Permissions-Policy** restricts feature access: `geolocation=(), camera=(), microphone=()`. Deploy these alongside HSTS for comprehensive security.
|
||||
|
||||
Certificate pinning (HPKP) is deprecated and removed from all major browsers as of 2020 due to operational risks—misconfiguration causes permanent lockout until max-age expires with no recovery mechanism. Modern alternatives include Expect-CT (transitional, reports violations), Certificate Transparency monitoring (detect unauthorized issuance), and application-level pinning for mobile apps. For web services, rely on proper certificate validation and CT monitoring rather than pinning.
|
||||
|
||||
## Production deployment security checklist
|
||||
|
||||
### TLS Configuration
|
||||
- **Disable vulnerable protocols**: SSLv2, SSLv3, TLS 1.0, TLS 1.1 completely
|
||||
- **Enable modern protocols**: TLS 1.2 (minimum), TLS 1.3 (preferred)
|
||||
- **Configure strong ciphers**: ECDHE/DHE + AES-GCM/ChaCha20-Poly1305 only
|
||||
- **Enable server cipher preference**: Prevent client-forced downgrades
|
||||
- **Use strong DH parameters**: Minimum 2048-bit, prefer 3072-bit or X25519
|
||||
- **Configure ALPN**: Enable HTTP/2 negotiation for performance
|
||||
|
||||
### Certificate Management
|
||||
- **Use short-lived certificates**: 90-day maximum, prefer automated renewal
|
||||
- **Set renewal at 30 days**: Provides buffer for failure recovery
|
||||
- **Deploy complete chains**: End-entity + intermediates, exclude root
|
||||
- **Enable OCSP stapling**: Cache revocation responses server-side
|
||||
- **Monitor expiration**: Alert at 30/15/7 days before expiry
|
||||
- **Use strong key algorithms**: RSA 2048+ or ECDSA P-256+, prefer ECDSA
|
||||
- **Secure private keys**: Filesystem permissions 400/600, consider HSM for high-value keys
|
||||
|
||||
### ACME Automation
|
||||
- **Test in staging first**: Use staging environment for all development
|
||||
- **Implement ARI**: Query renewal-info endpoints for rate-limit-exempt renewals
|
||||
- **Handle failures gracefully**: Exponential backoff, comprehensive logging
|
||||
- **Choose appropriate challenge**: HTTP-01 for standard, DNS-01 for wildcards
|
||||
- **Monitor rate limits**: Track consumption, implement client-side limiting
|
||||
- **Automate completely**: Renewal, deployment, monitoring—no manual steps
|
||||
|
||||
### Security Headers
|
||||
- **HSTS**: max-age=63072000; includeSubDomains; preload (after testing)
|
||||
- **CSP**: Restrictive Content-Security-Policy preventing XSS
|
||||
- **X-Frame-Options**: DENY or SAMEORIGIN preventing clickjacking
|
||||
- **X-Content-Type-Options**: nosniff preventing MIME confusion
|
||||
- **Referrer-Policy**: strict-origin-when-cross-origin limiting leakage
|
||||
|
||||
### Monitoring and Testing
|
||||
- **Certificate transparency**: Monitor CT logs at crt.sh for unauthorized issuance
|
||||
- **SSL Labs scan**: Regular A+ rating validation at ssllabs.com/ssltest
|
||||
- **Expiration monitoring**: Automated checks, Prometheus alerts, CloudWatch alarms
|
||||
- **Log all operations**: Certificate requests, renewals, failures, rotations
|
||||
- **Test regularly**: Quarterly disaster recovery drills, renewal failure scenarios
|
||||
|
||||
### Incident Response
|
||||
- **Document procedures**: Key compromise response, certificate revocation process
|
||||
- **Maintain backups**: Encrypted private key backups in multiple secure locations
|
||||
- **Plan rollback**: Keep previous certificates available for emergency rollback
|
||||
- **Test recovery**: Quarterly restoration drills, verify backup integrity
|
||||
|
||||
## Common vulnerabilities and mitigations
|
||||
|
||||
**POODLE** (Padding Oracle On Downgraded Legacy Encryption, 2014) exploited SSL 3.0 CBC padding validation, allowing plaintext recovery through 256 requests per byte. Mitigation: Disable SSL 3.0 completely, reject TLS_FALLBACK_SCSV downgrade attempts.
|
||||
|
||||
**BEAST** (Browser Exploit Against SSL/TLS, 2011) attacked TLS 1.0 CBC ciphers through chosen-plaintext attacks on initialization vectors. Mitigation: Disable TLS 1.0, prefer TLS 1.2+ with AEAD ciphers (GCM, ChaCha20-Poly1305).
|
||||
|
||||
**CRIME** (Compression Ratio Info-leak Made Easy, 2012) and **BREACH** (2013) extracted secrets through TLS/HTTP compression side channels. Mitigation: Disable TLS compression, carefully evaluate HTTP compression for sensitive data.
|
||||
|
||||
**Heartbleed** (2014) exploited OpenSSL heartbeat extension buffer over-read, leaking memory contents including private keys. Mitigation: Update OpenSSL immediately (1.0.1g+), rotate potentially compromised keys, monitor for unauthorized certificate issuance.
|
||||
|
||||
**FREAK** (Factoring RSA Export Keys, 2015) and **Logjam** (2015) forced weak export-grade cryptography through protocol implementation flaws. Mitigation: Disable export ciphers completely, use strong DH parameters (2048-bit minimum), prefer ECDHE over DHE.
|
||||
|
||||
**DROWN** (Decrypting RSA with Obsolete and Weakened eNcryption, 2016) enabled cross-protocol attacks when servers supported both SSLv2 and modern TLS with same keys. Mitigation: Disable SSLv2 on ALL servers using the same keys, never reuse keys across protocols.
|
||||
|
||||
**ROBOT** (Return Of Bleichenbacher's Oracle Threat, 2017) resurged Bleichenbacher padding oracle attacks against RSA key exchange. Mitigation: Disable RSA key exchange cipher suites, use only ECDHE/DHE with forward secrecy.
|
||||
|
||||
**Sweet32** (2016) exploited 64-bit block ciphers (3DES, Blowfish) through birthday attacks after 32GB traffic. Mitigation: Disable 3DES completely, use 128-bit+ block ciphers (AES).
|
||||
|
||||
## Implementation patterns for Haskell applications
|
||||
|
||||
### HTTP client with custom validation
|
||||
|
||||
```haskell
|
||||
import Network.HTTP.Client
|
||||
import Network.HTTP.Client.TLS
|
||||
import Network.Connection
|
||||
import qualified Network.TLS as TLS
|
||||
|
||||
customTlsManager :: IO Manager
|
||||
customTlsManager = do
|
||||
let tlsParams = (TLS.defaultParamsClient "example.com" "")
|
||||
{ TLS.clientSupported = def
|
||||
{ TLS.supportedCiphers = ciphersuite_strong
|
||||
, TLS.supportedVersions = [TLS.TLS13, TLS.TLS12]
|
||||
}
|
||||
, TLS.clientHooks = def
|
||||
{ TLS.onServerCertificate = customValidation
|
||||
}
|
||||
}
|
||||
tlsSettings = TLSSettings tlsParams
|
||||
|
||||
newManager $ mkManagerSettings tlsSettings Nothing
|
||||
|
||||
customValidation :: CertificateStore -> ValidationCache -> ServiceID
|
||||
-> CertificateChain -> IO [FailedReason]
|
||||
customValidation store cache sid chain = do
|
||||
-- Perform standard validation
|
||||
failures <- validateDefault store cache sid chain
|
||||
|
||||
-- Add custom checks (certificate pinning, etc.)
|
||||
if null failures
|
||||
then return []
|
||||
else return failures
|
||||
```
|
||||
|
||||
### SNI-based virtual hosting
|
||||
|
||||
```haskell
|
||||
import Network.Wai.Handler.WarpTLS
|
||||
import qualified Network.TLS as TLS
|
||||
|
||||
loadCredentials :: HostName -> IO TLS.Credential
|
||||
loadCredentials hostname = do
|
||||
let certFile = "certs/" ++ hostname ++ ".crt"
|
||||
keyFile = "certs/" ++ hostname ++ ".key"
|
||||
either error id <$> TLS.credentialLoadX509 certFile keyFile
|
||||
|
||||
main :: IO ()
|
||||
main = do
|
||||
let tlsConfig = tlsSettingsSni
|
||||
(return $ Just loadCredentials)
|
||||
"default.crt"
|
||||
"default.key"
|
||||
|
||||
runTLS tlsConfig defaultSettings app
|
||||
```
|
||||
|
||||
### Session resumption for performance
|
||||
|
||||
```haskell
|
||||
import Network.TLS
|
||||
import Data.IORef
|
||||
|
||||
setupSessionManager :: IO SessionManager
|
||||
setupSessionManager = do
|
||||
sessions <- newIORef Map.empty
|
||||
|
||||
return SessionManager
|
||||
{ sessionResume = \sessionID -> do
|
||||
cache <- readIORef sessions
|
||||
return $ Map.lookup sessionID cache
|
||||
|
||||
, sessionEstablish = \sessionID sessionData -> do
|
||||
modifyIORef' sessions (Map.insert sessionID sessionData)
|
||||
|
||||
, sessionInvalidate = \sessionID -> do
|
||||
modifyIORef' sessions (Map.delete sessionID)
|
||||
}
|
||||
|
||||
clientWithResumption :: HostName -> IO ()
|
||||
clientWithResumption hostname = do
|
||||
manager <- setupSessionManager
|
||||
|
||||
let params = (defaultParamsClient hostname "")
|
||||
{ clientShared = def
|
||||
{ sharedSessionManager = manager
|
||||
}
|
||||
}
|
||||
-- Subsequent connections reuse sessions
|
||||
```
|
||||
|
||||
## Conclusion: Building secure TLS infrastructure
|
||||
|
||||
Modern TLS infrastructure demands automation, monitoring, and defense-in-depth. TLS 1.3 provides mandatory forward secrecy, simplified cipher selection, and improved performance through 1-RTT handshakes. ACME automation eliminates manual certificate management, while short-lived 90-day certificates reduce compromise windows. HSTS prevents protocol downgrades, OCSP stapling improves privacy and performance, and comprehensive monitoring prevents outages.
|
||||
|
||||
The Haskell ecosystem provides production-ready TLS implementations through pure Haskell libraries avoiding OpenSSL dependencies. The tls library supports modern protocols and cipher suites, warp-tls enables high-performance HTTPS servers, and hasencrypt automates ACME certificate acquisition. Integration patterns enable custom validation, session resumption, and SNI-based virtual hosting.
|
||||
|
||||
Security requires vigilance: disable TLS 1.0/1.1, enforce strong cipher suites, implement HSTS with careful rollout, enable OCSP stapling, monitor Certificate Transparency logs, and maintain comprehensive incident response procedures. Test configurations with SSL Labs, automate renewal with ARI-based scheduling, and rehearse failure scenarios quarterly. With proper implementation, TLS infrastructure provides confidentiality, integrity, and authenticity for modern production systems.
|
||||
|
|
@ -1,449 +0,0 @@
|
|||
### docs/research/waf-design.md
|
||||
|
||||
# Web Application Firewall (WAF) Rule Engine Design Research
|
||||
|
||||
### 1. Executive Summary
|
||||
|
||||
A Web Application Firewall (WAF) is a critical component of modern web application security, acting as a shield that inspects and filters HTTP traffic between a web application and the Internet. This document provides an in-depth analysis of the design and implementation of a robust WAF rule engine. It explores the core components of a WAF, strategies for detecting the OWASP Top 10 vulnerabilities, and the intricacies of rule creation and management. By examining detection techniques ranging from traditional signature-based methods to advanced machine learning models, this research outlines a blueprint for a high-performance, low-latency WAF that effectively mitigates threats while minimizing false positives. The document also delves into the ModSecurity rule format as a source of inspiration for a powerful and flexible custom Domain Specific Language (DSL) for defining security rules. Finally, it addresses the ever-evolving landscape of attacker evasion techniques, providing insights into building a resilient WAF architecture.
|
||||
|
||||
### 2. OWASP Top 10 Detection Strategies
|
||||
|
||||
The Open Web Application Security Project (OWASP) Top 10 represents a consensus among security experts about the most critical web application security risks. A modern WAF must have effective strategies to detect and mitigate these vulnerabilities.
|
||||
|
||||
#### **A01:2021 - Broken Access Control**
|
||||
|
||||
Broken Access Control remains a prevalent and severe vulnerability. A WAF can help enforce access control policies by:
|
||||
|
||||
* **URL and Parameter-Based Access Control:** Defining rules that restrict access to specific URLs, directories, and API endpoints based on user roles or IP addresses.
|
||||
* **Session and Token Analysis:** Inspecting session cookies and tokens to ensure they are valid and correspond to the appropriate user privileges for the requested resource.
|
||||
* **Business Logic Anomaly Detection:** Utilizing anomaly detection to identify unusual patterns in user behavior that might indicate an attempt to bypass access controls, such as a standard user attempting to access administrative functions.
|
||||
|
||||
#### **A02:2021 - Cryptographic Failures**
|
||||
|
||||
While primarily an application-level concern, a WAF can contribute to mitigating cryptographic failures by:
|
||||
|
||||
* **Enforcing HTTPS:** Redirecting all HTTP traffic to HTTPS to ensure data is encrypted in transit.
|
||||
* **Inspecting SSL/TLS Handshakes:** Identifying and blocking connections that use weak or deprecated cipher suites.
|
||||
* **Detecting Sensitive Data Exposure:** Using regular expressions to identify and potentially mask sensitive data, such as credit card numbers or social security numbers, in server responses to prevent accidental leakage.
|
||||
|
||||
#### **A03:2021 - Injection**
|
||||
|
||||
Injection flaws, such as SQL, NoSQL, and Command Injection, are a broad category of attacks. A WAF can detect these by:
|
||||
|
||||
* **Signature-Based Detection:** Employing a database of known attack patterns and signatures to identify malicious payloads. This includes looking for common SQL keywords (`SELECT`, `UNION`, `INSERT`), command injection payloads (`/bin/sh`, `powershell`), and other malicious strings.
|
||||
* **Behavioral Analysis:** Understanding the normal structure of SQL queries and application traffic to identify anomalous requests that may indicate an injection attempt.
|
||||
* **Input Validation and Sanitization:** Defining rules that enforce strict input validation, rejecting requests that contain unexpected or malicious characters. In some cases, the WAF can sanitize input by removing or encoding dangerous characters.
|
||||
|
||||
#### **A04:2021 - Insecure Design**
|
||||
|
||||
Insecure design is a broad category that is challenging to address solely with a WAF, as it often stems from fundamental architectural flaws. However, a WAF can provide a layer of defense by:
|
||||
|
||||
* **Virtual Patching:** Applying rules that block known exploits against insecure design patterns until the underlying application code can be fixed.
|
||||
* **Enforcing Security Best Practices:** Implementing rules that check for common insecure design flaws, such as predictable resource locations or the exposure of sensitive files through directory traversal.
|
||||
|
||||
#### **A05:2021 - Security Misconfiguration**
|
||||
|
||||
A WAF can help detect and prevent attacks that exploit security misconfigurations by:
|
||||
|
||||
* **Header Inspection:** Checking for misconfigured security headers (e.g., `Content-Security-Policy`, `Strict-Transport-Security`) and enforcing secure configurations.
|
||||
* **Blocking Verbose Error Messages:** Preventing the leakage of sensitive information by intercepting and sanitizing detailed error messages that could reveal underlying system details.
|
||||
* **Enforcing Whitelists:** Defining strict rules that only allow access to specific, known-safe resources and blocking everything else.
|
||||
|
||||
#### **A06:2021 - Vulnerable and Outdated Components**
|
||||
|
||||
A WAF can mitigate the risks associated with vulnerable components through:
|
||||
|
||||
* **Virtual Patching:** Creating rules to block requests that attempt to exploit known vulnerabilities in third-party libraries and frameworks. This provides a temporary shield while developers work on updating the components.
|
||||
* **Signature-Based Detection:** Using signatures of known exploits for vulnerable components to identify and block attacks.
|
||||
* **Information Leakage Prevention:** Preventing the disclosure of component versions in HTTP headers and error messages, making it harder for attackers to identify vulnerable systems.
|
||||
|
||||
#### **A07:2021 - Identification and Authentication Failures**
|
||||
|
||||
A WAF can help protect against authentication-related attacks by:
|
||||
|
||||
* **Brute-Force Protection:** Implementing rate-limiting rules to block or slow down repeated login attempts from a single IP address.
|
||||
* **Credential Stuffing Detection:** Using reputation-based blocking of IPs known for malicious activity and identifying automated login attempts.
|
||||
* **Session Hijacking Prevention:** Enforcing the use of secure and HttpOnly cookies and monitoring for suspicious session ID manipulation.
|
||||
|
||||
#### **A08:2021 - Software and Data Integrity Failures**
|
||||
|
||||
This category, which includes issues like insecure deserialization, can be addressed by a WAF through:
|
||||
|
||||
* **Signature-Based Detection:** Using rules that look for the signatures of known insecure deserialization payloads.
|
||||
* **Content-Type Validation:** Enforcing strict content-type validation to prevent unexpected data formats from being processed by the application.
|
||||
|
||||
#### **A09:2021 - Security Logging and Monitoring Failures**
|
||||
|
||||
While a WAF is not a replacement for a comprehensive logging and monitoring solution, it plays a crucial role by:
|
||||
|
||||
* **Generating Detailed Logs:** Providing rich logs of all inspected traffic, including blocked requests, matched rules, and request metadata, which can be fed into a SIEM for analysis.
|
||||
* **Alerting on Malicious Activity:** Generating real-time alerts when high-severity rules are triggered, enabling a rapid response to potential attacks.
|
||||
|
||||
#### **A10:2021 - Server-Side Request Forgery (SSRF)**
|
||||
|
||||
A WAF can help prevent SSRF attacks by:
|
||||
|
||||
* **Enforcing Whitelists of Allowed Domains:** Creating rules that restrict outgoing requests from the server to a predefined list of trusted domains.
|
||||
* **Blocking Internal IP Addresses:** Preventing requests to internal or loopback IP addresses.
|
||||
* **URL Schema Validation:** Ensuring that user-supplied URLs conform to expected formats and protocols.
|
||||
|
||||
### 3. Detection Techniques
|
||||
|
||||
A multi-layered approach to detection is essential for an effective WAF.
|
||||
|
||||
#### **Regex Patterns (Signature-Based Detection)**
|
||||
|
||||
Regular expressions are a cornerstone of traditional WAFs, used to identify known attack patterns. For example, a simple regex to detect a basic SQL injection attempt might be `/(union|select|insert|update|delete|from|where)/i`. While effective against common attacks, regex-based detection can be bypassed by sophisticated attackers and can lead to a high number of false positives if not carefully crafted.
|
||||
|
||||
#### **Machine Learning (ML) and Anomaly Detection**
|
||||
|
||||
Modern WAFs are increasingly incorporating machine learning to move beyond static signatures and detect novel attacks. ML models can be trained on vast datasets of both legitimate and malicious traffic to learn the normal behavior of a web application.
|
||||
|
||||
* **Supervised Learning:** Models are trained on labeled data to classify requests as malicious or benign. This is effective for identifying known attack types with high accuracy.
|
||||
* **Unsupervised Learning (Anomaly Detection):** Models are trained on a baseline of normal traffic to identify deviations that could indicate an attack. This approach is particularly useful for detecting zero-day exploits and other unknown threats. Techniques like clustering and statistical modeling can be used to flag requests that fall outside the established norm.
|
||||
|
||||
#### **Behavioral Analysis**
|
||||
|
||||
Behavioral analysis focuses on the sequence and context of user actions. By creating a profile of normal user behavior, a WAF can detect anomalies such as:
|
||||
|
||||
* **Anomalous Sequences of Requests:** A user suddenly accessing pages in an illogical order or performing actions at an unusually high speed.
|
||||
* **Atypical Parameter Values:** Submitting data in a format or range that is not typical for a given user or the application.
|
||||
* **Session Hijacking Indicators:** Sudden changes in the user agent, IP address, or other session-related parameters.
|
||||
|
||||
### 4. False Positive Reduction Strategies
|
||||
|
||||
False positives, where legitimate traffic is incorrectly blocked, are a significant challenge for WAF administrators. Effective strategies for reduction include:
|
||||
|
||||
* **Rule Tuning and Customization:** Regularly reviewing and refining WAF rules based on the specific application's traffic patterns is crucial. This includes creating exceptions for known safe IP addresses or specific URLs.
|
||||
* **Risk-Based Scoring:** Instead of a simple block/allow decision, a WAF can assign a risk score to each request based on a variety of factors. Requests with a low score are allowed, those with a high score are blocked, and those in the middle may be subjected to further scrutiny, such as a CAPTCHA challenge.
|
||||
* **Learning Mode:** Running the WAF in a non-blocking "learning mode" to gather data on normal traffic patterns before enforcing blocking rules.
|
||||
* **Integration with Dynamic Application Security Testing (DAST):** Combining a WAF with a DAST scanner can help to identify which vulnerabilities are actually exploitable and tune WAF rules accordingly.
|
||||
* **Feedback Loops:** Providing a mechanism for users or administrators to report false positives, which can then be used to refine the rule set.
|
||||
|
||||
### 5. ModSecurity Rule Format (for inspiration)
|
||||
|
||||
ModSecurity is a widely-used open-source WAF, and its rule language provides a powerful and flexible model for a custom DSL. The basic syntax of a ModSecurity rule is:
|
||||
|
||||
`SecRule VARIABLES OPERATOR [ACTIONS]`
|
||||
|
||||
* **VARIABLES:** Specifies where to look in the HTTP request or response (e.g., `ARGS`, `REQUEST_HEADERS`, `RESPONSE_BODY`).
|
||||
* **OPERATOR:** Defines how to inspect the variable (e.g., `@rx` for regular expression matching, `@streq` for string equality).
|
||||
* **ACTIONS:** Specifies what to do if the rule matches (e.g., `deny`, `log`, `pass`, `t:lowercase` for transformation).
|
||||
|
||||
ModSecurity's phased processing model (request headers, request body, response headers, response body, logging) is also a valuable concept for ensuring that rules are executed at the appropriate stage of the transaction.
|
||||
|
||||
### 6. Custom Rule DSL Design
|
||||
|
||||
Designing a custom Domain Specific Language (DSL) for WAF rules should prioritize clarity, expressiveness, and ease of use. Key considerations include:
|
||||
|
||||
* **Human-Readability:** The syntax should be intuitive and easy for security analysts to understand and write.
|
||||
* **Structured Format:** Using a structured format like YAML or JSON can make rules easier to parse and manage programmatically.
|
||||
* **Modularity and Reusability:** The DSL should allow for the creation of reusable rule sets and policies that can be applied to different applications.
|
||||
* **Extensibility:** The language should be designed to easily accommodate new detection techniques and actions as threats evolve.
|
||||
* **Clear Semantics for Conditions and Actions:** The DSL should have a well-defined set of conditions (e.g., `matches_regex`, `is_in_ip_list`) and corresponding actions (`block`, `log`, `redirect`, `add_header`).
|
||||
|
||||
### 7. Performance Optimization
|
||||
|
||||
A WAF must inspect traffic with minimal impact on latency to avoid degrading the user experience. Performance optimization strategies include:
|
||||
|
||||
* **Efficient Rule Processing:** The rule engine should be designed for high-speed matching. This can involve using efficient regular expression engines and compiling rules into a faster format.
|
||||
* **Caching of Rules and Results:** Caching frequently used rules and the results of certain checks can reduce redundant processing.
|
||||
* **Asynchronous Logging:** Decoupling the logging of events from the request processing path can prevent logging operations from becoming a bottleneck.
|
||||
* **Hardware Acceleration:** Utilizing specialized hardware for tasks like SSL/TLS decryption and pattern matching can significantly improve performance.
|
||||
* **Selective Inspection:** Applying more resource-intensive rules only to specific parts of the application or to traffic that has already been flagged as suspicious.
|
||||
|
||||
### 8. Evasion Techniques Attackers Use
|
||||
|
||||
Attackers constantly devise new ways to bypass WAFs. A robust WAF design must anticipate and counter these techniques:
|
||||
|
||||
* **Encoding and Obfuscation:** Attackers use various encoding schemes (URL encoding, Base64, etc.) to disguise malicious payloads. A WAF must normalize and decode all input before inspection.
|
||||
* **Polyglot Payloads:** Crafting payloads that are valid in multiple contexts (e.g., a string that is both valid HTML and JavaScript) to confuse WAF parsers.
|
||||
* **HTTP Parameter Pollution (HPP):** Sending multiple parameters with the same name to see how the WAF and the backend application handle the duplicate parameters.
|
||||
* **Request Smuggling:** Exploiting discrepancies in how a WAF and the backend server parse `Content-Length` and `Transfer-Encoding` headers to smuggle malicious requests.
|
||||
* **Case Variation:** Using different cases for characters in attack payloads (e.g., `SeLeCt` instead of `select`) to bypass case-sensitive regex patterns.
|
||||
* **Whitespace and Comment Obfuscation:** Inserting whitespace characters or comments within attack payloads to break up known signatures.
|
||||
|
||||
### 9. Example Rule Library
|
||||
|
||||
A baseline rule library should provide broad protection against common attacks.
|
||||
|
||||
#### **SQL Injection**
|
||||
|
||||
```
|
||||
rule "SQL Injection - Common Keywords" {
|
||||
match {
|
||||
request_body contains_any_case [
|
||||
"select ", " from ", " where ",
|
||||
"union all select", "insert into", "update set", "delete from"
|
||||
]
|
||||
}
|
||||
action {
|
||||
block
|
||||
log "High-risk SQL keywords detected in request body"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### **Cross-Site Scripting (XSS)**
|
||||
|
||||
```
|
||||
rule "XSS - Script Tags" {
|
||||
match {
|
||||
any_parameter contains_regex "<script.*>"
|
||||
}
|
||||
action {
|
||||
block
|
||||
log "Script tags detected in a parameter"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### **Cross-Site Request Forgery (CSRF)**
|
||||
|
||||
```
|
||||
rule "CSRF - Missing Anti-CSRF Token" {
|
||||
match {
|
||||
request_method == "POST"
|
||||
and not header_exists "X-CSRF-Token"
|
||||
}
|
||||
action {
|
||||
block
|
||||
log "POST request missing X-CSRF-Token header"
|
||||
}
|
||||
}```
|
||||
|
||||
**Key Architectural Trends:**
|
||||
|
||||
* **Cloud-Native and Containerized Deployment:** Traditional appliance-based WAFs are ill-suited for ephemeral, containerized environments. Modern WAFs are designed as lightweight, containerized services that can be deployed directly within a Kubernetes cluster, often as an ingress controller or a sidecar proxy. This "close-to-the-application" deployment model allows for granular, microservice-specific security policies and scales automatically with the application. Azure's Application Gateway for Containers, for instance, introduces a Kubernetes-native `WebApplicationFirewallPolicy` custom resource, allowing WAF policies to be defined and scoped directly within the cluster.
|
||||
* **Decoupled Control and Data Planes:** To enhance agility and performance, leading WAFs now separate the control plane (policy management) from the data plane (traffic inspection). This allows security teams to update policies without impacting the data path, preventing latency and enabling faster deployment cycles, a core tenet of DevOps and CI/CD pipelines.
|
||||
* **AI and Machine Learning at the Core:** The most significant evolution is the integration of Artificial Intelligence (AI) and Machine Learning (ML) to move beyond reactive signature-based detection. These systems establish a baseline of normal application behavior to detect anomalies and identify zero-day attacks that have no known signature.
|
||||
|
||||
### 2. OWASP Top 10 Detection Strategies (2025 Release Candidate Perspective)
|
||||
|
||||
The upcoming OWASP Top 10 for 2025, currently in its release candidate stage, reflects the changing threat landscape. A modern WAF must address these evolving risks with sophisticated detection strategies.
|
||||
|
||||
#### **A01:2025 - Broken Access Control (Unchanged at #1)**
|
||||
|
||||
Still the most prevalent risk, modern WAFs address this not just with URL-based rules, but through stateful analysis.
|
||||
|
||||
* **Stateful Policy Enforcement:** By tracking user sessions and understanding application logic, a WAF can detect when a user attempts to access resources outside their prescribed role, even if the URL pattern seems benign. This is a significant advancement over stateless pattern matching.
|
||||
* **API Endpoint Authorization:** Modern WAFs can parse and enforce access control policies on a per-endpoint basis for REST and GraphQL APIs, ensuring that only authorized users can perform specific mutations or queries.
|
||||
|
||||
#### **A02:2025 - Security Misconfiguration (Moved up from #5)**
|
||||
|
||||
This has risen in prominence with the complexity of cloud environments.
|
||||
|
||||
* **Cloud Security Posture Management (CSPM) Integration:** A WAF can integrate with CSPM tools to receive real-time updates about misconfigurations in the underlying cloud infrastructure (e.g., publicly exposed S3 buckets) and apply virtual patching rules to block attempts to exploit them.
|
||||
* **Automated Header Enforcement:** WAFs can be configured to automatically enforce secure HTTP headers (like Content-Security-Policy, Strict-Transport-Security) and block responses that lack them, ensuring a consistent security posture.
|
||||
|
||||
#### **A03:2025 - Software Supply Chain Failures (New Category)**
|
||||
|
||||
This is a new and critical area of focus.
|
||||
|
||||
* **Virtual Patching for Known Vulnerabilities:** When a vulnerability is discovered in a third-party library, a WAF can immediately deploy a virtual patch to block exploit attempts, giving developers time to update the component without exposing the application to risk.
|
||||
* **Threat Intelligence Integration:** Modern WAFs integrate with threat intelligence feeds to be aware of newly discovered vulnerabilities in open-source components and automatically apply relevant blocking rules.
|
||||
|
||||
#### **A05:2025 - Injection (Moved down from #3)**
|
||||
|
||||
While its ranking has slightly decreased, injection remains a severe threat that requires more than just basic regex.
|
||||
|
||||
* **Semantic Analysis:** Instead of just looking for keywords like `' OR '1'='1`, AI-powered WAFs can parse and understand the structure of a SQL query or a command. This allows them to detect syntactically correct but malicious queries that would bypass simple pattern matching.
|
||||
* **NoSQL and GraphQL Injection:** Modern WAFs have specific parsers for NoSQL databases and GraphQL, allowing them to detect injection attacks that are unique to these technologies.
|
||||
|
||||
### 3. Advanced Detection Techniques: Beyond Regex
|
||||
|
||||
* **Behavioral Analysis and Anomaly Detection:** This is the cornerstone of modern WAFs. By building a high-dimensional model of normal user behavior (including session duration, request frequency, and typical data patterns), the WAF can identify outliers that indicate an attack. This is highly effective against automated threats and zero-day exploits.
|
||||
* **Machine Learning Models:**
|
||||
* **Supervised Learning:** Trained on vast datasets of labeled malicious and benign traffic, models like Support Vector Machines (SVMs) and Random Forests can accurately classify known attack types.
|
||||
* **Unsupervised Learning:** Techniques like Isolation Forests and K-Means clustering are used to find anomalies without pre-labeled data, which is crucial for detecting novel attacks.
|
||||
* **Deep Learning:** For complex sequence-based attacks, models like Long Short-Term Memory (LSTM) networks are being used to analyze the sequence of characters in a payload, providing a deeper understanding of its intent.
|
||||
* **Threat Intelligence Integration:** Modern WAFs consume real-time threat intelligence feeds to block traffic from known malicious IPs, botnets, and anonymizing proxies, proactively reducing the attack surface.
|
||||
|
||||
### 4. False Positive Reduction: The Modern Approach
|
||||
|
||||
Minimizing the blocking of legitimate traffic is paramount.
|
||||
|
||||
* **Adaptive Learning and Automated Tuning:** The WAF continuously learns from traffic patterns to refine its rules and reduce false positives over time. This moves away from manual rule tuning, which is often slow and error-prone.
|
||||
* **Risk-Based Scoring:** Instead of a binary block/allow decision, each request is assigned a risk score. Low-score requests pass, high-score requests are blocked, and medium-score requests can be challenged with a CAPTCHA or subjected to more intense scrutiny.
|
||||
* **Context-Aware Policies:** Security rules can be fine-tuned based on the application's specific behavior. For example, a rule that blocks special characters might be relaxed for a specific form field where such characters are expected.
|
||||
|
||||
### 5. Custom Rule DSL Design: Moving Past ModSecurity
|
||||
|
||||
With ModSecurity reaching its end-of-life, the focus has shifted to more developer-friendly and expressive rule languages.
|
||||
|
||||
* **YAML/JSON-Based Syntax:** Modern rule formats are often based on human-readable formats like YAML or JSON, which are easy to parse and integrate into CI/CD pipelines.
|
||||
* **Expressive and Composable Rules:** The DSL should allow for complex logic, combining multiple conditions with `AND/OR` operators and referencing dynamic data like threat intelligence feeds.
|
||||
* **GitOps-Friendly:** Storing WAF policies as code in a Git repository allows for version control, peer review, and automated deployment, fully integrating security into the DevOps workflow.
|
||||
|
||||
### 6. Performance Optimization in the Modern Era
|
||||
|
||||
* **High-Performance Rule Engines:** Modern WAF engines are designed for speed, often using compiled rule sets and efficient algorithms to minimize latency.
|
||||
* **Hardware Offloading:** For high-traffic environments, WAFs can leverage hardware acceleration for computationally expensive tasks like SSL/TLS decryption.
|
||||
* **Edge Deployment:** Deploying the WAF at the edge, as part of a CDN, allows for malicious traffic to be blocked before it ever reaches the origin server, improving both security and performance.
|
||||
|
||||
### 7. The Latest in WAF Evasion Techniques
|
||||
|
||||
Attackers are constantly innovating to bypass WAFs.
|
||||
|
||||
* **HTTP/2 and HTTP/3 Based Attacks:** The newer HTTP protocols introduce complexities that can be exploited to smuggle requests or desynchronize how a WAF and a backend server interpret a request.
|
||||
* **Payload Obfuscation and Encoding:** Attackers use a variety of encoding techniques, including double encoding and mixing cases, to bypass WAFs that don't properly normalize input.
|
||||
* **Logical Flaws:** Exploiting business logic flaws, such as race conditions or Insecure Direct Object References (IDOR), can completely bypass WAFs that are focused on syntax rather than logic.
|
||||
* **Payload Fragmentation:** Splitting malicious payloads across multiple HTTP requests or IP fragments can make it difficult for a WAF to detect the attack.
|
||||
|
||||
### **OWASP Top 10 Detection Strategies (Detailed)**
|
||||
|
||||
This section provides a more detailed look at modern detection strategies for the OWASP Top 10.
|
||||
|
||||
* **A01: Broken Access Control:**
|
||||
* **Detection:** Monitor for users attempting to access API endpoints or data objects that are not associated with their session or role. Use anomaly detection to flag when a user's behavior deviates significantly from their typical access patterns.
|
||||
* **Example:** A user with a `customer` role suddenly attempting to access an `/admin` endpoint would be flagged, even if they have a valid session token.
|
||||
* **A02: Cryptographic Failures:**
|
||||
* **Detection:** Scan HTTP headers for weak TLS/SSL ciphers and protocols. Inspect responses for sensitive data (e.g., credit card numbers, API keys) that is not properly masked or encrypted. Enforce HSTS to prevent protocol downgrade attacks.
|
||||
* **A03: Injection:**
|
||||
* **Detection:** Use a combination of signature-based detection for common injection patterns and ML-based analysis to understand the grammatical structure of queries. A query that is syntactically valid but semantically anomalous (e.g., a query in a username field) would be flagged.
|
||||
* **A04: Insecure Design:**
|
||||
* **Detection:** While hard to detect directly, a WAF can be configured to enforce secure design principles. For example, it can block requests that attempt to enumerate resource IDs sequentially, a common tactic for exploiting IDOR vulnerabilities.
|
||||
* **A05: Security Misconfiguration:**
|
||||
* **Detection:** Continuously check for the presence and correctness of security headers. Block requests that attempt to access sensitive files (e.g., `.git`, `.env`) that may have been accidentally exposed.
|
||||
* **A06: Vulnerable and Outdated Components:**
|
||||
* **Detection:** Integrate with a Software Composition Analysis (SCA) tool to be aware of the components used by the application. The WAF can then apply virtual patches for any known vulnerabilities in those components.
|
||||
* **A07: Identification and Authentication Failures:**
|
||||
* **Detection:** Implement rate-limiting on login endpoints to prevent brute-force attacks. Use device fingerprinting and behavioral analysis to detect credential stuffing attempts, where an attacker uses stolen credentials from other breaches.
|
||||
* **A08: Software and Data Integrity Failures:**
|
||||
* **Detection:** For applications that use serialized data, the WAF can inspect the serialized objects for known malicious gadgets or patterns that could lead to insecure deserialization.
|
||||
* **A09: Security Logging and Monitoring Failures:**
|
||||
* **Detection:** While primarily a backend concern, a WAF is a critical source of logs. A modern WAF will provide detailed, structured logs (e.g., in JSON format) that can be easily ingested by a SIEM for analysis and alerting.
|
||||
* **A10: Server-Side Request Forgery (SSRF):**
|
||||
* **Detection:** Maintain a strict allowlist of domains and IP addresses that the application is allowed to make requests to. Block any requests that attempt to access internal IP ranges or metadata services (e.g., `169.254.169.254`).
|
||||
|
||||
### **Rule Format Specification (Modern, YAML-based)**
|
||||
|
||||
This proposed format is designed to be human-readable, expressive, and CI/CD-friendly.
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: "SQL Injection Prevention Rule"
|
||||
description: "Blocks common SQL injection patterns with a high confidence score."
|
||||
id: "sql-001"
|
||||
severity: "critical"
|
||||
enabled: true
|
||||
|
||||
# Conditions under which the rule is evaluated.
|
||||
match:
|
||||
- # This rule will only run on paths that accept user input.
|
||||
path:
|
||||
- "/api/v1/search"
|
||||
- "/api/v1/products"
|
||||
methods:
|
||||
- "POST"
|
||||
- "GET"
|
||||
|
||||
- # A list of conditions that are ANDed together.
|
||||
conditions:
|
||||
- # Check for SQL keywords in any part of the request body.
|
||||
target: "request.body"
|
||||
operator: "contains_any_case"
|
||||
values:
|
||||
- "select "
|
||||
- " from "
|
||||
- " where "
|
||||
- "union all select"
|
||||
- "insert into"
|
||||
- "update set"
|
||||
- "delete from"
|
||||
- # Also check for suspicious special characters.
|
||||
target: "request.body"
|
||||
operator: "contains_any"
|
||||
values:
|
||||
- "--"
|
||||
- ";"
|
||||
- "'"
|
||||
|
||||
# The action to take if the conditions are met.
|
||||
action:
|
||||
type: "block"
|
||||
response_code: 403
|
||||
log: true
|
||||
message: "SQL Injection attempt detected and blocked."
|
||||
```
|
||||
|
||||
### **Example Rule Library**
|
||||
|
||||
#### **Cross-Site Scripting (XSS) Protection**
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: "XSS Protection - Script Tags"
|
||||
description: "Blocks requests containing script tags in parameters."
|
||||
id: "xss-001"
|
||||
severity: "high"
|
||||
enabled: true
|
||||
|
||||
match:
|
||||
- conditions:
|
||||
- target: "request.all_params"
|
||||
operator: "matches_regex"
|
||||
value: "<script.*>"
|
||||
|
||||
action:
|
||||
type: "block"
|
||||
response_code: 403
|
||||
log: true
|
||||
message: "XSS attempt with script tags detected."
|
||||
```
|
||||
|
||||
#### **API - Broken Object Level Authorization (BOLA/IDOR)**
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: "API - BOLA/IDOR Prevention"
|
||||
description: "Prevents users from accessing resources that do not belong to them."
|
||||
id: "api-bola-001"
|
||||
severity: "critical"
|
||||
enabled: true
|
||||
|
||||
match:
|
||||
- path:
|
||||
- "/api/v1/users/{userId}/profile"
|
||||
methods:
|
||||
- "GET"
|
||||
- "PUT"
|
||||
|
||||
- conditions:
|
||||
- # This assumes the user's ID is stored in a JWT claim named 'sub'.
|
||||
target: "request.path.userId"
|
||||
operator: "not_equals"
|
||||
value: "jwt.claims.sub"
|
||||
|
||||
action:
|
||||
type: "block"
|
||||
response_code: 403
|
||||
log: true
|
||||
message: "Attempted to access another user's profile."
|
||||
```
|
||||
|
||||
#### **Rate Limiting for Brute-Force Prevention**
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: "Login Brute-Force Prevention"
|
||||
description: "Rate limits login attempts from a single IP address."
|
||||
id: "rate-limit-001"
|
||||
severity: "medium"
|
||||
enabled: true
|
||||
|
||||
match:
|
||||
- path:
|
||||
- "/login"
|
||||
methods:
|
||||
- "POST"
|
||||
|
||||
action:
|
||||
type: "rate_limit"
|
||||
# Allow 5 requests per IP every 1 minute.
|
||||
limit: 5
|
||||
period: 60 # in seconds
|
||||
key_by: "ip"
|
||||
log: true
|
||||
message: "Rate limit exceeded on login endpoint."
|
||||
```
|
||||
|
|
@ -29,7 +29,7 @@ async def test_websocket(host="localhost", port=8081):
|
|||
print(f"\n[TEST] WebSocket: {uri}")
|
||||
|
||||
try:
|
||||
async with websockets.connect(uri, extra_headers={"Host": "ws.localhost"}) as ws:
|
||||
async with websockets.connect(uri, additional_headers={"Host": "ws.localhost"}) as ws:
|
||||
print("[OK] WebSocket connected")
|
||||
|
||||
await ws.send("ping")
|
||||
|
|
|
|||
|
|
@ -0,0 +1,301 @@
|
|||
# =============================================================================
|
||||
# AngelaMos | 2026
|
||||
# Justfile - Aenebris (Haskell Reverse Proxy)
|
||||
# =============================================================================
|
||||
|
||||
set export
|
||||
set shell := ["bash", "-uc"]
|
||||
set windows-shell := ["powershell.exe", "-NoLogo", "-Command"]
|
||||
|
||||
project := file_name(justfile_directory())
|
||||
version := `git describe --tags --always 2>/dev/null || echo "dev"`
|
||||
default_config := "examples/config.yaml"
|
||||
log_file := "aenebris.log"
|
||||
backend_log := "backend.log"
|
||||
|
||||
# =============================================================================
|
||||
# Default
|
||||
# =============================================================================
|
||||
|
||||
default:
|
||||
@just --list --unsorted
|
||||
|
||||
# =============================================================================
|
||||
# Build
|
||||
# =============================================================================
|
||||
|
||||
[group('build')]
|
||||
build:
|
||||
stack build
|
||||
|
||||
[group('build')]
|
||||
build-fast:
|
||||
stack build --fast
|
||||
|
||||
[group('build')]
|
||||
build-watch:
|
||||
stack build --fast --file-watch
|
||||
|
||||
[group('build')]
|
||||
build-pedantic:
|
||||
stack build --pedantic
|
||||
|
||||
[group('build')]
|
||||
deps:
|
||||
stack build --only-dependencies
|
||||
|
||||
[group('build')]
|
||||
install:
|
||||
stack install
|
||||
|
||||
[group('build')]
|
||||
ghci *ARGS:
|
||||
stack ghci {{ARGS}}
|
||||
|
||||
# =============================================================================
|
||||
# Run (Proxy Lifecycle)
|
||||
# =============================================================================
|
||||
|
||||
[group('run')]
|
||||
run config=default_config:
|
||||
stack run -- {{config}}
|
||||
|
||||
[group('run')]
|
||||
run-bg config=default_config:
|
||||
@echo "Starting Aenebris with {{config}}..."
|
||||
@nohup stack run -- {{config}} > {{log_file}} 2>&1 &
|
||||
@sleep 2
|
||||
@echo "PID: $(pgrep -f '[a]enebris examples' | head -1)"
|
||||
@echo "Use 'just logs' to follow output, 'just stop' to halt"
|
||||
|
||||
[group('run')]
|
||||
https:
|
||||
stack run -- examples/config-https.yaml
|
||||
|
||||
[group('run')]
|
||||
sni:
|
||||
stack run -- examples/config-sni.yaml
|
||||
|
||||
[group('run')]
|
||||
lb:
|
||||
stack run -- examples/config-loadbalancing.yaml
|
||||
|
||||
[group('run')]
|
||||
advanced:
|
||||
stack run -- examples/config-advanced.yaml
|
||||
|
||||
[group('run')]
|
||||
ws-sse:
|
||||
stack run -- examples/websockets/config_ws_sse.yaml
|
||||
|
||||
[group('run')]
|
||||
stop:
|
||||
@pkill -f '[a]enebris examples' && echo "Proxy stopped" || echo "Proxy not running"
|
||||
|
||||
[group('run')]
|
||||
restart config=default_config:
|
||||
@just stop
|
||||
@sleep 1
|
||||
@just run-bg {{config}}
|
||||
|
||||
[group('run')]
|
||||
logs:
|
||||
tail -f {{log_file}}
|
||||
|
||||
[group('run')]
|
||||
status:
|
||||
@pgrep -af '[a]enebris examples' || echo "Proxy not running"
|
||||
|
||||
# =============================================================================
|
||||
# Backends (Python Test Servers)
|
||||
# =============================================================================
|
||||
|
||||
[group('backend')]
|
||||
backend:
|
||||
python3 examples/test_backend.py
|
||||
|
||||
[group('backend')]
|
||||
backend-bg:
|
||||
@nohup python3 examples/test_backend.py > {{backend_log}} 2>&1 &
|
||||
@sleep 1
|
||||
@echo "Backend on :8000 (PID: $(pgrep -f '[t]est_backend.py' | head -1))"
|
||||
|
||||
[group('backend')]
|
||||
backends:
|
||||
@bash examples/start_backends.sh
|
||||
|
||||
[group('backend')]
|
||||
ws-server:
|
||||
python3 examples/websockets/ws_echo_server.py
|
||||
|
||||
[group('backend')]
|
||||
sse-server:
|
||||
python3 examples/websockets/sse_server.py
|
||||
|
||||
[group('backend')]
|
||||
kill-backends:
|
||||
-@pkill -f '[t]est_backend\.py'
|
||||
-@pkill -f '[t]est_backend_multi\.py'
|
||||
-@pkill -f '[w]s_echo_server\.py'
|
||||
-@pkill -f '[s]se_server\.py'
|
||||
@echo "Test backends stopped"
|
||||
|
||||
[group('backend')]
|
||||
kill-all:
|
||||
@just stop
|
||||
@just kill-backends
|
||||
|
||||
# =============================================================================
|
||||
# Tests (HSpec + Integration)
|
||||
# =============================================================================
|
||||
|
||||
[group('test')]
|
||||
test *ARGS:
|
||||
stack test {{ARGS}}
|
||||
|
||||
[group('test')]
|
||||
test-fast:
|
||||
stack test --fast
|
||||
|
||||
[group('test')]
|
||||
test-watch:
|
||||
stack test --fast --file-watch
|
||||
|
||||
[group('test')]
|
||||
test-match PATTERN:
|
||||
stack test --test-arguments='--match="{{PATTERN}}"'
|
||||
|
||||
[group('test')]
|
||||
test-cov:
|
||||
stack test --coverage
|
||||
|
||||
[group('test')]
|
||||
test-list:
|
||||
stack test --test-arguments='--dry-run'
|
||||
|
||||
[group('test')]
|
||||
test-config:
|
||||
@just test-match "Config"
|
||||
|
||||
[group('test')]
|
||||
test-lb:
|
||||
@just test-match "LoadBalancer"
|
||||
|
||||
[group('test')]
|
||||
test-backend-unit:
|
||||
@just test-match "Backend"
|
||||
|
||||
[group('test')]
|
||||
test-security:
|
||||
@just test-match "Security headers"
|
||||
|
||||
[group('test')]
|
||||
test-redirect:
|
||||
@just test-match "HTTPS redirect"
|
||||
|
||||
[group('test')]
|
||||
test-ratelimit:
|
||||
@just test-match "RateLimit"
|
||||
|
||||
[group('test')]
|
||||
test-ddos:
|
||||
stack test --test-arguments='--match="EarlyData" --match="IPJail" --match="MemoryShed" --match="ConnLimit"'
|
||||
|
||||
[group('test')]
|
||||
test-ja4h:
|
||||
@just test-match "JA4H fingerprint"
|
||||
|
||||
[group('test')]
|
||||
test-waf:
|
||||
@just test-match "WAF"
|
||||
|
||||
[group('test')]
|
||||
test-honeypot:
|
||||
@just test-match "Honeypot"
|
||||
|
||||
[group('test')]
|
||||
test-geo:
|
||||
@just test-match "Geo/ASN"
|
||||
|
||||
[group('test')]
|
||||
test-ml:
|
||||
stack test --test-arguments='--match="ML.Features" --match="ML.Model"'
|
||||
|
||||
[group('test')]
|
||||
ws-test host='localhost' port='8081':
|
||||
python3 examples/websockets/test_client.py {{host}} {{port}}
|
||||
|
||||
# =============================================================================
|
||||
# TLS / Certificates
|
||||
# =============================================================================
|
||||
|
||||
[group('tls')]
|
||||
gen-certs:
|
||||
bash examples/generate-test-certs.sh
|
||||
|
||||
[group('tls')]
|
||||
clean-certs:
|
||||
-rm -rf examples/certs
|
||||
@echo "Test certs removed (regenerate with 'just gen-certs')"
|
||||
|
||||
[group('tls')]
|
||||
tls-headers:
|
||||
curl -kI https://localhost:8443/
|
||||
|
||||
[group('tls')]
|
||||
tls13:
|
||||
openssl s_client -connect localhost:8443 -tls1_3 < /dev/null 2>&1 | grep -E "Protocol|Cipher"
|
||||
|
||||
[group('tls')]
|
||||
sni-curl domain='api.localhost':
|
||||
curl -k -H "Host: {{domain}}" -v https://localhost:8443/
|
||||
|
||||
# =============================================================================
|
||||
# CI / Quality
|
||||
# =============================================================================
|
||||
|
||||
[group('ci')]
|
||||
ci: build test
|
||||
|
||||
[group('ci')]
|
||||
check: build
|
||||
|
||||
# =============================================================================
|
||||
# Setup
|
||||
# =============================================================================
|
||||
|
||||
[group('setup')]
|
||||
setup:
|
||||
@echo "First-time setup: building deps + generating test certs..."
|
||||
stack build --only-dependencies
|
||||
bash examples/generate-test-certs.sh
|
||||
@echo "Setup complete - try 'just backends' then 'just run'"
|
||||
|
||||
# =============================================================================
|
||||
# Utilities
|
||||
# =============================================================================
|
||||
|
||||
[group('util')]
|
||||
info:
|
||||
@echo "Project: {{project}}"
|
||||
@echo "Version: {{version}}"
|
||||
@echo "OS: {{os()}} ({{arch()}})"
|
||||
@stack --version 2>/dev/null || true
|
||||
@ghc --version 2>/dev/null || true
|
||||
|
||||
[group('util')]
|
||||
clean:
|
||||
stack clean
|
||||
-rm -f {{log_file}} {{backend_log}} nohup.out
|
||||
-rm -f /tmp/backend-8000.log /tmp/backend-8001.log /tmp/backend-8002.log
|
||||
@echo "Build artifacts and logs cleaned"
|
||||
|
||||
[group('util')]
|
||||
[confirm("Remove .stack-work, dist-newstyle, certs, and all logs?")]
|
||||
nuke:
|
||||
-stack purge
|
||||
-rm -rf .stack-work dist-newstyle
|
||||
-rm -rf examples/certs
|
||||
-rm -f {{log_file}} {{backend_log}} nohup.out
|
||||
-rm -f /tmp/backend-*.log
|
||||
@echo "Nuke complete"
|
||||
|
|
@ -11,10 +11,15 @@ module Aenebris.Config
|
|||
, HealthCheck(..)
|
||||
, Route(..)
|
||||
, PathRoute(..)
|
||||
, DDoSConfig(..)
|
||||
, defaultDDoSConfig
|
||||
, loadConfig
|
||||
, validateConfig
|
||||
) where
|
||||
|
||||
import Aenebris.Honeypot (HoneypotConfigYaml)
|
||||
import Aenebris.Geo (GeoConfigYaml)
|
||||
|
||||
import Control.Monad (when, forM_)
|
||||
import Data.Aeson
|
||||
import Data.Text (Text)
|
||||
|
|
@ -28,6 +33,10 @@ data Config = Config
|
|||
, configListen :: [ListenConfig]
|
||||
, configUpstreams :: [Upstream]
|
||||
, configRoutes :: [Route]
|
||||
, configRateLimit :: Maybe Text
|
||||
, configDDoS :: Maybe DDoSConfig
|
||||
, configHoneypot :: Maybe HoneypotConfigYaml
|
||||
, configGeo :: Maybe GeoConfigYaml
|
||||
} deriving (Show, Eq, Generic)
|
||||
|
||||
instance FromJSON Config where
|
||||
|
|
@ -36,6 +45,47 @@ instance FromJSON Config where
|
|||
<*> v .: "listen"
|
||||
<*> v .: "upstreams"
|
||||
<*> v .: "routes"
|
||||
<*> v .:? "rate_limit"
|
||||
<*> v .:? "ddos"
|
||||
<*> v .:? "honeypot"
|
||||
<*> v .:? "geo"
|
||||
|
||||
data DDoSConfig = DDoSConfig
|
||||
{ ddosEarlyDataReject :: Bool
|
||||
, ddosPerIPConnections :: Maybe Int
|
||||
, ddosMemoryShedBytes :: Maybe Integer
|
||||
, ddosMemoryShedHighWater :: Maybe Double
|
||||
, ddosMaxConcurrentStreams :: Maybe Int
|
||||
, ddosMaxHeaderBytes :: Maybe Int
|
||||
, ddosSlowlorisSeconds :: Maybe Int
|
||||
, ddosJailCooldownSeconds :: Maybe Int
|
||||
, ddosReusePort :: Bool
|
||||
} deriving (Show, Eq, Generic)
|
||||
|
||||
instance FromJSON DDoSConfig where
|
||||
parseJSON = withObject "DDoSConfig" $ \v -> DDoSConfig
|
||||
<$> v .:? "early_data_reject" .!= True
|
||||
<*> v .:? "per_ip_connections"
|
||||
<*> v .:? "memory_shed_bytes"
|
||||
<*> v .:? "memory_shed_high_water"
|
||||
<*> v .:? "max_concurrent_streams"
|
||||
<*> v .:? "max_header_bytes"
|
||||
<*> v .:? "slowloris_seconds"
|
||||
<*> v .:? "jail_cooldown_seconds"
|
||||
<*> v .:? "reuse_port" .!= False
|
||||
|
||||
defaultDDoSConfig :: DDoSConfig
|
||||
defaultDDoSConfig = DDoSConfig
|
||||
{ ddosEarlyDataReject = True
|
||||
, ddosPerIPConnections = Nothing
|
||||
, ddosMemoryShedBytes = Nothing
|
||||
, ddosMemoryShedHighWater = Nothing
|
||||
, ddosMaxConcurrentStreams = Nothing
|
||||
, ddosMaxHeaderBytes = Nothing
|
||||
, ddosSlowlorisSeconds = Nothing
|
||||
, ddosJailCooldownSeconds = Nothing
|
||||
, ddosReusePort = False
|
||||
}
|
||||
|
||||
-- | Listen port configuration
|
||||
data ListenConfig = ListenConfig
|
||||
|
|
|
|||
|
|
@ -0,0 +1,109 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
ConnLimit.hs
|
||||
-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE RecordWildCards #-}
|
||||
|
||||
module Aenebris.DDoS.ConnLimit
|
||||
( ConnLimiter
|
||||
, ConnLimitConfig(..)
|
||||
, defaultConnLimitConfig
|
||||
, newConnLimiter
|
||||
, tryAcquire
|
||||
, release
|
||||
, currentCount
|
||||
, connLimitOnOpen
|
||||
, connLimitOnClose
|
||||
, defaultPerIPLimit
|
||||
, ipBytesFromSockAddr
|
||||
) where
|
||||
|
||||
import Control.Concurrent.STM
|
||||
( STM
|
||||
, TVar
|
||||
, atomically
|
||||
, modifyTVar'
|
||||
, newTVarIO
|
||||
, readTVar
|
||||
, writeTVar
|
||||
)
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString.Char8 as BS8
|
||||
import Data.Map.Strict (Map)
|
||||
import qualified Data.Map.Strict as Map
|
||||
import Network.Socket
|
||||
( HostAddress6
|
||||
, SockAddr(..)
|
||||
, hostAddress6ToTuple
|
||||
, hostAddressToTuple
|
||||
)
|
||||
import Numeric (showHex)
|
||||
import Text.Printf (printf)
|
||||
|
||||
defaultPerIPLimit :: Int
|
||||
defaultPerIPLimit = 16
|
||||
|
||||
data ConnLimitConfig = ConnLimitConfig
|
||||
{ clcPerIPLimit :: !Int
|
||||
} deriving (Show, Eq)
|
||||
|
||||
defaultConnLimitConfig :: ConnLimitConfig
|
||||
defaultConnLimitConfig = ConnLimitConfig { clcPerIPLimit = defaultPerIPLimit }
|
||||
|
||||
data ConnLimiter = ConnLimiter
|
||||
{ clCounts :: TVar (Map ByteString Int)
|
||||
, clConfig :: !ConnLimitConfig
|
||||
}
|
||||
|
||||
newConnLimiter :: ConnLimitConfig -> IO ConnLimiter
|
||||
newConnLimiter cfg = do
|
||||
tv <- newTVarIO Map.empty
|
||||
pure ConnLimiter { clCounts = tv, clConfig = cfg }
|
||||
|
||||
tryAcquire :: ConnLimiter -> ByteString -> STM Bool
|
||||
tryAcquire ConnLimiter{..} ip = do
|
||||
m <- readTVar clCounts
|
||||
let current = Map.findWithDefault 0 ip m
|
||||
limit = clcPerIPLimit clConfig
|
||||
if current >= limit
|
||||
then pure False
|
||||
else do
|
||||
writeTVar clCounts (Map.insert ip (current + 1) m)
|
||||
pure True
|
||||
|
||||
release :: ConnLimiter -> ByteString -> STM ()
|
||||
release ConnLimiter{..} ip =
|
||||
modifyTVar' clCounts (Map.update decrement ip)
|
||||
where
|
||||
decrement n
|
||||
| n <= 1 = Nothing
|
||||
| otherwise = Just (n - 1)
|
||||
|
||||
currentCount :: ConnLimiter -> ByteString -> STM Int
|
||||
currentCount ConnLimiter{..} ip = do
|
||||
m <- readTVar clCounts
|
||||
pure (Map.findWithDefault 0 ip m)
|
||||
|
||||
connLimitOnOpen :: ConnLimiter -> SockAddr -> IO Bool
|
||||
connLimitOnOpen cl sa = atomically (tryAcquire cl (ipBytesFromSockAddr sa))
|
||||
|
||||
connLimitOnClose :: ConnLimiter -> SockAddr -> IO ()
|
||||
connLimitOnClose cl sa = atomically (release cl (ipBytesFromSockAddr sa))
|
||||
|
||||
ipBytesFromSockAddr :: SockAddr -> ByteString
|
||||
ipBytesFromSockAddr (SockAddrInet _ ha) =
|
||||
let (a, b, c, d) = hostAddressToTuple ha
|
||||
in BS8.pack (printf "%d.%d.%d.%d" a b c d)
|
||||
ipBytesFromSockAddr (SockAddrInet6 _ _ ha6 _) = v6Bytes ha6
|
||||
ipBytesFromSockAddr (SockAddrUnix p) = BS8.pack ("unix:" <> p)
|
||||
|
||||
v6Bytes :: HostAddress6 -> ByteString
|
||||
v6Bytes ha =
|
||||
let (a, b, c, d, e, f, g, h) = hostAddress6ToTuple ha
|
||||
in BS8.pack (joinColons (map (`showHex` "") [a, b, c, d, e, f, g, h]))
|
||||
|
||||
joinColons :: [String] -> String
|
||||
joinColons [] = ""
|
||||
joinColons [x] = x
|
||||
joinColons (x : xs) = x <> ":" <> joinColons xs
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
EarlyData.hs
|
||||
-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Aenebris.DDoS.EarlyData
|
||||
( earlyDataGuard
|
||||
, isIdempotent
|
||||
, isEarlyData
|
||||
, status425
|
||||
) where
|
||||
|
||||
import Data.ByteString (ByteString)
|
||||
import Network.HTTP.Types (Status, mkStatus, methodGet, methodHead)
|
||||
import Network.Wai
|
||||
( Middleware
|
||||
, Request
|
||||
, requestHeaders
|
||||
, requestMethod
|
||||
, responseLBS
|
||||
)
|
||||
|
||||
status425 :: Status
|
||||
status425 = mkStatus 425 "Too Early"
|
||||
|
||||
isIdempotent :: Request -> Bool
|
||||
isIdempotent req = requestMethod req == methodGet || requestMethod req == methodHead
|
||||
|
||||
isEarlyData :: Request -> Bool
|
||||
isEarlyData req = lookup "Early-Data" (requestHeaders req) == Just earlyDataHeaderValue
|
||||
|
||||
earlyDataHeaderValue :: ByteString
|
||||
earlyDataHeaderValue = "1"
|
||||
|
||||
earlyDataGuard :: Middleware
|
||||
earlyDataGuard app req respond
|
||||
| isEarlyData req && not (isIdempotent req) =
|
||||
respond $ responseLBS status425
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("Cache-Control", "no-store")
|
||||
]
|
||||
"425 Too Early: non-idempotent method in 0-RTT data (RFC 8470)"
|
||||
| otherwise = app req respond
|
||||
|
|
@ -0,0 +1,108 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
IPJail.hs
|
||||
-}
|
||||
{-# LANGUAGE NumericUnderscores #-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Aenebris.DDoS.IPJail
|
||||
( IPJail
|
||||
, JailedEntry(..)
|
||||
, IPJailConfig(..)
|
||||
, defaultIPJailConfig
|
||||
, newIPJail
|
||||
, jail
|
||||
, isJailed
|
||||
, purgeExpired
|
||||
, startJailSweeper
|
||||
, ipJailMiddleware
|
||||
, defaultJailCooldown
|
||||
, defaultSweepIntervalMicros
|
||||
) where
|
||||
|
||||
import Control.Concurrent (threadDelay)
|
||||
import Control.Concurrent.Async (Async, async)
|
||||
import Control.Concurrent.STM
|
||||
( STM
|
||||
, TVar
|
||||
, atomically
|
||||
, modifyTVar'
|
||||
, newTVarIO
|
||||
, readTVar
|
||||
)
|
||||
import Control.Monad (forever)
|
||||
import Data.ByteString (ByteString)
|
||||
import Data.Map.Strict (Map)
|
||||
import qualified Data.Map.Strict as Map
|
||||
import Data.Time.Clock.POSIX (POSIXTime, getPOSIXTime)
|
||||
import Network.HTTP.Types (status403)
|
||||
import Network.Wai (Middleware, responseLBS)
|
||||
|
||||
import Aenebris.RateLimit (clientIPKey)
|
||||
|
||||
defaultJailCooldown :: POSIXTime
|
||||
defaultJailCooldown = 300
|
||||
|
||||
defaultSweepIntervalMicros :: Int
|
||||
defaultSweepIntervalMicros = 30_000_000
|
||||
|
||||
data JailedEntry = JailedEntry
|
||||
{ jeExpiresAt :: !POSIXTime
|
||||
, jeReason :: !ByteString
|
||||
} deriving (Show, Eq)
|
||||
|
||||
data IPJailConfig = IPJailConfig
|
||||
{ ijcDefaultCooldown :: !POSIXTime
|
||||
, ijcSweepIntervalMicros :: !Int
|
||||
} deriving (Show, Eq)
|
||||
|
||||
defaultIPJailConfig :: IPJailConfig
|
||||
defaultIPJailConfig = IPJailConfig
|
||||
{ ijcDefaultCooldown = defaultJailCooldown
|
||||
, ijcSweepIntervalMicros = defaultSweepIntervalMicros
|
||||
}
|
||||
|
||||
newtype IPJail = IPJail { ijMap :: TVar (Map ByteString JailedEntry) }
|
||||
|
||||
newIPJail :: IO IPJail
|
||||
newIPJail = IPJail <$> newTVarIO Map.empty
|
||||
|
||||
jail :: IPJail -> ByteString -> POSIXTime -> ByteString -> POSIXTime -> STM ()
|
||||
jail (IPJail tv) ip cooldown reason now =
|
||||
modifyTVar' tv (Map.insert ip entry)
|
||||
where
|
||||
entry = JailedEntry { jeExpiresAt = now + cooldown, jeReason = reason }
|
||||
|
||||
isJailed :: IPJail -> ByteString -> POSIXTime -> STM (Maybe JailedEntry)
|
||||
isJailed (IPJail tv) ip now = do
|
||||
m <- readTVar tv
|
||||
case Map.lookup ip m of
|
||||
Just e | jeExpiresAt e > now -> pure (Just e)
|
||||
_ -> pure Nothing
|
||||
|
||||
purgeExpired :: IPJail -> POSIXTime -> STM Int
|
||||
purgeExpired (IPJail tv) now = do
|
||||
m <- readTVar tv
|
||||
let (stale, fresh) = Map.partition (\e -> jeExpiresAt e <= now) m
|
||||
modifyTVar' tv (const fresh)
|
||||
pure (Map.size stale)
|
||||
|
||||
startJailSweeper :: IPJailConfig -> IPJail -> IO (Async ())
|
||||
startJailSweeper cfg j = async $ forever $ do
|
||||
threadDelay (ijcSweepIntervalMicros cfg)
|
||||
now <- getPOSIXTime
|
||||
_ <- atomically (purgeExpired j now)
|
||||
pure ()
|
||||
|
||||
ipJailMiddleware :: IPJail -> Middleware
|
||||
ipJailMiddleware j app req respond = do
|
||||
now <- getPOSIXTime
|
||||
let ip = clientIPKey req
|
||||
jailed <- atomically (isJailed j ip now)
|
||||
case jailed of
|
||||
Just _entry -> respond $ responseLBS status403
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("Cache-Control", "no-store")
|
||||
]
|
||||
"403 Forbidden: source address is temporarily restricted"
|
||||
Nothing -> app req respond
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
MemoryShed.hs
|
||||
-}
|
||||
{-# LANGUAGE NumericUnderscores #-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Aenebris.DDoS.MemoryShed
|
||||
( MemoryShed
|
||||
, MemoryShedConfig(..)
|
||||
, defaultMemoryShedConfig
|
||||
, newMemoryShed
|
||||
, startMemoryShedPoller
|
||||
, memoryShedMiddleware
|
||||
, isShedding
|
||||
, updateShedding
|
||||
, defaultPollIntervalMicros
|
||||
, defaultHighWaterFraction
|
||||
) where
|
||||
|
||||
import Control.Concurrent (threadDelay)
|
||||
import Control.Concurrent.Async (Async, async)
|
||||
import Control.Monad (forever)
|
||||
import Control.Concurrent.STM (STM, TVar, atomically, newTVarIO, readTVar, writeTVar)
|
||||
import Data.Word (Word64)
|
||||
import GHC.Stats (RTSStats(..), GCDetails(..), getRTSStats, getRTSStatsEnabled)
|
||||
import Network.HTTP.Types (status503)
|
||||
import Network.Wai (Middleware, responseLBS)
|
||||
|
||||
defaultPollIntervalMicros :: Int
|
||||
defaultPollIntervalMicros = 1_000_000
|
||||
|
||||
defaultHighWaterFraction :: Double
|
||||
defaultHighWaterFraction = 0.70
|
||||
|
||||
data MemoryShedConfig = MemoryShedConfig
|
||||
{ mscHeapBudgetBytes :: !Word64
|
||||
, mscHighWaterFraction :: !Double
|
||||
, mscPollIntervalMicros :: !Int
|
||||
} deriving (Show, Eq)
|
||||
|
||||
defaultMemoryShedConfig :: Word64 -> MemoryShedConfig
|
||||
defaultMemoryShedConfig budget = MemoryShedConfig
|
||||
{ mscHeapBudgetBytes = budget
|
||||
, mscHighWaterFraction = defaultHighWaterFraction
|
||||
, mscPollIntervalMicros = defaultPollIntervalMicros
|
||||
}
|
||||
|
||||
newtype MemoryShed = MemoryShed { msFlag :: TVar Bool }
|
||||
|
||||
newMemoryShed :: IO MemoryShed
|
||||
newMemoryShed = MemoryShed <$> newTVarIO False
|
||||
|
||||
isShedding :: MemoryShed -> STM Bool
|
||||
isShedding = readTVar . msFlag
|
||||
|
||||
updateShedding :: MemoryShed -> Bool -> STM ()
|
||||
updateShedding (MemoryShed tv) = writeTVar tv
|
||||
|
||||
startMemoryShedPoller :: MemoryShedConfig -> MemoryShed -> IO (Async ())
|
||||
startMemoryShedPoller cfg shed = async $ do
|
||||
enabled <- getRTSStatsEnabled
|
||||
if not enabled
|
||||
then pure ()
|
||||
else forever $ do
|
||||
threadDelay (mscPollIntervalMicros cfg)
|
||||
stats <- getRTSStats
|
||||
let live = gcdetails_live_bytes (gc stats)
|
||||
threshold = floor (fromIntegral (mscHeapBudgetBytes cfg) * mscHighWaterFraction cfg :: Double) :: Word64
|
||||
shedNow = live > threshold
|
||||
atomically (updateShedding shed shedNow)
|
||||
|
||||
memoryShedMiddleware :: MemoryShed -> Middleware
|
||||
memoryShedMiddleware shed app req respond = do
|
||||
shedding <- atomically (isShedding shed)
|
||||
if shedding
|
||||
then respond $ responseLBS status503
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("Retry-After", "1")
|
||||
]
|
||||
"503 Service Unavailable: memory pressure, shedding load"
|
||||
else app req respond
|
||||
|
|
@ -0,0 +1,168 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
JA4H.hs
|
||||
-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Aenebris.Fingerprint.JA4H
|
||||
( JA4H(..)
|
||||
, computeJA4H
|
||||
, renderJA4H
|
||||
, ja4hMiddleware
|
||||
, ja4hHeaderName
|
||||
, methodCode
|
||||
, versionCode
|
||||
, acceptLanguagePrefix
|
||||
, parseCookieNames
|
||||
, parseCookiePairs
|
||||
, emptyHashPlaceholder
|
||||
) where
|
||||
|
||||
import Crypto.Hash (Digest, SHA256, hash)
|
||||
import Data.ByteArray.Encoding (Base(Base16), convertToBase)
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString as BS
|
||||
import qualified Data.ByteString.Char8 as BC
|
||||
import Data.CaseInsensitive (CI)
|
||||
import qualified Data.CaseInsensitive as CI
|
||||
import Data.Char (toLower, isAlphaNum)
|
||||
import Data.List (sortBy, sortOn)
|
||||
import Network.HTTP.Types (HttpVersion, http10, http11, http20, Method)
|
||||
import Network.Wai
|
||||
( Middleware
|
||||
, Request
|
||||
, requestHeaders
|
||||
, requestMethod
|
||||
, httpVersion
|
||||
, mapResponseHeaders
|
||||
)
|
||||
|
||||
data JA4H = JA4H
|
||||
{ ja4hPartA :: !ByteString
|
||||
, ja4hHeaderHash :: !ByteString
|
||||
, ja4hCookieNameHash :: !ByteString
|
||||
, ja4hCookiePairHash :: !ByteString
|
||||
} deriving (Eq, Show)
|
||||
|
||||
emptyHashPlaceholder :: ByteString
|
||||
emptyHashPlaceholder = "000000000000"
|
||||
|
||||
hashTruncated :: ByteString -> ByteString
|
||||
hashTruncated input =
|
||||
let digest :: Digest SHA256
|
||||
digest = hash input
|
||||
hex = convertToBase Base16 digest :: ByteString
|
||||
in BS.take 12 hex
|
||||
|
||||
methodCode :: Method -> ByteString
|
||||
methodCode m = case m of
|
||||
"GET" -> "ge"
|
||||
"POST" -> "po"
|
||||
"PUT" -> "pu"
|
||||
"DELETE" -> "de"
|
||||
"HEAD" -> "he"
|
||||
"OPTIONS" -> "op"
|
||||
"PATCH" -> "pa"
|
||||
"CONNECT" -> "co"
|
||||
_ -> "xx"
|
||||
|
||||
versionCode :: HttpVersion -> ByteString
|
||||
versionCode v
|
||||
| v == http10 = "10"
|
||||
| v == http11 = "11"
|
||||
| v == http20 = "20"
|
||||
| otherwise = "00"
|
||||
|
||||
padTwo :: Int -> ByteString
|
||||
padTwo n
|
||||
| n >= 99 = "99"
|
||||
| n < 10 = BC.pack ('0' : show n)
|
||||
| otherwise = BC.pack (show n)
|
||||
|
||||
cookieHeaderName :: CI ByteString
|
||||
cookieHeaderName = "cookie"
|
||||
|
||||
refererHeaderName :: CI ByteString
|
||||
refererHeaderName = "referer"
|
||||
|
||||
acceptLanguageHeaderName :: CI ByteString
|
||||
acceptLanguageHeaderName = "accept-language"
|
||||
|
||||
lookupCI :: CI ByteString -> [(CI ByteString, ByteString)] -> Maybe ByteString
|
||||
lookupCI k hs = case filter ((== k) . fst) hs of
|
||||
((_, v) : _) -> Just v
|
||||
_ -> Nothing
|
||||
|
||||
acceptLanguagePrefix :: ByteString -> ByteString
|
||||
acceptLanguagePrefix bs =
|
||||
let firstTag = BS.takeWhile (\c -> c /= 0x2c && c /= 0x3b && c /= 0x20) bs
|
||||
lowered = BC.map toLower firstTag
|
||||
cleaned = BC.filter (\c -> isAlphaNum c || c == '-') lowered
|
||||
stripped = BC.filter (/= '-') cleaned
|
||||
padded = stripped <> BC.replicate 4 '0'
|
||||
in BS.take 4 padded
|
||||
|
||||
parseCookieHeader :: ByteString -> [(ByteString, ByteString)]
|
||||
parseCookieHeader raw =
|
||||
let pieces = BC.split ';' raw
|
||||
trimmed = map (BC.dropWhile (== ' ')) pieces
|
||||
nonEmpty = filter (not . BS.null) trimmed
|
||||
in map splitCookie nonEmpty
|
||||
where
|
||||
splitCookie piece =
|
||||
case BC.break (== '=') piece of
|
||||
(name, valueWithEq)
|
||||
| BS.null valueWithEq -> (name, BS.empty)
|
||||
| otherwise -> (name, BS.drop 1 valueWithEq)
|
||||
|
||||
parseCookieNames :: [(CI ByteString, ByteString)] -> [ByteString]
|
||||
parseCookieNames hs =
|
||||
concatMap (map fst . parseCookieHeader . snd)
|
||||
$ filter ((== cookieHeaderName) . fst) hs
|
||||
|
||||
parseCookiePairs :: [(CI ByteString, ByteString)] -> [(ByteString, ByteString)]
|
||||
parseCookiePairs hs =
|
||||
concatMap (parseCookieHeader . snd)
|
||||
$ filter ((== cookieHeaderName) . fst) hs
|
||||
|
||||
computeJA4H :: Request -> JA4H
|
||||
computeJA4H req =
|
||||
let rawHeaders = requestHeaders req
|
||||
method = methodCode (requestMethod req)
|
||||
version = versionCode (httpVersion req)
|
||||
cookieFlag = if any ((== cookieHeaderName) . fst) rawHeaders then "c" else "n"
|
||||
refererFlag = if any ((== refererHeaderName) . fst) rawHeaders then "r" else "n"
|
||||
filteredHeaders =
|
||||
filter (\(k, _) -> k /= cookieHeaderName && k /= refererHeaderName) rawHeaders
|
||||
headerCount = padTwo (length filteredHeaders)
|
||||
langPrefix = maybe "0000" acceptLanguagePrefix
|
||||
$ lookupCI acceptLanguageHeaderName rawHeaders
|
||||
partA = BS.concat [method, version, cookieFlag, refererFlag, headerCount, langPrefix]
|
||||
headerNameList = BS.intercalate "," (map (CI.original . fst) filteredHeaders)
|
||||
headerHash = hashTruncated headerNameList
|
||||
cookieNames = parseCookieNames rawHeaders
|
||||
cookieNameHash =
|
||||
if null cookieNames
|
||||
then emptyHashPlaceholder
|
||||
else hashTruncated (BS.intercalate "," (sortBy compare cookieNames))
|
||||
cookiePairs = parseCookiePairs rawHeaders
|
||||
cookiePairHash =
|
||||
if null cookiePairs
|
||||
then emptyHashPlaceholder
|
||||
else hashTruncated
|
||||
$ BS.intercalate ","
|
||||
$ map (\(n, v) -> n <> "=" <> v)
|
||||
$ sortOn fst cookiePairs
|
||||
in JA4H partA headerHash cookieNameHash cookiePairHash
|
||||
|
||||
renderJA4H :: JA4H -> ByteString
|
||||
renderJA4H (JA4H a b c d) = BS.intercalate "_" [a, b, c, d]
|
||||
|
||||
ja4hHeaderName :: CI ByteString
|
||||
ja4hHeaderName = "x-ja4h"
|
||||
|
||||
ja4hMiddleware :: Middleware
|
||||
ja4hMiddleware app req respond =
|
||||
let fp = renderJA4H (computeJA4H req)
|
||||
addFp = mapResponseHeaders ((ja4hHeaderName, fp) :)
|
||||
in app req (respond . addFp)
|
||||
|
|
@ -0,0 +1,360 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
Geo.hs
|
||||
-}
|
||||
{-# LANGUAGE BangPatterns #-}
|
||||
{-# LANGUAGE DeriveGeneric #-}
|
||||
{-# LANGUAGE NumericUnderscores #-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE RecordWildCards #-}
|
||||
|
||||
module Aenebris.Geo
|
||||
( GeoAction(..)
|
||||
, GeoConfig(..)
|
||||
, GeoConfigYaml(..)
|
||||
, GeoInfo(..)
|
||||
, GeoDecision(..)
|
||||
, AsnWindow(..)
|
||||
, Geo(..)
|
||||
, defaultGeoLanguage
|
||||
, defaultGeoConcentrationWindowSeconds
|
||||
, defaultGeoConcentrationThreshold
|
||||
, defaultGeoJailCooldownSeconds
|
||||
, defaultGeoSweepIntervalMicros
|
||||
, defaultGeoFlaggedAsns
|
||||
, geoResponseHeaderName
|
||||
, parseGeoAction
|
||||
, sockAddrToIP
|
||||
, countryBlocked
|
||||
, emptyGeoInfo
|
||||
, lookupGeo
|
||||
, bumpAsnCounter
|
||||
, asnConcentrationScore
|
||||
, purgeAsnCounters
|
||||
, startAsnSweeper
|
||||
, decideGeo
|
||||
, buildGeoConfig
|
||||
, openGeo
|
||||
, geoMiddleware
|
||||
, renderGeoHeader
|
||||
) where
|
||||
|
||||
import Aenebris.DDoS.IPJail (IPJail, jail)
|
||||
import Aenebris.RateLimit (clientIPKey)
|
||||
|
||||
import Control.Concurrent (threadDelay)
|
||||
import Control.Concurrent.Async (Async, async)
|
||||
import Control.Concurrent.STM
|
||||
( STM
|
||||
, TVar
|
||||
, atomically
|
||||
, modifyTVar'
|
||||
, newTVarIO
|
||||
, readTVar
|
||||
, writeTVar
|
||||
)
|
||||
import Control.Monad (forever, when)
|
||||
import Data.Aeson (FromJSON(..), withObject, (.:?), (.!=))
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString.Char8 as BC
|
||||
import qualified Data.ByteString.Lazy as LBS
|
||||
import Data.GeoIP2 (GeoDB, GeoResult(..), AS(..), findGeoData, openGeoDB)
|
||||
import Data.IP (IP(..), fromHostAddress, fromHostAddress6)
|
||||
import Data.Map.Strict (Map)
|
||||
import qualified Data.Map.Strict as Map
|
||||
import Data.Maybe (fromMaybe, isJust)
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import qualified Data.Text.Encoding as TE
|
||||
import Data.Time.Clock.POSIX (POSIXTime, getPOSIXTime)
|
||||
import GHC.Generics (Generic)
|
||||
import Network.HTTP.Types (HeaderName, status403)
|
||||
import Network.Socket (SockAddr(..))
|
||||
import Network.Wai
|
||||
( Middleware
|
||||
, mapResponseHeaders
|
||||
, remoteHost
|
||||
, responseLBS
|
||||
)
|
||||
|
||||
defaultGeoLanguage :: Text
|
||||
defaultGeoLanguage = "en"
|
||||
|
||||
defaultGeoConcentrationWindowSeconds :: Int
|
||||
defaultGeoConcentrationWindowSeconds = 60
|
||||
|
||||
defaultGeoConcentrationThreshold :: Int
|
||||
defaultGeoConcentrationThreshold = 500
|
||||
|
||||
defaultGeoJailCooldownSeconds :: Int
|
||||
defaultGeoJailCooldownSeconds = 600
|
||||
|
||||
defaultGeoSweepIntervalMicros :: Int
|
||||
defaultGeoSweepIntervalMicros = 60_000_000
|
||||
|
||||
defaultGeoFlaggedAsns :: [Int]
|
||||
defaultGeoFlaggedAsns = []
|
||||
|
||||
geoResponseHeaderName :: HeaderName
|
||||
geoResponseHeaderName = "x-aenebris-geo"
|
||||
|
||||
data GeoAction
|
||||
= GeoActionLog
|
||||
| GeoActionJail
|
||||
deriving (Show, Eq, Generic)
|
||||
|
||||
parseGeoAction :: Text -> GeoAction
|
||||
parseGeoAction t = case T.toLower t of
|
||||
"jail" -> GeoActionJail
|
||||
"log" -> GeoActionLog
|
||||
_ -> GeoActionLog
|
||||
|
||||
data GeoConfig = GeoConfig
|
||||
{ gcCountryDb :: !(Maybe FilePath)
|
||||
, gcAsnDb :: !(Maybe FilePath)
|
||||
, gcBlockedCountries :: ![Text]
|
||||
, gcAllowedCountries :: ![Text]
|
||||
, gcFlaggedAsns :: ![Int]
|
||||
, gcConcentrationWindowSeconds :: !Int
|
||||
, gcConcentrationThreshold :: !Int
|
||||
, gcJailCooldownSeconds :: !Int
|
||||
, gcAction :: !GeoAction
|
||||
, gcAnnotateHeader :: !Bool
|
||||
, gcLanguage :: !Text
|
||||
} deriving (Show, Eq)
|
||||
|
||||
data GeoConfigYaml = GeoConfigYaml
|
||||
{ gcyEnabled :: !Bool
|
||||
, gcyCountryDb :: !(Maybe FilePath)
|
||||
, gcyAsnDb :: !(Maybe FilePath)
|
||||
, gcyBlockedCountries :: ![Text]
|
||||
, gcyAllowedCountries :: ![Text]
|
||||
, gcyFlaggedAsns :: ![Int]
|
||||
, gcyWindowSeconds :: !(Maybe Int)
|
||||
, gcyThreshold :: !(Maybe Int)
|
||||
, gcyJailCooldownSeconds :: !(Maybe Int)
|
||||
, gcyAction :: !(Maybe Text)
|
||||
, gcyAnnotateHeader :: !Bool
|
||||
, gcyLanguage :: !(Maybe Text)
|
||||
} deriving (Show, Eq, Generic)
|
||||
|
||||
instance FromJSON GeoConfigYaml where
|
||||
parseJSON = withObject "GeoConfigYaml" $ \v -> GeoConfigYaml
|
||||
<$> v .:? "enabled" .!= False
|
||||
<*> v .:? "country_db"
|
||||
<*> v .:? "asn_db"
|
||||
<*> v .:? "blocked_countries" .!= []
|
||||
<*> v .:? "allowed_countries" .!= []
|
||||
<*> v .:? "flagged_asns" .!= []
|
||||
<*> v .:? "window_seconds"
|
||||
<*> v .:? "threshold"
|
||||
<*> v .:? "jail_cooldown_seconds"
|
||||
<*> v .:? "action"
|
||||
<*> v .:? "annotate_header" .!= True
|
||||
<*> v .:? "language"
|
||||
|
||||
buildGeoConfig :: Maybe GeoConfigYaml -> Maybe GeoConfig
|
||||
buildGeoConfig Nothing = Nothing
|
||||
buildGeoConfig (Just y)
|
||||
| not (gcyEnabled y) = Nothing
|
||||
| otherwise = Just GeoConfig
|
||||
{ gcCountryDb = gcyCountryDb y
|
||||
, gcAsnDb = gcyAsnDb y
|
||||
, gcBlockedCountries = map T.toUpper (gcyBlockedCountries y)
|
||||
, gcAllowedCountries = map T.toUpper (gcyAllowedCountries y)
|
||||
, gcFlaggedAsns = gcyFlaggedAsns y
|
||||
, gcConcentrationWindowSeconds =
|
||||
fromMaybe defaultGeoConcentrationWindowSeconds (gcyWindowSeconds y)
|
||||
, gcConcentrationThreshold =
|
||||
fromMaybe defaultGeoConcentrationThreshold (gcyThreshold y)
|
||||
, gcJailCooldownSeconds =
|
||||
fromMaybe defaultGeoJailCooldownSeconds (gcyJailCooldownSeconds y)
|
||||
, gcAction = maybe GeoActionLog parseGeoAction (gcyAction y)
|
||||
, gcAnnotateHeader = gcyAnnotateHeader y
|
||||
, gcLanguage = fromMaybe defaultGeoLanguage (gcyLanguage y)
|
||||
}
|
||||
|
||||
data GeoInfo = GeoInfo
|
||||
{ giCountryISO :: !(Maybe Text)
|
||||
, giAsnNumber :: !(Maybe Int)
|
||||
, giAsnOrg :: !(Maybe Text)
|
||||
, giFlaggedAsn :: !Bool
|
||||
} deriving (Show, Eq)
|
||||
|
||||
emptyGeoInfo :: GeoInfo
|
||||
emptyGeoInfo = GeoInfo Nothing Nothing Nothing False
|
||||
|
||||
data GeoDecision
|
||||
= GeoAllow
|
||||
| GeoBlockCountry !Text
|
||||
| GeoJailAsn !Int !Double
|
||||
deriving (Show, Eq)
|
||||
|
||||
data AsnWindow = AsnWindow
|
||||
{ awCount :: !Int
|
||||
, awWindowStart :: !POSIXTime
|
||||
} deriving (Show, Eq)
|
||||
|
||||
data Geo = Geo
|
||||
{ geoCountryDb :: !(Maybe GeoDB)
|
||||
, geoAsnDb :: !(Maybe GeoDB)
|
||||
, geoConfig :: !GeoConfig
|
||||
, geoAsnCounts :: !(TVar (Map Int AsnWindow))
|
||||
}
|
||||
|
||||
openGeo :: GeoConfig -> IO Geo
|
||||
openGeo cfg = do
|
||||
countryDb <- traverse openGeoDB (gcCountryDb cfg)
|
||||
asnDb <- traverse openGeoDB (gcAsnDb cfg)
|
||||
counts <- newTVarIO Map.empty
|
||||
pure Geo
|
||||
{ geoCountryDb = countryDb
|
||||
, geoAsnDb = asnDb
|
||||
, geoConfig = cfg
|
||||
, geoAsnCounts = counts
|
||||
}
|
||||
|
||||
sockAddrToIP :: SockAddr -> Maybe IP
|
||||
sockAddrToIP (SockAddrInet _ ha) = Just (IPv4 (fromHostAddress ha))
|
||||
sockAddrToIP (SockAddrInet6 _ _ ha6 _) = Just (IPv6 (fromHostAddress6 ha6))
|
||||
sockAddrToIP (SockAddrUnix _) = Nothing
|
||||
|
||||
countryBlocked :: GeoConfig -> Maybe Text -> Maybe Text
|
||||
countryBlocked GeoConfig{..} mIso = case mIso of
|
||||
Nothing
|
||||
| not (null gcAllowedCountries) -> Just "??"
|
||||
| otherwise -> Nothing
|
||||
Just iso ->
|
||||
let up = T.toUpper iso
|
||||
inBlocked = up `elem` gcBlockedCountries
|
||||
inAllowed = null gcAllowedCountries || up `elem` gcAllowedCountries
|
||||
in if inBlocked || not inAllowed then Just up else Nothing
|
||||
|
||||
lookupGeo :: Geo -> IP -> IO GeoInfo
|
||||
lookupGeo Geo{..} ip = do
|
||||
let lang = gcLanguage geoConfig
|
||||
countryRes = (\db -> findGeoData db lang ip) <$> geoCountryDb
|
||||
asnRes = (\db -> findGeoData db lang ip) <$> geoAsnDb
|
||||
country = case countryRes of
|
||||
Just (Right r) -> geoCountryISO r
|
||||
_ -> Nothing
|
||||
(asn, org) = case asnRes of
|
||||
Just (Right r) -> case geoAS r of
|
||||
Just a -> (Just (asNumber a), Just (asOrganization a))
|
||||
Nothing -> (Nothing, Nothing)
|
||||
_ -> (Nothing, Nothing)
|
||||
flagged = case asn of
|
||||
Just n -> n `elem` gcFlaggedAsns geoConfig
|
||||
Nothing -> False
|
||||
pure GeoInfo
|
||||
{ giCountryISO = country
|
||||
, giAsnNumber = asn
|
||||
, giAsnOrg = org
|
||||
, giFlaggedAsn = flagged
|
||||
}
|
||||
|
||||
bumpAsnCounter :: Geo -> Int -> POSIXTime -> STM Int
|
||||
bumpAsnCounter Geo{..} n now = do
|
||||
let window = fromIntegral (gcConcentrationWindowSeconds geoConfig) :: POSIXTime
|
||||
m <- readTVar geoAsnCounts
|
||||
let entry = case Map.lookup n m of
|
||||
Just w
|
||||
| now - awWindowStart w < window ->
|
||||
w { awCount = awCount w + 1 }
|
||||
_ -> AsnWindow { awCount = 1, awWindowStart = now }
|
||||
writeTVar geoAsnCounts $! Map.insert n entry m
|
||||
pure (awCount entry)
|
||||
|
||||
asnConcentrationScore :: Geo -> Int -> Double
|
||||
asnConcentrationScore Geo{..} count =
|
||||
let threshold = max 1 (gcConcentrationThreshold geoConfig)
|
||||
ratio = fromIntegral count / fromIntegral threshold :: Double
|
||||
in min 1.0 (max 0.0 ratio)
|
||||
|
||||
purgeAsnCounters :: Geo -> POSIXTime -> STM Int
|
||||
purgeAsnCounters Geo{..} now = do
|
||||
let window = fromIntegral (gcConcentrationWindowSeconds geoConfig) :: POSIXTime
|
||||
m <- readTVar geoAsnCounts
|
||||
let (stale, fresh) = Map.partition (\w -> now - awWindowStart w >= window) m
|
||||
modifyTVar' geoAsnCounts (const fresh)
|
||||
pure (Map.size stale)
|
||||
|
||||
startAsnSweeper :: Geo -> IO (Async ())
|
||||
startAsnSweeper g = async $ forever $ do
|
||||
threadDelay defaultGeoSweepIntervalMicros
|
||||
now <- getPOSIXTime
|
||||
_ <- atomically (purgeAsnCounters g now)
|
||||
pure ()
|
||||
|
||||
decideGeo :: GeoConfig -> GeoInfo -> Int -> GeoDecision
|
||||
decideGeo cfg info count =
|
||||
case countryBlocked cfg (giCountryISO info) of
|
||||
Just iso -> GeoBlockCountry iso
|
||||
Nothing ->
|
||||
let threshold = gcConcentrationThreshold cfg
|
||||
safeThreshold = max 1 threshold
|
||||
score = min 1.0
|
||||
$ fromIntegral count / fromIntegral safeThreshold :: Double
|
||||
shouldJail = gcAction cfg == GeoActionJail
|
||||
&& giFlaggedAsn info
|
||||
&& count >= threshold
|
||||
in case giAsnNumber info of
|
||||
Just n | shouldJail -> GeoJailAsn n score
|
||||
_ -> GeoAllow
|
||||
|
||||
renderGeoHeader :: GeoInfo -> Int -> ByteString
|
||||
renderGeoHeader info count =
|
||||
let countryPart = case giCountryISO info of
|
||||
Just iso -> "country=" <> TE.encodeUtf8 iso
|
||||
Nothing -> "country=??"
|
||||
asnPart = case giAsnNumber info of
|
||||
Just n -> "asn=" <> BC.pack (show n)
|
||||
Nothing -> "asn=0"
|
||||
flagPart = if giFlaggedAsn info then "flag=1" else "flag=0"
|
||||
countPart = "count=" <> BC.pack (show count)
|
||||
in BC.intercalate " " [countryPart, asnPart, flagPart, countPart]
|
||||
|
||||
forbiddenByCountry :: Text -> LBS.ByteString
|
||||
forbiddenByCountry iso =
|
||||
LBS.fromStrict ("403 Forbidden: country " <> TE.encodeUtf8 iso <> " is restricted")
|
||||
|
||||
geoMiddleware :: Geo -> Maybe IPJail -> Middleware
|
||||
geoMiddleware g@Geo{..} mJail app req respond = do
|
||||
let mIp = sockAddrToIP (remoteHost req)
|
||||
(info, count) <- case mIp of
|
||||
Nothing -> pure (emptyGeoInfo, 0)
|
||||
Just ip -> do
|
||||
infoRaw <- lookupGeo g ip
|
||||
now <- getPOSIXTime
|
||||
c <- case giAsnNumber infoRaw of
|
||||
Just n -> atomically (bumpAsnCounter g n now)
|
||||
Nothing -> pure 0
|
||||
pure (infoRaw, c)
|
||||
let decision = decideGeo geoConfig info count
|
||||
annotate = gcAnnotateHeader geoConfig
|
||||
headerBS = renderGeoHeader info count
|
||||
withGeoHeader =
|
||||
if annotate
|
||||
then mapResponseHeaders ((geoResponseHeaderName, headerBS) :)
|
||||
else id
|
||||
case decision of
|
||||
GeoAllow -> app req (respond . withGeoHeader)
|
||||
GeoBlockCountry iso -> respond $ withGeoHeader $ responseLBS status403
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("Cache-Control", "no-store")
|
||||
]
|
||||
(forbiddenByCountry iso)
|
||||
GeoJailAsn asn _score -> do
|
||||
now <- getPOSIXTime
|
||||
when (isJust mJail) $
|
||||
atomically $ case mJail of
|
||||
Just j -> jail j (clientIPKey req)
|
||||
(fromIntegral (gcJailCooldownSeconds geoConfig))
|
||||
("geo:asn:" <> BC.pack (show asn))
|
||||
now
|
||||
Nothing -> pure ()
|
||||
respond $ withGeoHeader $ responseLBS status403
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("Cache-Control", "no-store")
|
||||
]
|
||||
"403 Forbidden: source network is temporarily restricted"
|
||||
|
|
@ -0,0 +1,333 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
Honeypot.hs
|
||||
-}
|
||||
{-# LANGUAGE BangPatterns #-}
|
||||
{-# LANGUAGE DeriveGeneric #-}
|
||||
{-# LANGUAGE NumericUnderscores #-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE RecordWildCards #-}
|
||||
|
||||
module Aenebris.Honeypot
|
||||
( TrapPattern(..)
|
||||
, HoneypotAction(..)
|
||||
, HoneypotConfig(..)
|
||||
, HoneypotConfigYaml(..)
|
||||
, defaultTrapPatterns
|
||||
, defaultHoneypotConfig
|
||||
, defaultHoneypotCooldown
|
||||
, defaultLabyrinthPrefix
|
||||
, defaultLabyrinthFanout
|
||||
, honeypotResponseHeader
|
||||
, robotsResponseHeader
|
||||
, matchTrap
|
||||
, isAllowed
|
||||
, honeypotMiddleware
|
||||
, labyrinthBody
|
||||
, robotsTxtBody
|
||||
, parseHoneypotAction
|
||||
, buildHoneypotConfig
|
||||
) where
|
||||
|
||||
import Aenebris.DDoS.IPJail (IPJail, jail)
|
||||
import Aenebris.RateLimit (clientIPKey)
|
||||
import Control.Concurrent (threadDelay)
|
||||
import Control.Concurrent.STM (atomically)
|
||||
import Data.Aeson (FromJSON(..), withObject, (.!=), (.:?))
|
||||
import Data.Bits (shiftR, xor)
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString as BS
|
||||
import qualified Data.ByteString.Char8 as BC
|
||||
import qualified Data.ByteString.Lazy as LBS
|
||||
import Data.CaseInsensitive (CI)
|
||||
import Data.Maybe (fromMaybe)
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import qualified Data.Text.Encoding as TE
|
||||
import Data.Time.Clock.POSIX (POSIXTime, getPOSIXTime)
|
||||
import Data.Word (Word64)
|
||||
import GHC.Generics (Generic)
|
||||
import Network.HTTP.Types (Status, mkStatus)
|
||||
import Network.Wai
|
||||
( Middleware
|
||||
, Response
|
||||
, rawPathInfo
|
||||
, requestMethod
|
||||
, responseLBS
|
||||
)
|
||||
import Numeric (showHex)
|
||||
|
||||
data TrapPattern
|
||||
= TrapExact !ByteString
|
||||
| TrapPrefix !ByteString
|
||||
deriving (Eq, Show)
|
||||
|
||||
data HoneypotAction
|
||||
= HoneypotJail
|
||||
| HoneypotLog
|
||||
| HoneypotLabyrinth
|
||||
deriving (Eq, Show)
|
||||
|
||||
data HoneypotConfig = HoneypotConfig
|
||||
{ hpPatterns :: ![TrapPattern]
|
||||
, hpAction :: !HoneypotAction
|
||||
, hpJailCooldown :: !POSIXTime
|
||||
, hpResponseDelayMicros :: !(Maybe Int)
|
||||
, hpAllowedIPs :: ![ByteString]
|
||||
, hpServeRobotsTxt :: !Bool
|
||||
, hpLabyrinthFanout :: !Int
|
||||
} deriving (Eq, Show)
|
||||
|
||||
defaultHoneypotCooldown :: POSIXTime
|
||||
defaultHoneypotCooldown = 3600
|
||||
|
||||
defaultLabyrinthPrefix :: ByteString
|
||||
defaultLabyrinthPrefix = "/_labyrinth/"
|
||||
|
||||
defaultLabyrinthFanout :: Int
|
||||
defaultLabyrinthFanout = 24
|
||||
|
||||
defaultTrapPatterns :: [TrapPattern]
|
||||
defaultTrapPatterns =
|
||||
[ TrapExact "/.env"
|
||||
, TrapExact "/.env.local"
|
||||
, TrapExact "/.env.production"
|
||||
, TrapExact "/.env.backup"
|
||||
, TrapExact "/wp-login.php"
|
||||
, TrapExact "/wp-admin"
|
||||
, TrapExact "/xmlrpc.php"
|
||||
, TrapExact "/phpmyadmin"
|
||||
, TrapExact "/pma"
|
||||
, TrapExact "/myadmin"
|
||||
, TrapExact "/administrator"
|
||||
, TrapExact "/admin/config.php"
|
||||
, TrapExact "/server-status"
|
||||
, TrapExact "/server-info"
|
||||
, TrapExact "/.DS_Store"
|
||||
, TrapExact "/.htaccess"
|
||||
, TrapExact "/.htpasswd"
|
||||
, TrapExact "/backup.sql"
|
||||
, TrapExact "/db.sql"
|
||||
, TrapExact "/dump.sql"
|
||||
, TrapExact "/database.sql"
|
||||
, TrapExact "/config.php.bak"
|
||||
, TrapExact "/web.config"
|
||||
, TrapExact "/sftp-config.json"
|
||||
, TrapPrefix "/.git/"
|
||||
, TrapPrefix "/.svn/"
|
||||
, TrapPrefix "/.hg/"
|
||||
, TrapPrefix "/.aws/"
|
||||
, TrapPrefix "/.ssh/"
|
||||
, TrapPrefix "/.vscode/"
|
||||
, TrapPrefix "/.idea/"
|
||||
, TrapPrefix "/wp-content/plugins/"
|
||||
, TrapPrefix "/wp-includes/"
|
||||
, TrapPrefix "/vendor/phpunit/"
|
||||
, TrapPrefix "/cgi-bin/"
|
||||
, TrapPrefix "/actuator/"
|
||||
, TrapPrefix "/_ignition/"
|
||||
, TrapPrefix "/druid/indexer/"
|
||||
, TrapPrefix "/jenkins/script"
|
||||
, TrapPrefix "/solr/admin/"
|
||||
, TrapPrefix "/manager/html"
|
||||
, TrapPrefix "/console/login"
|
||||
, TrapPrefix defaultLabyrinthPrefix
|
||||
]
|
||||
|
||||
defaultHoneypotConfig :: HoneypotConfig
|
||||
defaultHoneypotConfig = HoneypotConfig
|
||||
{ hpPatterns = defaultTrapPatterns
|
||||
, hpAction = HoneypotJail
|
||||
, hpJailCooldown = defaultHoneypotCooldown
|
||||
, hpResponseDelayMicros = Nothing
|
||||
, hpAllowedIPs = []
|
||||
, hpServeRobotsTxt = True
|
||||
, hpLabyrinthFanout = defaultLabyrinthFanout
|
||||
}
|
||||
|
||||
data HoneypotConfigYaml = HoneypotConfigYaml
|
||||
{ hpyEnabled :: !Bool
|
||||
, hpyAction :: !Text
|
||||
, hpyCooldownSeconds :: !(Maybe Int)
|
||||
, hpyResponseDelayMillis :: !(Maybe Int)
|
||||
, hpyExtraExact :: ![Text]
|
||||
, hpyExtraPrefix :: ![Text]
|
||||
, hpyUseDefaults :: !Bool
|
||||
, hpyAllowedIPs :: ![Text]
|
||||
, hpyServeRobotsTxt :: !Bool
|
||||
, hpyLabyrinthFanout :: !(Maybe Int)
|
||||
} deriving (Eq, Show, Generic)
|
||||
|
||||
instance FromJSON HoneypotConfigYaml where
|
||||
parseJSON = withObject "HoneypotConfig" $ \v -> HoneypotConfigYaml
|
||||
<$> v .:? "enabled" .!= True
|
||||
<*> v .:? "action" .!= "jail"
|
||||
<*> v .:? "cooldown_seconds"
|
||||
<*> v .:? "response_delay_millis"
|
||||
<*> v .:? "extra_exact_paths" .!= []
|
||||
<*> v .:? "extra_prefix_paths" .!= []
|
||||
<*> v .:? "use_default_traps" .!= True
|
||||
<*> v .:? "allowed_ips" .!= []
|
||||
<*> v .:? "serve_robots_txt" .!= True
|
||||
<*> v .:? "labyrinth_fanout"
|
||||
|
||||
parseHoneypotAction :: Text -> HoneypotAction
|
||||
parseHoneypotAction t = case T.toLower t of
|
||||
"jail" -> HoneypotJail
|
||||
"labyrinth" -> HoneypotLabyrinth
|
||||
"log" -> HoneypotLog
|
||||
_ -> HoneypotLog
|
||||
|
||||
buildHoneypotConfig :: Maybe HoneypotConfigYaml -> Maybe HoneypotConfig
|
||||
buildHoneypotConfig Nothing = Nothing
|
||||
buildHoneypotConfig (Just y)
|
||||
| not (hpyEnabled y) = Nothing
|
||||
| otherwise = Just HoneypotConfig
|
||||
{ hpPatterns = basePatterns <> extraExact <> extraPrefix
|
||||
, hpAction = parseHoneypotAction (hpyAction y)
|
||||
, hpJailCooldown = maybe defaultHoneypotCooldown fromIntegral (hpyCooldownSeconds y)
|
||||
, hpResponseDelayMicros = fmap (\ms -> ms * 1000) (hpyResponseDelayMillis y)
|
||||
, hpAllowedIPs = map TE.encodeUtf8 (hpyAllowedIPs y)
|
||||
, hpServeRobotsTxt = hpyServeRobotsTxt y
|
||||
, hpLabyrinthFanout = fromMaybe defaultLabyrinthFanout (hpyLabyrinthFanout y)
|
||||
}
|
||||
where
|
||||
basePatterns = if hpyUseDefaults y then defaultTrapPatterns else []
|
||||
extraExact = [TrapExact (TE.encodeUtf8 t) | t <- hpyExtraExact y]
|
||||
extraPrefix = [TrapPrefix (TE.encodeUtf8 t) | t <- hpyExtraPrefix y]
|
||||
|
||||
honeypotResponseHeader :: CI ByteString
|
||||
honeypotResponseHeader = "x-aenebris-honeypot"
|
||||
|
||||
robotsResponseHeader :: CI ByteString
|
||||
robotsResponseHeader = "x-aenebris-robots"
|
||||
|
||||
matchTrap :: ByteString -> [TrapPattern] -> Maybe TrapPattern
|
||||
matchTrap _ [] = Nothing
|
||||
matchTrap path (p : rest) = case p of
|
||||
TrapExact e | e == path -> Just p
|
||||
TrapPrefix pr | BS.isPrefixOf pr path -> Just p
|
||||
_ -> matchTrap path rest
|
||||
|
||||
isAllowed :: ByteString -> [ByteString] -> Bool
|
||||
isAllowed = elem
|
||||
|
||||
isLabyrinthPath :: ByteString -> Bool
|
||||
isLabyrinthPath = BS.isPrefixOf defaultLabyrinthPrefix
|
||||
|
||||
trapLabel :: TrapPattern -> ByteString
|
||||
trapLabel (TrapExact e) = e
|
||||
trapLabel (TrapPrefix p) = p <> "*"
|
||||
|
||||
status404 :: Status
|
||||
status404 = mkStatus 404 "Not Found"
|
||||
|
||||
status200 :: Status
|
||||
status200 = mkStatus 200 "OK"
|
||||
|
||||
honeypotMiddleware :: HoneypotConfig -> Maybe IPJail -> Middleware
|
||||
honeypotMiddleware cfg@HoneypotConfig{..} mJail app req respond
|
||||
| hpServeRobotsTxt && requestMethod req == "GET"
|
||||
&& rawPathInfo req == "/robots.txt" =
|
||||
respond (robotsResponse cfg)
|
||||
| otherwise = case matchTrap (rawPathInfo req) hpPatterns of
|
||||
Nothing -> app req respond
|
||||
Just trap -> do
|
||||
now <- getPOSIXTime
|
||||
let ip = clientIPKey req
|
||||
label = trapLabel trap
|
||||
allowed = isAllowed ip hpAllowedIPs
|
||||
case (hpAction, allowed, mJail) of
|
||||
(HoneypotJail, False, Just j) ->
|
||||
atomically (jail j ip hpJailCooldown ("honeypot:" <> label) now)
|
||||
(HoneypotLabyrinth, False, Just j) ->
|
||||
atomically (jail j ip hpJailCooldown ("honeypot:" <> label) now)
|
||||
_ -> pure ()
|
||||
maybe (pure ()) threadDelay hpResponseDelayMicros
|
||||
respond (trapResponse cfg (rawPathInfo req) label allowed)
|
||||
|
||||
trapResponse :: HoneypotConfig -> ByteString -> ByteString -> Bool -> Response
|
||||
trapResponse HoneypotConfig{..} path label allowed
|
||||
| hpAction == HoneypotLabyrinth || isLabyrinthPath path =
|
||||
responseLBS status200
|
||||
[ ("Content-Type", "text/html; charset=utf-8")
|
||||
, ("Cache-Control", "no-store, no-cache, must-revalidate")
|
||||
, ("Pragma", "no-cache")
|
||||
, ("X-Robots-Tag", "noindex, nofollow")
|
||||
, (honeypotResponseHeader, marker "labyrinth")
|
||||
]
|
||||
(labyrinthBody path hpLabyrinthFanout)
|
||||
| otherwise =
|
||||
responseLBS status404
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("Cache-Control", "no-store")
|
||||
, (honeypotResponseHeader, marker "trap")
|
||||
]
|
||||
"404 Not Found"
|
||||
where
|
||||
marker kind =
|
||||
kind <> "=" <> label <> (if allowed then " allow=1" else "")
|
||||
|
||||
robotsResponse :: HoneypotConfig -> Response
|
||||
robotsResponse cfg =
|
||||
responseLBS status200
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("Cache-Control", "public, max-age=3600")
|
||||
, (robotsResponseHeader, "generated")
|
||||
]
|
||||
(LBS.fromStrict (robotsTxtBody cfg))
|
||||
|
||||
robotsTxtBody :: HoneypotConfig -> ByteString
|
||||
robotsTxtBody HoneypotConfig{..} = BS.concat $
|
||||
[ "User-agent: *\n"
|
||||
, "# Honeypot trap paths — Disallow per RFC 9309. Visiting these\n"
|
||||
, "# paths is treated as a violation signal regardless of declared UA.\n"
|
||||
] <> map disallowLine hpPatterns
|
||||
where
|
||||
disallowLine (TrapExact e) = "Disallow: " <> e <> "\n"
|
||||
disallowLine (TrapPrefix p) = "Disallow: " <> p <> "\n"
|
||||
|
||||
labyrinthBody :: ByteString -> Int -> LBS.ByteString
|
||||
labyrinthBody requestPath fanout = LBS.fromStrict $ BS.concat
|
||||
[ "<!doctype html><html><head>"
|
||||
, "<title>", titleFor requestPath, "</title>"
|
||||
, "<meta name=\"robots\" content=\"noindex, nofollow\">"
|
||||
, "</head><body>"
|
||||
, "<h1>", titleFor requestPath, "</h1>"
|
||||
, "<p>Resource index. Pages may have moved; see the related entries below.</p>"
|
||||
, "<ul>", linkList, "</ul>"
|
||||
, "<p>Archive snapshots and historical mirrors are linked from the resource graph.</p>"
|
||||
, "</body></html>"
|
||||
]
|
||||
where
|
||||
seed = fnv1a requestPath
|
||||
titleFor p = "Index " <> hexBytes (fnv1a p)
|
||||
linkList = BS.concat
|
||||
[ "<li><a href=\"" <> defaultLabyrinthPrefix
|
||||
<> hexBytes (mix seed i) <> "/"
|
||||
<> BC.pack (show i)
|
||||
<> "\">node-" <> BC.pack (show i) <> "</a></li>"
|
||||
| i <- [1 .. max 1 fanout]
|
||||
]
|
||||
|
||||
mix :: Word64 -> Int -> Word64
|
||||
mix s i = fnv1aStep s (fromIntegral (i `mod` 256))
|
||||
|
||||
fnv1a :: ByteString -> Word64
|
||||
fnv1a = BS.foldl' (\h w -> fnv1aStep h (fromIntegral w)) fnvOffset
|
||||
where
|
||||
fnvOffset :: Word64
|
||||
fnvOffset = 14695981039346656037
|
||||
|
||||
fnv1aStep :: Word64 -> Word64 -> Word64
|
||||
fnv1aStep h b = (h `xor` b) * fnvPrime
|
||||
where
|
||||
fnvPrime :: Word64
|
||||
fnvPrime = 1099511628211
|
||||
|
||||
hexBytes :: Word64 -> ByteString
|
||||
hexBytes w = BC.pack (pad (showHex (w `shiftR` 32) "") 8)
|
||||
where
|
||||
pad s n
|
||||
| length s >= n = take n s
|
||||
| otherwise = replicate (n - length s) '0' <> s
|
||||
|
|
@ -0,0 +1,559 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
Features.hs
|
||||
-}
|
||||
{-# LANGUAGE BangPatterns #-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE RecordWildCards #-}
|
||||
|
||||
module Aenebris.ML.Features
|
||||
( FeatureVector(..)
|
||||
, FeatureContext(..)
|
||||
, emptyFeatureContext
|
||||
, extractFeatures
|
||||
, featureVectorLength
|
||||
, featureVectorToList
|
||||
, featureVectorToVector
|
||||
, featureNames
|
||||
, headerCountCap
|
||||
, pathDepthCap
|
||||
, queryParamCountCap
|
||||
, userAgentLengthCap
|
||||
, pathEntropyMax
|
||||
, acceptValueLengthCap
|
||||
, botKeywordPatterns
|
||||
, headlessPatterns
|
||||
, suspiciousPathExtensions
|
||||
, commonBrowserMarkers
|
||||
, idempotentMethods
|
||||
, commonBrowserMarkerThreshold
|
||||
, secFetchModeValidValues
|
||||
, secFetchDestNavigationValues
|
||||
, headerOrderTrackedNames
|
||||
, chromeHeaderCanonicalOrder
|
||||
, firefoxHeaderCanonicalOrder
|
||||
, shannonEntropyBytes
|
||||
, uaSecChConsistency
|
||||
, uaContainsBotKeyword
|
||||
, uaContainsHeadlessMarker
|
||||
, uaIsCommonBrowser
|
||||
, uaPlatformConsistency
|
||||
, pathDepth
|
||||
, pathHasSuspiciousExtension
|
||||
, methodIsIdempotent
|
||||
, secFetchModeIsValid
|
||||
, secFetchTripleIsCoherent
|
||||
, acceptIsWildcard
|
||||
, headerOrderIsCanonicalBrowser
|
||||
, clamp01
|
||||
, normalizedRatio
|
||||
) where
|
||||
|
||||
import Aenebris.Geo (GeoInfo(..), emptyGeoInfo)
|
||||
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString as BS
|
||||
import qualified Data.ByteString.Char8 as BC
|
||||
import Data.CaseInsensitive (CI)
|
||||
import Data.Char (toLower)
|
||||
import Data.Map.Strict (Map)
|
||||
import qualified Data.Map.Strict as Map
|
||||
import Data.Maybe (fromMaybe, isJust)
|
||||
import Data.Text (Text)
|
||||
import Data.Vector.Unboxed (Vector)
|
||||
import qualified Data.Vector.Unboxed as VU
|
||||
import Data.Word (Word8)
|
||||
import Network.Wai
|
||||
( Request
|
||||
, queryString
|
||||
, rawPathInfo
|
||||
, requestHeaders
|
||||
, requestMethod
|
||||
)
|
||||
|
||||
headerCountCap :: Double
|
||||
headerCountCap = 32.0
|
||||
|
||||
pathDepthCap :: Double
|
||||
pathDepthCap = 16.0
|
||||
|
||||
queryParamCountCap :: Double
|
||||
queryParamCountCap = 32.0
|
||||
|
||||
userAgentLengthCap :: Double
|
||||
userAgentLengthCap = 256.0
|
||||
|
||||
pathEntropyMax :: Double
|
||||
pathEntropyMax = 8.0
|
||||
|
||||
acceptValueLengthCap :: Double
|
||||
acceptValueLengthCap = 200.0
|
||||
|
||||
commonBrowserMarkerThreshold :: Int
|
||||
commonBrowserMarkerThreshold = 2
|
||||
|
||||
secFetchModeValidValues :: [ByteString]
|
||||
secFetchModeValidValues =
|
||||
["cors", "no-cors", "navigate", "same-origin", "websocket"]
|
||||
|
||||
secFetchDestNavigationValues :: [ByteString]
|
||||
secFetchDestNavigationValues =
|
||||
["document", "iframe", "frame", "embed", "object"]
|
||||
|
||||
secFetchSiteValidValues :: [ByteString]
|
||||
secFetchSiteValidValues =
|
||||
["none", "same-origin", "same-site", "cross-site"]
|
||||
|
||||
headerOrderTrackedNames :: [CI ByteString]
|
||||
headerOrderTrackedNames =
|
||||
["host", "user-agent", "accept", "accept-encoding", "accept-language"]
|
||||
|
||||
chromeHeaderCanonicalOrder :: [CI ByteString]
|
||||
chromeHeaderCanonicalOrder =
|
||||
["host", "user-agent", "accept", "accept-encoding", "accept-language"]
|
||||
|
||||
firefoxHeaderCanonicalOrder :: [CI ByteString]
|
||||
firefoxHeaderCanonicalOrder =
|
||||
["host", "user-agent", "accept", "accept-language", "accept-encoding"]
|
||||
|
||||
botKeywordPatterns :: [ByteString]
|
||||
botKeywordPatterns =
|
||||
[ "bot"
|
||||
, "crawl"
|
||||
, "spider"
|
||||
, "scrape"
|
||||
, "fetch"
|
||||
, "monitor"
|
||||
, "checker"
|
||||
, "preview"
|
||||
, "wget"
|
||||
, "curl"
|
||||
, "python-requests"
|
||||
, "go-http-client"
|
||||
, "java/"
|
||||
, "okhttp"
|
||||
, "libwww"
|
||||
, "httpclient"
|
||||
]
|
||||
|
||||
headlessPatterns :: [ByteString]
|
||||
headlessPatterns =
|
||||
[ "headlesschrome"
|
||||
, "phantomjs"
|
||||
, "puppeteer"
|
||||
, "playwright"
|
||||
, "selenium"
|
||||
, "webdriver"
|
||||
, "nightmare"
|
||||
, "splash"
|
||||
]
|
||||
|
||||
suspiciousPathExtensions :: [ByteString]
|
||||
suspiciousPathExtensions =
|
||||
[ ".php"
|
||||
, ".asp"
|
||||
, ".aspx"
|
||||
, ".cgi"
|
||||
, ".env"
|
||||
, ".bak"
|
||||
, ".sql"
|
||||
, ".log"
|
||||
, ".swp"
|
||||
, ".old"
|
||||
, ".htaccess"
|
||||
, ".htpasswd"
|
||||
, ".git"
|
||||
, ".ini"
|
||||
, ".conf"
|
||||
, ".yml"
|
||||
, ".yaml"
|
||||
, ".pem"
|
||||
, ".key"
|
||||
]
|
||||
|
||||
commonBrowserMarkers :: [ByteString]
|
||||
commonBrowserMarkers =
|
||||
[ "mozilla/5.0"
|
||||
, "applewebkit"
|
||||
, "chrome/"
|
||||
, "safari/"
|
||||
, "firefox/"
|
||||
, "edg/"
|
||||
, "gecko/"
|
||||
, "version/"
|
||||
]
|
||||
|
||||
idempotentMethods :: [ByteString]
|
||||
idempotentMethods = ["GET", "HEAD", "OPTIONS", "TRACE"]
|
||||
|
||||
chromiumFamilyMarkers :: [ByteString]
|
||||
chromiumFamilyMarkers = ["chrome/", "edg/", "opr/", "chromium/", "yabrowser/"]
|
||||
|
||||
acceptLanguageHeader :: CI ByteString
|
||||
acceptLanguageHeader = "accept-language"
|
||||
|
||||
userAgentHeader :: CI ByteString
|
||||
userAgentHeader = "user-agent"
|
||||
|
||||
acceptEncodingHeader :: CI ByteString
|
||||
acceptEncodingHeader = "accept-encoding"
|
||||
|
||||
refererHeader :: CI ByteString
|
||||
refererHeader = "referer"
|
||||
|
||||
cookieHeader :: CI ByteString
|
||||
cookieHeader = "cookie"
|
||||
|
||||
secChUaHeader :: CI ByteString
|
||||
secChUaHeader = "sec-ch-ua"
|
||||
|
||||
secChUaPlatformHeader :: CI ByteString
|
||||
secChUaPlatformHeader = "sec-ch-ua-platform"
|
||||
|
||||
secFetchSiteHeader :: CI ByteString
|
||||
secFetchSiteHeader = "sec-fetch-site"
|
||||
|
||||
secFetchModeHeader :: CI ByteString
|
||||
secFetchModeHeader = "sec-fetch-mode"
|
||||
|
||||
secFetchDestHeader :: CI ByteString
|
||||
secFetchDestHeader = "sec-fetch-dest"
|
||||
|
||||
acceptHeader :: CI ByteString
|
||||
acceptHeader = "accept"
|
||||
|
||||
clamp01 :: Double -> Double
|
||||
clamp01 x
|
||||
| x < 0.0 = 0.0
|
||||
| x > 1.0 = 1.0
|
||||
| otherwise = x
|
||||
|
||||
normalizedRatio :: Int -> Double -> Double
|
||||
normalizedRatio n cap
|
||||
| cap <= 0.0 = 0.0
|
||||
| otherwise = clamp01 (fromIntegral n / cap)
|
||||
|
||||
asciiToLower :: ByteString -> ByteString
|
||||
asciiToLower = BC.map toLower
|
||||
|
||||
shannonEntropyBytes :: ByteString -> Double
|
||||
shannonEntropyBytes bs
|
||||
| BS.null bs = 0.0
|
||||
| otherwise =
|
||||
let !total = fromIntegral (BS.length bs) :: Double
|
||||
freq :: Map Word8 Int
|
||||
freq = BS.foldl'
|
||||
(\m b -> Map.insertWith (+) b 1 m)
|
||||
Map.empty
|
||||
bs
|
||||
step !acc !c =
|
||||
let p = fromIntegral c / total
|
||||
in acc - p * logBase 2 p
|
||||
in Map.foldl' step 0.0 freq
|
||||
|
||||
uaSecChConsistency :: Maybe ByteString -> Maybe ByteString -> Double
|
||||
uaSecChConsistency mUa mSecCh = case mSecCh of
|
||||
Nothing -> 1.0
|
||||
Just _ -> case mUa of
|
||||
Nothing -> 0.0
|
||||
Just ua ->
|
||||
let lower = asciiToLower ua
|
||||
isChromium = any (`BS.isInfixOf` lower) chromiumFamilyMarkers
|
||||
in if isChromium then 1.0 else 0.0
|
||||
|
||||
uaContainsBotKeyword :: ByteString -> Bool
|
||||
uaContainsBotKeyword ua =
|
||||
let lower = asciiToLower ua
|
||||
in any (`BS.isInfixOf` lower) botKeywordPatterns
|
||||
|
||||
uaContainsHeadlessMarker :: ByteString -> Bool
|
||||
uaContainsHeadlessMarker ua =
|
||||
let lower = asciiToLower ua
|
||||
in any (`BS.isInfixOf` lower) headlessPatterns
|
||||
|
||||
uaIsCommonBrowser :: ByteString -> Bool
|
||||
uaIsCommonBrowser ua =
|
||||
let lower = asciiToLower ua
|
||||
hits = length (filter (`BS.isInfixOf` lower) commonBrowserMarkers)
|
||||
in hits >= commonBrowserMarkerThreshold
|
||||
|
||||
pathDepth :: ByteString -> Int
|
||||
pathDepth path =
|
||||
let segments = filter (not . BS.null) (BC.split '/' path)
|
||||
in length segments
|
||||
|
||||
pathHasSuspiciousExtension :: ByteString -> Bool
|
||||
pathHasSuspiciousExtension path =
|
||||
let lower = asciiToLower path
|
||||
in any (`BS.isSuffixOf` lower) suspiciousPathExtensions
|
||||
|
||||
methodIsIdempotent :: ByteString -> Bool
|
||||
methodIsIdempotent m = m `elem` idempotentMethods
|
||||
|
||||
secFetchModeIsValid :: ByteString -> Bool
|
||||
secFetchModeIsValid v = asciiToLower v `elem` secFetchModeValidValues
|
||||
|
||||
secFetchTripleIsCoherent
|
||||
:: Maybe ByteString
|
||||
-> Maybe ByteString
|
||||
-> Maybe ByteString
|
||||
-> Bool
|
||||
secFetchTripleIsCoherent mSite mMode mDest =
|
||||
case (asciiToLower <$> mSite, asciiToLower <$> mMode, asciiToLower <$> mDest) of
|
||||
(Just site, Just mode, Just dest) ->
|
||||
site `elem` secFetchSiteValidValues
|
||||
&& mode `elem` secFetchModeValidValues
|
||||
&& tripleConsistent site mode dest
|
||||
_ -> False
|
||||
where
|
||||
tripleConsistent site mode dest
|
||||
| site == "none" && mode /= "navigate" = False
|
||||
| mode == "navigate" && dest `notElem` secFetchDestNavigationValues = False
|
||||
| mode == "cors" && site == "none" = False
|
||||
| mode == "websocket" && dest /= "websocket" && dest /= "empty" = False
|
||||
| otherwise = True
|
||||
|
||||
uaPlatformConsistency :: Maybe ByteString -> Maybe ByteString -> Double
|
||||
uaPlatformConsistency mUa mPlatform = case mPlatform of
|
||||
Nothing -> 1.0
|
||||
Just p ->
|
||||
let lowerP = asciiToLower (stripQuotes p)
|
||||
in case mUa of
|
||||
Nothing -> 0.0
|
||||
Just ua ->
|
||||
let lowerUa = asciiToLower ua
|
||||
markers = platformUaMarkers lowerP
|
||||
in if null markers
|
||||
then 0.0
|
||||
else
|
||||
if any (`BS.isInfixOf` lowerUa) markers
|
||||
then 1.0
|
||||
else 0.0
|
||||
|
||||
stripQuotes :: ByteString -> ByteString
|
||||
stripQuotes bs =
|
||||
let trimmed = BS.dropWhile (== quoteByte) (BS.reverse bs)
|
||||
back = BS.reverse trimmed
|
||||
in BS.dropWhile (== quoteByte) back
|
||||
where
|
||||
quoteByte = 34
|
||||
|
||||
platformUaMarkers :: ByteString -> [ByteString]
|
||||
platformUaMarkers p
|
||||
| p == "windows" = ["windows nt"]
|
||||
| p == "macos" = ["mac os x", "macintosh"]
|
||||
| p == "linux" = ["linux", "x11"]
|
||||
| p == "android" = ["android"]
|
||||
| p == "ios" = ["iphone", "ipad", "ipod"]
|
||||
| p == "chrome os" = ["cros"]
|
||||
| p == "chromeos" = ["cros"]
|
||||
| otherwise = []
|
||||
|
||||
acceptIsWildcard :: Maybe ByteString -> Bool
|
||||
acceptIsWildcard mAccept = case mAccept of
|
||||
Nothing -> False
|
||||
Just v ->
|
||||
let trimmed = BS.dropWhile (== spaceByte)
|
||||
$ BS.reverse
|
||||
$ BS.dropWhile (== spaceByte)
|
||||
$ BS.reverse v
|
||||
in trimmed == "*/*"
|
||||
where
|
||||
spaceByte = 32
|
||||
|
||||
headerOrderIsCanonicalBrowser :: [(CI ByteString, ByteString)] -> Bool
|
||||
headerOrderIsCanonicalBrowser headers =
|
||||
let projected = filter (`elem` headerOrderTrackedNames)
|
||||
$ map fst headers
|
||||
chromeMatch = projected == filter (`elem` projected) chromeHeaderCanonicalOrder
|
||||
&& all (`elem` projected) requiredCore
|
||||
firefoxMatch = projected == filter (`elem` projected) firefoxHeaderCanonicalOrder
|
||||
&& all (`elem` projected) requiredCore
|
||||
in chromeMatch || firefoxMatch
|
||||
where
|
||||
requiredCore = ["host", "user-agent", "accept"]
|
||||
|
||||
data FeatureContext = FeatureContext
|
||||
{ fcGeoInfo :: !GeoInfo
|
||||
, fcAsnConcentration :: !Double
|
||||
} deriving (Eq, Show)
|
||||
|
||||
emptyFeatureContext :: FeatureContext
|
||||
emptyFeatureContext = FeatureContext
|
||||
{ fcGeoInfo = emptyGeoInfo
|
||||
, fcAsnConcentration = 0.0
|
||||
}
|
||||
|
||||
data FeatureVector = FeatureVector
|
||||
{ fMissingAcceptLanguage :: {-# UNPACK #-} !Double
|
||||
, fMissingUserAgent :: {-# UNPACK #-} !Double
|
||||
, fMissingAcceptEncoding :: {-# UNPACK #-} !Double
|
||||
, fMissingReferer :: {-# UNPACK #-} !Double
|
||||
, fHasCookie :: {-# UNPACK #-} !Double
|
||||
, fHasSecChUa :: {-# UNPACK #-} !Double
|
||||
, fHeaderCount :: {-# UNPACK #-} !Double
|
||||
, fUaSecChConsistent :: {-# UNPACK #-} !Double
|
||||
, fUaBotKeyword :: {-# UNPACK #-} !Double
|
||||
, fUaHeadless :: {-# UNPACK #-} !Double
|
||||
, fUaCommonBrowser :: {-# UNPACK #-} !Double
|
||||
, fUaLength :: {-# UNPACK #-} !Double
|
||||
, fPathDepth :: {-# UNPACK #-} !Double
|
||||
, fPathEntropy :: {-# UNPACK #-} !Double
|
||||
, fSuspiciousPathExt :: {-# UNPACK #-} !Double
|
||||
, fQueryParamCount :: {-# UNPACK #-} !Double
|
||||
, fMethodIdempotent :: {-# UNPACK #-} !Double
|
||||
, fFlaggedAsn :: {-# UNPACK #-} !Double
|
||||
, fAsnConcentration :: {-# UNPACK #-} !Double
|
||||
, fCountryUnknown :: {-# UNPACK #-} !Double
|
||||
, fMissingSecFetchSite :: {-# UNPACK #-} !Double
|
||||
, fSecFetchModeValid :: {-# UNPACK #-} !Double
|
||||
, fSecFetchContextCoherent :: {-# UNPACK #-} !Double
|
||||
, fChUaPlatformPresent :: {-# UNPACK #-} !Double
|
||||
, fChUaPlatformConsistent :: {-# UNPACK #-} !Double
|
||||
, fAcceptIsWildcard :: {-# UNPACK #-} !Double
|
||||
, fAcceptValueLength :: {-# UNPACK #-} !Double
|
||||
, fHeaderOrderCanonical :: {-# UNPACK #-} !Double
|
||||
} deriving (Eq, Show)
|
||||
|
||||
featureVectorLength :: Int
|
||||
featureVectorLength = 28
|
||||
|
||||
featureNames :: [Text]
|
||||
featureNames =
|
||||
[ "missing_accept_language"
|
||||
, "missing_user_agent"
|
||||
, "missing_accept_encoding"
|
||||
, "missing_referer"
|
||||
, "has_cookie"
|
||||
, "has_sec_ch_ua"
|
||||
, "header_count"
|
||||
, "ua_sec_ch_consistent"
|
||||
, "ua_bot_keyword"
|
||||
, "ua_headless"
|
||||
, "ua_common_browser"
|
||||
, "ua_length"
|
||||
, "path_depth"
|
||||
, "path_entropy"
|
||||
, "suspicious_path_ext"
|
||||
, "query_param_count"
|
||||
, "method_idempotent"
|
||||
, "flagged_asn"
|
||||
, "asn_concentration"
|
||||
, "country_unknown"
|
||||
, "missing_sec_fetch_site"
|
||||
, "sec_fetch_mode_valid"
|
||||
, "sec_fetch_context_coherent"
|
||||
, "ch_ua_platform_present"
|
||||
, "ch_ua_platform_consistent"
|
||||
, "accept_is_wildcard"
|
||||
, "accept_value_length"
|
||||
, "header_order_canonical"
|
||||
]
|
||||
|
||||
featureVectorToList :: FeatureVector -> [Double]
|
||||
featureVectorToList FeatureVector{..} =
|
||||
[ fMissingAcceptLanguage
|
||||
, fMissingUserAgent
|
||||
, fMissingAcceptEncoding
|
||||
, fMissingReferer
|
||||
, fHasCookie
|
||||
, fHasSecChUa
|
||||
, fHeaderCount
|
||||
, fUaSecChConsistent
|
||||
, fUaBotKeyword
|
||||
, fUaHeadless
|
||||
, fUaCommonBrowser
|
||||
, fUaLength
|
||||
, fPathDepth
|
||||
, fPathEntropy
|
||||
, fSuspiciousPathExt
|
||||
, fQueryParamCount
|
||||
, fMethodIdempotent
|
||||
, fFlaggedAsn
|
||||
, fAsnConcentration
|
||||
, fCountryUnknown
|
||||
, fMissingSecFetchSite
|
||||
, fSecFetchModeValid
|
||||
, fSecFetchContextCoherent
|
||||
, fChUaPlatformPresent
|
||||
, fChUaPlatformConsistent
|
||||
, fAcceptIsWildcard
|
||||
, fAcceptValueLength
|
||||
, fHeaderOrderCanonical
|
||||
]
|
||||
|
||||
featureVectorToVector :: FeatureVector -> Vector Double
|
||||
featureVectorToVector = VU.fromList . featureVectorToList
|
||||
|
||||
boolToDouble :: Bool -> Double
|
||||
boolToDouble True = 1.0
|
||||
boolToDouble False = 0.0
|
||||
|
||||
extractFeatures :: FeatureContext -> Request -> FeatureVector
|
||||
extractFeatures FeatureContext{..} req =
|
||||
let !headers = requestHeaders req
|
||||
!path = rawPathInfo req
|
||||
!method = requestMethod req
|
||||
!mAcceptLang = lookupCI acceptLanguageHeader headers
|
||||
!mUserAgent = lookupCI userAgentHeader headers
|
||||
!mAcceptEnc = lookupCI acceptEncodingHeader headers
|
||||
!mReferer = lookupCI refererHeader headers
|
||||
!mCookie = lookupCI cookieHeader headers
|
||||
!mSecChUa = lookupCI secChUaHeader headers
|
||||
!mSecChPlat = lookupCI secChUaPlatformHeader headers
|
||||
!mSecFetchSite = lookupCI secFetchSiteHeader headers
|
||||
!mSecFetchMode = lookupCI secFetchModeHeader headers
|
||||
!mSecFetchDest = lookupCI secFetchDestHeader headers
|
||||
!mAccept = lookupCI acceptHeader headers
|
||||
!uaBytes = fromMaybe BS.empty mUserAgent
|
||||
!uaLen = BS.length uaBytes
|
||||
!depth = pathDepth path
|
||||
!entropy = shannonEntropyBytes path
|
||||
!queryCount = length (queryString req)
|
||||
!geo = fcGeoInfo
|
||||
!acceptLen = maybe 0 BS.length mAccept
|
||||
!modeValid = maybe False secFetchModeIsValid mSecFetchMode
|
||||
in FeatureVector
|
||||
{ fMissingAcceptLanguage = boolToDouble (not (isJust mAcceptLang))
|
||||
, fMissingUserAgent = boolToDouble (not (isJust mUserAgent))
|
||||
, fMissingAcceptEncoding = boolToDouble (not (isJust mAcceptEnc))
|
||||
, fMissingReferer = boolToDouble (not (isJust mReferer))
|
||||
, fHasCookie = boolToDouble (isJust mCookie)
|
||||
, fHasSecChUa = boolToDouble (isJust mSecChUa)
|
||||
, fHeaderCount = normalizedRatio (length headers) headerCountCap
|
||||
, fUaSecChConsistent = uaSecChConsistency mUserAgent mSecChUa
|
||||
, fUaBotKeyword = boolToDouble
|
||||
(maybe False uaContainsBotKeyword mUserAgent)
|
||||
, fUaHeadless = boolToDouble
|
||||
(maybe False uaContainsHeadlessMarker mUserAgent)
|
||||
, fUaCommonBrowser = boolToDouble
|
||||
(maybe False uaIsCommonBrowser mUserAgent)
|
||||
, fUaLength = normalizedRatio uaLen userAgentLengthCap
|
||||
, fPathDepth = normalizedRatio depth pathDepthCap
|
||||
, fPathEntropy = clamp01 (entropy / pathEntropyMax)
|
||||
, fSuspiciousPathExt = boolToDouble (pathHasSuspiciousExtension path)
|
||||
, fQueryParamCount = normalizedRatio queryCount queryParamCountCap
|
||||
, fMethodIdempotent = boolToDouble (methodIsIdempotent method)
|
||||
, fFlaggedAsn = boolToDouble (giFlaggedAsn geo)
|
||||
, fAsnConcentration = clamp01 fcAsnConcentration
|
||||
, fCountryUnknown = boolToDouble
|
||||
(case giCountryISO geo of
|
||||
Nothing -> True
|
||||
Just _ -> False)
|
||||
, fMissingSecFetchSite = boolToDouble (not (isJust mSecFetchSite))
|
||||
, fSecFetchModeValid = boolToDouble modeValid
|
||||
, fSecFetchContextCoherent = boolToDouble
|
||||
(secFetchTripleIsCoherent
|
||||
mSecFetchSite mSecFetchMode mSecFetchDest)
|
||||
, fChUaPlatformPresent = boolToDouble (isJust mSecChPlat)
|
||||
, fChUaPlatformConsistent = uaPlatformConsistency mUserAgent mSecChPlat
|
||||
, fAcceptIsWildcard = boolToDouble (acceptIsWildcard mAccept)
|
||||
, fAcceptValueLength = normalizedRatio acceptLen acceptValueLengthCap
|
||||
, fHeaderOrderCanonical = boolToDouble
|
||||
(headerOrderIsCanonicalBrowser headers)
|
||||
}
|
||||
|
||||
lookupCI :: CI ByteString -> [(CI ByteString, ByteString)] -> Maybe ByteString
|
||||
lookupCI k hs = case filter ((== k) . fst) hs of
|
||||
((_, v) : _) -> Just v
|
||||
_ -> Nothing
|
||||
|
|
@ -0,0 +1,359 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
Model.hs
|
||||
-}
|
||||
{-# LANGUAGE BangPatterns #-}
|
||||
{-# LANGUAGE DeriveGeneric #-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Aenebris.ML.Model
|
||||
( Tree(..)
|
||||
, Ensemble(..)
|
||||
, Objective(..)
|
||||
, MissingType(..)
|
||||
, SplitKind(..)
|
||||
, decisionTypeBits
|
||||
, missingTypeFromDecisionType
|
||||
, defaultLeftFromDecisionType
|
||||
, splitKindFromDecisionType
|
||||
, makeDecisionType
|
||||
, kCategoricalMask
|
||||
, kDefaultLeftMask
|
||||
, kMissingTypeShift
|
||||
, kMissingTypeMask
|
||||
, leafSentinel
|
||||
, noChildIndex
|
||||
, defaultRootIndex
|
||||
, currentEnsembleVersion
|
||||
, minimumEnsembleVersion
|
||||
, maximumEnsembleVersion
|
||||
, defaultSigmoidScale
|
||||
, makeLeafTree
|
||||
, makeStumpTree
|
||||
, makeStumpTreeWithMissing
|
||||
, makeCategoricalStumpTree
|
||||
, treeNodeCount
|
||||
, ensembleTreeCount
|
||||
, nodeIsLeaf
|
||||
, validateTree
|
||||
, validateEnsemble
|
||||
, parseObjective
|
||||
, renderObjective
|
||||
) where
|
||||
|
||||
import Data.Bits ((.&.), (.|.), shiftR, shiftL)
|
||||
import Data.Int (Int8)
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import Data.Vector (Vector)
|
||||
import qualified Data.Vector as V
|
||||
import qualified Data.Vector.Unboxed as VU
|
||||
import Data.Word (Word32)
|
||||
import GHC.Generics (Generic)
|
||||
|
||||
leafSentinel :: Int
|
||||
leafSentinel = -1
|
||||
|
||||
noChildIndex :: Int
|
||||
noChildIndex = -1
|
||||
|
||||
defaultRootIndex :: Int
|
||||
defaultRootIndex = 0
|
||||
|
||||
currentEnsembleVersion :: Int
|
||||
currentEnsembleVersion = 1
|
||||
|
||||
minimumEnsembleVersion :: Int
|
||||
minimumEnsembleVersion = 1
|
||||
|
||||
maximumEnsembleVersion :: Int
|
||||
maximumEnsembleVersion = 1
|
||||
|
||||
defaultSigmoidScale :: Double
|
||||
defaultSigmoidScale = 1.0
|
||||
|
||||
kCategoricalMask :: Int8
|
||||
kCategoricalMask = 1
|
||||
|
||||
kDefaultLeftMask :: Int8
|
||||
kDefaultLeftMask = 2
|
||||
|
||||
kMissingTypeShift :: Int
|
||||
kMissingTypeShift = 2
|
||||
|
||||
kMissingTypeMask :: Int8
|
||||
kMissingTypeMask = 12
|
||||
|
||||
data Objective
|
||||
= ObjectiveBinaryLogistic
|
||||
| ObjectiveRegression
|
||||
deriving (Eq, Show, Generic)
|
||||
|
||||
parseObjective :: Text -> Either String Objective
|
||||
parseObjective t = case T.toLower t of
|
||||
"binary" -> Right ObjectiveBinaryLogistic
|
||||
"binary_logistic" -> Right ObjectiveBinaryLogistic
|
||||
"logistic" -> Right ObjectiveBinaryLogistic
|
||||
"regression" -> Right ObjectiveRegression
|
||||
"regression_l2" -> Right ObjectiveRegression
|
||||
other -> Left ("Unknown objective: " <> T.unpack other)
|
||||
|
||||
renderObjective :: Objective -> Text
|
||||
renderObjective ObjectiveBinaryLogistic = "binary_logistic"
|
||||
renderObjective ObjectiveRegression = "regression"
|
||||
|
||||
data MissingType
|
||||
= MissingTypeNone
|
||||
| MissingTypeZero
|
||||
| MissingTypeNaN
|
||||
deriving (Eq, Show, Generic)
|
||||
|
||||
data SplitKind
|
||||
= SplitNumerical
|
||||
| SplitCategorical
|
||||
deriving (Eq, Show, Generic)
|
||||
|
||||
decisionTypeBits :: Int8 -> (SplitKind, Bool, MissingType)
|
||||
decisionTypeBits dt =
|
||||
( splitKindFromDecisionType dt
|
||||
, defaultLeftFromDecisionType dt
|
||||
, missingTypeFromDecisionType dt
|
||||
)
|
||||
|
||||
splitKindFromDecisionType :: Int8 -> SplitKind
|
||||
splitKindFromDecisionType dt
|
||||
| dt .&. kCategoricalMask /= 0 = SplitCategorical
|
||||
| otherwise = SplitNumerical
|
||||
|
||||
defaultLeftFromDecisionType :: Int8 -> Bool
|
||||
defaultLeftFromDecisionType dt = dt .&. kDefaultLeftMask /= 0
|
||||
|
||||
missingTypeFromDecisionType :: Int8 -> MissingType
|
||||
missingTypeFromDecisionType dt =
|
||||
case (dt .&. kMissingTypeMask) `shiftR` kMissingTypeShift of
|
||||
0 -> MissingTypeNone
|
||||
1 -> MissingTypeZero
|
||||
2 -> MissingTypeNaN
|
||||
_ -> MissingTypeNone
|
||||
|
||||
makeDecisionType :: SplitKind -> Bool -> MissingType -> Int8
|
||||
makeDecisionType kind defaultLeft mtype =
|
||||
let kBit = case kind of
|
||||
SplitNumerical -> 0
|
||||
SplitCategorical -> kCategoricalMask
|
||||
dBit = if defaultLeft then kDefaultLeftMask else 0
|
||||
mBits = case mtype of
|
||||
MissingTypeNone -> 0
|
||||
MissingTypeZero -> 1 `shiftL` kMissingTypeShift
|
||||
MissingTypeNaN -> 2 `shiftL` kMissingTypeShift
|
||||
in kBit .|. dBit .|. mBits
|
||||
|
||||
data Tree = Tree
|
||||
{ treeFeatureIdx :: !(VU.Vector Int)
|
||||
, treeThreshold :: !(VU.Vector Double)
|
||||
, treeLeftChild :: !(VU.Vector Int)
|
||||
, treeRightChild :: !(VU.Vector Int)
|
||||
, treeLeafValue :: !(VU.Vector Double)
|
||||
, treeDecisionType :: !(VU.Vector Int8)
|
||||
, treeCatBoundaries :: !(VU.Vector Int)
|
||||
, treeCatThreshold :: !(VU.Vector Word32)
|
||||
} deriving (Eq, Show)
|
||||
|
||||
data Ensemble = Ensemble
|
||||
{ ensembleVersion :: !Int
|
||||
, ensembleFeatureCount :: !Int
|
||||
, ensembleObjective :: !Objective
|
||||
, ensembleBaseScore :: !Double
|
||||
, ensembleSigmoidScale :: !Double
|
||||
, ensembleAverageOutput :: !Bool
|
||||
, ensembleTrees :: !(Vector Tree)
|
||||
} deriving (Eq, Show)
|
||||
|
||||
treeNodeCount :: Tree -> Int
|
||||
treeNodeCount = VU.length . treeFeatureIdx
|
||||
|
||||
ensembleTreeCount :: Ensemble -> Int
|
||||
ensembleTreeCount = V.length . ensembleTrees
|
||||
|
||||
nodeIsLeaf :: Tree -> Int -> Bool
|
||||
nodeIsLeaf t i = treeFeatureIdx t VU.! i == leafSentinel
|
||||
|
||||
makeLeafTree :: Double -> Tree
|
||||
makeLeafTree v = Tree
|
||||
{ treeFeatureIdx = VU.singleton leafSentinel
|
||||
, treeThreshold = VU.singleton 0.0
|
||||
, treeLeftChild = VU.singleton noChildIndex
|
||||
, treeRightChild = VU.singleton noChildIndex
|
||||
, treeLeafValue = VU.singleton v
|
||||
, treeDecisionType = VU.singleton 0
|
||||
, treeCatBoundaries = VU.empty
|
||||
, treeCatThreshold = VU.empty
|
||||
}
|
||||
|
||||
makeStumpTree :: Int -> Double -> Double -> Double -> Tree
|
||||
makeStumpTree featureIdx threshold leftValue rightValue =
|
||||
makeStumpTreeWithMissing featureIdx threshold leftValue rightValue
|
||||
True MissingTypeNone
|
||||
|
||||
makeStumpTreeWithMissing
|
||||
:: Int
|
||||
-> Double
|
||||
-> Double
|
||||
-> Double
|
||||
-> Bool
|
||||
-> MissingType
|
||||
-> Tree
|
||||
makeStumpTreeWithMissing featureIdx threshold leftValue rightValue defaultLeft mtype =
|
||||
Tree
|
||||
{ treeFeatureIdx = VU.fromList [featureIdx, leafSentinel, leafSentinel]
|
||||
, treeThreshold = VU.fromList [threshold, 0.0, 0.0]
|
||||
, treeLeftChild = VU.fromList [1, noChildIndex, noChildIndex]
|
||||
, treeRightChild = VU.fromList [2, noChildIndex, noChildIndex]
|
||||
, treeLeafValue = VU.fromList [0.0, leftValue, rightValue]
|
||||
, treeDecisionType = VU.fromList
|
||||
[ makeDecisionType SplitNumerical defaultLeft mtype
|
||||
, 0
|
||||
, 0
|
||||
]
|
||||
, treeCatBoundaries = VU.empty
|
||||
, treeCatThreshold = VU.empty
|
||||
}
|
||||
|
||||
makeCategoricalStumpTree
|
||||
:: Int
|
||||
-> [Word32]
|
||||
-> Double
|
||||
-> Double
|
||||
-> Tree
|
||||
makeCategoricalStumpTree featureIdx bitmap leftValue rightValue =
|
||||
let bitmapVec = VU.fromList bitmap
|
||||
boundaries = VU.fromList [0, VU.length bitmapVec]
|
||||
in Tree
|
||||
{ treeFeatureIdx = VU.fromList [featureIdx, leafSentinel, leafSentinel]
|
||||
, treeThreshold = VU.fromList [0.0, 0.0, 0.0]
|
||||
, treeLeftChild = VU.fromList [1, noChildIndex, noChildIndex]
|
||||
, treeRightChild = VU.fromList [2, noChildIndex, noChildIndex]
|
||||
, treeLeafValue = VU.fromList [0.0, leftValue, rightValue]
|
||||
, treeDecisionType = VU.fromList
|
||||
[ makeDecisionType SplitCategorical False MissingTypeNone
|
||||
, 0
|
||||
, 0
|
||||
]
|
||||
, treeCatBoundaries = boundaries
|
||||
, treeCatThreshold = bitmapVec
|
||||
}
|
||||
|
||||
validateTree :: Int -> Tree -> Either String ()
|
||||
validateTree featureCount t = do
|
||||
let nFeat = VU.length (treeFeatureIdx t)
|
||||
nThr = VU.length (treeThreshold t)
|
||||
nLeft = VU.length (treeLeftChild t)
|
||||
nRight = VU.length (treeRightChild t)
|
||||
nLeaf = VU.length (treeLeafValue t)
|
||||
nDec = VU.length (treeDecisionType t)
|
||||
if nFeat == 0
|
||||
then Left "Tree must contain at least one node"
|
||||
else Right ()
|
||||
if nThr == nFeat
|
||||
&& nLeft == nFeat
|
||||
&& nRight == nFeat
|
||||
&& nLeaf == nFeat
|
||||
&& nDec == nFeat
|
||||
then Right ()
|
||||
else Left "Tree SoA arrays have inconsistent lengths"
|
||||
validateCategoricalArrays t
|
||||
let nodeCount = nFeat
|
||||
indices = [0 .. nodeCount - 1]
|
||||
mapM_ (validateNode featureCount nodeCount t) indices
|
||||
|
||||
validateCategoricalArrays :: Tree -> Either String ()
|
||||
validateCategoricalArrays t =
|
||||
let nBoundaries = VU.length (treeCatBoundaries t)
|
||||
nThresholds = VU.length (treeCatThreshold t)
|
||||
in if nBoundaries == 0 && nThresholds == 0
|
||||
then Right ()
|
||||
else if nBoundaries < 2
|
||||
then Left "Categorical boundaries must have length >= 2 if present"
|
||||
else
|
||||
let lastBoundary = treeCatBoundaries t VU.! (nBoundaries - 1)
|
||||
in if lastBoundary == nThresholds
|
||||
then Right ()
|
||||
else Left
|
||||
$ "Categorical bitmap length "
|
||||
<> show nThresholds
|
||||
<> " does not match last boundary "
|
||||
<> show lastBoundary
|
||||
|
||||
validateNode :: Int -> Int -> Tree -> Int -> Either String ()
|
||||
validateNode featureCount nodeCount t i =
|
||||
let !fIdx = treeFeatureIdx t VU.! i
|
||||
!lIdx = treeLeftChild t VU.! i
|
||||
!rIdx = treeRightChild t VU.! i
|
||||
!dt = treeDecisionType t VU.! i
|
||||
kind = splitKindFromDecisionType dt
|
||||
in if fIdx == leafSentinel
|
||||
then validateLeafNode i lIdx rIdx
|
||||
else case kind of
|
||||
SplitNumerical -> validateSplitNode featureCount nodeCount i fIdx lIdx rIdx
|
||||
SplitCategorical ->
|
||||
validateCategoricalNode featureCount nodeCount t i fIdx lIdx rIdx
|
||||
|
||||
validateLeafNode :: Int -> Int -> Int -> Either String ()
|
||||
validateLeafNode i lIdx rIdx
|
||||
| lIdx /= noChildIndex =
|
||||
Left ("Leaf node " <> show i <> " must have left child = -1")
|
||||
| rIdx /= noChildIndex =
|
||||
Left ("Leaf node " <> show i <> " must have right child = -1")
|
||||
| otherwise = Right ()
|
||||
|
||||
validateSplitNode :: Int -> Int -> Int -> Int -> Int -> Int -> Either String ()
|
||||
validateSplitNode featureCount nodeCount i fIdx lIdx rIdx
|
||||
| fIdx < 0 || fIdx >= featureCount =
|
||||
Left ("Split node " <> show i
|
||||
<> " has out-of-range feature index " <> show fIdx)
|
||||
| lIdx < 0 || lIdx >= nodeCount =
|
||||
Left ("Split node " <> show i
|
||||
<> " has out-of-range left child " <> show lIdx)
|
||||
| rIdx < 0 || rIdx >= nodeCount =
|
||||
Left ("Split node " <> show i
|
||||
<> " has out-of-range right child " <> show rIdx)
|
||||
| lIdx == i =
|
||||
Left ("Split node " <> show i <> " has self-referential left child")
|
||||
| rIdx == i =
|
||||
Left ("Split node " <> show i <> " has self-referential right child")
|
||||
| lIdx == rIdx =
|
||||
Left ("Split node " <> show i <> " has identical left and right children")
|
||||
| otherwise = Right ()
|
||||
|
||||
validateCategoricalNode
|
||||
:: Int -> Int -> Tree -> Int -> Int -> Int -> Int -> Either String ()
|
||||
validateCategoricalNode featureCount nodeCount t i fIdx lIdx rIdx = do
|
||||
validateSplitNode featureCount nodeCount i fIdx lIdx rIdx
|
||||
let catIdx = floor (treeThreshold t VU.! i) :: Int
|
||||
nBound = VU.length (treeCatBoundaries t)
|
||||
if nBound < 2
|
||||
then Left ("Categorical node " <> show i
|
||||
<> " requires non-empty cat_boundaries")
|
||||
else if catIdx < 0 || catIdx >= nBound - 1
|
||||
then Left ("Categorical node " <> show i
|
||||
<> " has out-of-range bitmap slice index " <> show catIdx)
|
||||
else Right ()
|
||||
|
||||
validateEnsemble :: Int -> Ensemble -> Either String ()
|
||||
validateEnsemble expectedFeatures ens = do
|
||||
let v = ensembleVersion ens
|
||||
if v < minimumEnsembleVersion || v > maximumEnsembleVersion
|
||||
then Left ("Unsupported ensemble version: " <> show v)
|
||||
else Right ()
|
||||
if ensembleFeatureCount ens /= expectedFeatures
|
||||
then Left ("Ensemble feature count " <> show (ensembleFeatureCount ens)
|
||||
<> " does not match expected " <> show expectedFeatures)
|
||||
else Right ()
|
||||
if V.null (ensembleTrees ens)
|
||||
then Left "Ensemble must contain at least one tree"
|
||||
else Right ()
|
||||
V.imapM_
|
||||
(\i tree -> case validateTree (ensembleFeatureCount ens) tree of
|
||||
Right () -> Right ()
|
||||
Left err -> Left ("Tree " <> show i <> ": " <> err))
|
||||
(ensembleTrees ens)
|
||||
|
|
@ -19,6 +19,52 @@ import Aenebris.TLS
|
|||
import Aenebris.Tunnel
|
||||
import Aenebris.Middleware.Security
|
||||
import Aenebris.Middleware.Redirect
|
||||
import Aenebris.RateLimit (RateLimiter, createRateLimiter, parseRateSpec, rateLimitMiddleware)
|
||||
import Aenebris.DDoS.EarlyData (earlyDataGuard)
|
||||
import Aenebris.DDoS.MemoryShed
|
||||
( MemoryShed
|
||||
, MemoryShedConfig(..)
|
||||
, defaultHighWaterFraction
|
||||
, memoryShedMiddleware
|
||||
, newMemoryShed
|
||||
, startMemoryShedPoller
|
||||
)
|
||||
import Aenebris.DDoS.IPJail
|
||||
( IPJail
|
||||
, defaultIPJailConfig
|
||||
, ipJailMiddleware
|
||||
, newIPJail
|
||||
, startJailSweeper
|
||||
)
|
||||
import Aenebris.DDoS.ConnLimit
|
||||
( ConnLimiter
|
||||
, ConnLimitConfig(..)
|
||||
, connLimitOnClose
|
||||
, connLimitOnOpen
|
||||
, newConnLimiter
|
||||
)
|
||||
import Aenebris.Fingerprint.JA4H (ja4hMiddleware)
|
||||
import Aenebris.WAF.Engine (wafMiddleware)
|
||||
import Aenebris.WAF.Patterns (defaultRuleSet)
|
||||
import Aenebris.WAF.Rule (RuleSet)
|
||||
import Aenebris.Honeypot
|
||||
( HoneypotConfig(..)
|
||||
, buildHoneypotConfig
|
||||
, honeypotMiddleware
|
||||
)
|
||||
import Aenebris.Geo
|
||||
( Geo
|
||||
, buildGeoConfig
|
||||
, geoConfig
|
||||
, gcCountryDb
|
||||
, gcAsnDb
|
||||
, gcFlaggedAsns
|
||||
, gcBlockedCountries
|
||||
, openGeo
|
||||
, startAsnSweeper
|
||||
, geoMiddleware
|
||||
)
|
||||
import Control.Concurrent.STM (TVar, newTVarIO)
|
||||
import Control.Concurrent.Async (Async, async, waitAnyCancel)
|
||||
import Control.Exception (try, SomeException)
|
||||
import Data.Function ((&))
|
||||
|
|
@ -30,13 +76,22 @@ import Data.Ord (comparing)
|
|||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import qualified Data.Text.Encoding as TE
|
||||
import Network.HTTP.Client (Manager, httpLbs, withResponse, parseRequest, RequestBody(..), brRead)
|
||||
import Network.HTTP.Client (Manager, withResponse, parseRequest, RequestBody(..), brRead)
|
||||
import qualified Network.HTTP.Client as HTTP
|
||||
import Network.HTTP.Types
|
||||
import Network.Wai
|
||||
import Data.ByteString.Builder (byteString)
|
||||
import Control.Monad (unless)
|
||||
import Network.Wai.Handler.Warp (run, defaultSettings, setPort)
|
||||
import Network.Wai.Handler.Warp
|
||||
( Settings
|
||||
, defaultSettings
|
||||
, runSettings
|
||||
, setMaxTotalHeaderLength
|
||||
, setOnClose
|
||||
, setOnOpen
|
||||
, setPort
|
||||
, setTimeout
|
||||
)
|
||||
import Network.Wai.Handler.WarpTLS (runTLS)
|
||||
import System.IO (hPutStrLn, stderr)
|
||||
import Data.ByteString (ByteString)
|
||||
|
|
@ -50,6 +105,12 @@ data ProxyState = ProxyState
|
|||
, psLoadBalancers :: Map Text LoadBalancer -- upstream name -> load balancer
|
||||
, psHealthCheckers :: [Async ()]
|
||||
, psManager :: Manager
|
||||
, psRateLimiter :: Maybe RateLimiter
|
||||
, psMemoryShed :: Maybe MemoryShed
|
||||
, psIPJail :: Maybe IPJail
|
||||
, psConnLimiter :: Maybe ConnLimiter
|
||||
, psWafRuleSet :: TVar RuleSet
|
||||
, psGeo :: Maybe Geo
|
||||
}
|
||||
|
||||
-- | Initialize proxy state from config
|
||||
|
|
@ -62,11 +123,55 @@ initProxyState config manager = do
|
|||
-- Start health checkers for all upstreams
|
||||
checkers <- mapM startUpstreamHealthChecker (configUpstreams config)
|
||||
|
||||
rateLimiter <- case configRateLimit config >>= parseRateSpec of
|
||||
Just spec -> Just <$> createRateLimiter spec
|
||||
Nothing -> pure Nothing
|
||||
|
||||
let ddos = configDDoS config
|
||||
|
||||
memShed <- case ddos >>= ddosMemoryShedBytes of
|
||||
Just budgetBytes -> do
|
||||
ms <- newMemoryShed
|
||||
let cfg = MemoryShedConfig
|
||||
{ mscHeapBudgetBytes = fromInteger budgetBytes
|
||||
, mscHighWaterFraction = fromMaybe defaultHighWaterFraction (ddos >>= ddosMemoryShedHighWater)
|
||||
, mscPollIntervalMicros = 1000000
|
||||
}
|
||||
_ <- startMemoryShedPoller cfg ms
|
||||
pure (Just ms)
|
||||
Nothing -> pure Nothing
|
||||
|
||||
ipJail <- case ddos >>= ddosJailCooldownSeconds of
|
||||
Just _ -> do
|
||||
j <- newIPJail
|
||||
_ <- startJailSweeper defaultIPJailConfig j
|
||||
pure (Just j)
|
||||
Nothing -> pure Nothing
|
||||
|
||||
connLimiter <- case ddos >>= ddosPerIPConnections of
|
||||
Just n -> Just <$> newConnLimiter (ConnLimitConfig n)
|
||||
Nothing -> pure Nothing
|
||||
|
||||
wafVar <- newTVarIO defaultRuleSet
|
||||
|
||||
geoHandle <- case buildGeoConfig (configGeo config) of
|
||||
Just gcfg -> do
|
||||
g <- openGeo gcfg
|
||||
_ <- startAsnSweeper g
|
||||
pure (Just g)
|
||||
Nothing -> pure Nothing
|
||||
|
||||
return ProxyState
|
||||
{ psConfig = config
|
||||
, psLoadBalancers = lbMap
|
||||
, psHealthCheckers = checkers
|
||||
, psManager = manager
|
||||
, psRateLimiter = rateLimiter
|
||||
, psMemoryShed = memShed
|
||||
, psIPJail = ipJail
|
||||
, psConnLimiter = connLimiter
|
||||
, psWafRuleSet = wafVar
|
||||
, psGeo = geoHandle
|
||||
}
|
||||
where
|
||||
-- Create load balancer for an upstream
|
||||
|
|
@ -112,49 +217,118 @@ startProxy ProxyState{..} = do
|
|||
case configListen psConfig of
|
||||
[] -> error "No listen ports configured"
|
||||
listenConfigs -> do
|
||||
-- Launch a server for each listen port concurrently
|
||||
servers <- mapM (launchServer psConfig psLoadBalancers psManager) listenConfigs
|
||||
case psRateLimiter of
|
||||
Just _ -> putStrLn "Rate limiting enabled"
|
||||
Nothing -> pure ()
|
||||
|
||||
-- Wait for any server to fail (shouldn't happen in normal operation)
|
||||
waitAnyCancel servers
|
||||
putStrLn "WAF enabled (Phase 1: paranoia level 2, default rule pack)"
|
||||
case buildHoneypotConfig (configHoneypot psConfig) of
|
||||
Just hp -> putStrLn $ "Honeypot enabled (" ++ show (length (hpPatterns hp))
|
||||
++ " trap patterns, action=" ++ show (hpAction hp) ++ ")"
|
||||
Nothing -> pure ()
|
||||
case psGeo of
|
||||
Just g ->
|
||||
let gc = geoConfig g
|
||||
parts = [ "country_db=" ++ maybe "off" (const "on") (gcCountryDb gc)
|
||||
, "asn_db=" ++ maybe "off" (const "on") (gcAsnDb gc)
|
||||
, "blocked=" ++ show (length (gcBlockedCountries gc))
|
||||
, "flagged_asns=" ++ show (length (gcFlaggedAsns gc))
|
||||
]
|
||||
in putStrLn $ "Geo/ASN enabled (" ++ unwords parts ++ ")"
|
||||
Nothing -> pure ()
|
||||
servers <- mapM (launchServer psConfig psLoadBalancers psManager psRateLimiter psMemoryShed psIPJail psConnLimiter psWafRuleSet psGeo) listenConfigs
|
||||
|
||||
_ <- waitAnyCancel servers
|
||||
|
||||
putStrLn "All servers stopped"
|
||||
|
||||
-- | Launch a single server instance (HTTP or HTTPS)
|
||||
launchServer :: Config -> Map Text LoadBalancer -> Manager -> ListenConfig -> IO (Async ())
|
||||
launchServer config loadBalancers manager listenConfig = async $ do
|
||||
launchServer
|
||||
:: Config
|
||||
-> Map Text LoadBalancer
|
||||
-> Manager
|
||||
-> Maybe RateLimiter
|
||||
-> Maybe MemoryShed
|
||||
-> Maybe IPJail
|
||||
-> Maybe ConnLimiter
|
||||
-> TVar RuleSet
|
||||
-> Maybe Geo
|
||||
-> ListenConfig
|
||||
-> IO (Async ())
|
||||
launchServer config loadBalancers manager mRateLimiter mMemShed mIPJail mConnLim wafVar mGeo listenConfig = async $ do
|
||||
let port = listenPort listenConfig
|
||||
shouldRedirect = fromMaybe False (listenRedirectHTTPS listenConfig)
|
||||
ddosCfg = fromMaybe defaultDDoSConfig (configDDoS config)
|
||||
|
||||
-- Build the base application
|
||||
baseApp = proxyApp config loadBalancers manager
|
||||
|
||||
-- Add security headers (production level by default)
|
||||
securedApp = addSecurityHeaders defaultSecurityConfig baseApp
|
||||
fingerprintedApp = ja4hMiddleware baseApp
|
||||
|
||||
wafApp = wafMiddleware wafVar fingerprintedApp
|
||||
|
||||
securedApp = addSecurityHeaders defaultSecurityConfig wafApp
|
||||
|
||||
earlyDataApp = if ddosEarlyDataReject ddosCfg
|
||||
then earlyDataGuard securedApp
|
||||
else securedApp
|
||||
|
||||
mHoneypotCfg = buildHoneypotConfig (configHoneypot config)
|
||||
|
||||
honeypotApp = case mHoneypotCfg of
|
||||
Just hp -> honeypotMiddleware hp mIPJail earlyDataApp
|
||||
Nothing -> earlyDataApp
|
||||
|
||||
geoApp = case mGeo of
|
||||
Just g -> geoMiddleware g mIPJail honeypotApp
|
||||
Nothing -> honeypotApp
|
||||
|
||||
jailedApp = case mIPJail of
|
||||
Just j -> ipJailMiddleware j geoApp
|
||||
Nothing -> geoApp
|
||||
|
||||
shedApp = case mMemShed of
|
||||
Just ms -> memoryShedMiddleware ms jailedApp
|
||||
Nothing -> jailedApp
|
||||
|
||||
limitedApp = case mRateLimiter of
|
||||
Just rl -> rateLimitMiddleware rl shedApp
|
||||
Nothing -> shedApp
|
||||
|
||||
warpSettings = applyDDoSSettings ddosCfg mConnLim (defaultSettings & setPort port)
|
||||
|
||||
case listenTLS listenConfig of
|
||||
Nothing -> do
|
||||
-- Plain HTTP server
|
||||
let app = if shouldRedirect
|
||||
then httpsRedirect securedApp -- Redirect all HTTP to HTTPS
|
||||
else securedApp
|
||||
then httpsRedirect limitedApp
|
||||
else limitedApp
|
||||
|
||||
putStrLn $ "✓ HTTP server listening on :" ++ show port
|
||||
if shouldRedirect
|
||||
then putStrLn $ " └─ Redirecting all traffic to HTTPS"
|
||||
else return ()
|
||||
|
||||
run port app
|
||||
runSettings warpSettings app
|
||||
|
||||
Just tlsConfig -> do
|
||||
-- HTTPS server - check if single cert or SNI
|
||||
let isSNI = case tlsSNI tlsConfig of
|
||||
Just domains -> not (null domains)
|
||||
Nothing -> False
|
||||
|
||||
if isSNI
|
||||
then launchHTTPSWithSNI port tlsConfig securedApp
|
||||
else launchHTTPS port tlsConfig securedApp
|
||||
then launchHTTPSWithSNI port tlsConfig limitedApp
|
||||
else launchHTTPS port tlsConfig limitedApp
|
||||
|
||||
applyDDoSSettings :: DDoSConfig -> Maybe ConnLimiter -> Settings -> Settings
|
||||
applyDDoSSettings ddos mConnLim s0 =
|
||||
let s1 = case ddosMaxHeaderBytes ddos of
|
||||
Just n -> setMaxTotalHeaderLength n s1Inner
|
||||
Nothing -> s1Inner
|
||||
s1Inner = case ddosSlowlorisSeconds ddos of
|
||||
Just n -> setTimeout n s0
|
||||
Nothing -> s0
|
||||
s2 = case mConnLim of
|
||||
Just cl -> setOnClose (connLimitOnClose cl) (setOnOpen (connLimitOnOpen cl) s1)
|
||||
Nothing -> s1
|
||||
in s2
|
||||
|
||||
-- | Launch HTTPS server with single certificate
|
||||
launchHTTPS :: Int -> TLSConfig -> Application -> IO ()
|
||||
|
|
@ -331,18 +505,24 @@ selectUpstream config hostHeader requestPath =
|
|||
-- | Forward request to backend server with streaming support
|
||||
forwardRequest :: Manager -> Request -> Text -> (Response -> IO ResponseReceived) -> IO ResponseReceived
|
||||
forwardRequest manager clientReq backendHost respond = do
|
||||
requestBody <- strictRequestBody clientReq
|
||||
|
||||
let backendUrl = "http://" ++ T.unpack backendHost ++
|
||||
BS8.unpack (rawPathInfo clientReq) ++
|
||||
BS8.unpack (rawQueryString clientReq)
|
||||
|
||||
initReq <- parseRequest backendUrl
|
||||
|
||||
let backendReq = initReq
|
||||
let streamingBody = case requestBodyLength clientReq of
|
||||
ChunkedBody ->
|
||||
RequestBodyStreamChunked $ \needsPopper ->
|
||||
needsPopper (getRequestBodyChunk clientReq)
|
||||
KnownLength len ->
|
||||
RequestBodyStream (fromIntegral len) $ \needsPopper ->
|
||||
needsPopper (getRequestBodyChunk clientReq)
|
||||
|
||||
backendReq = initReq
|
||||
{ HTTP.method = requestMethod clientReq
|
||||
, HTTP.requestHeaders = filterHeaders (requestHeaders clientReq)
|
||||
, HTTP.requestBody = RequestBodyLBS requestBody
|
||||
, HTTP.requestBody = streamingBody
|
||||
}
|
||||
|
||||
withResponse backendReq manager $ \backendResponse -> do
|
||||
|
|
@ -419,18 +599,6 @@ filterHeaders headers = filter (\(name, _) -> name `notElem` hopByHopHeaders) he
|
|||
, "Upgrade"
|
||||
]
|
||||
|
||||
-- | Filter headers for WebSocket upgrade (preserve Upgrade and Connection)
|
||||
filterHeadersForUpgrade :: [(HeaderName, BS.ByteString)] -> [(HeaderName, BS.ByteString)]
|
||||
filterHeadersForUpgrade headers = filter (\(name, _) -> name `notElem` hopByHopHeaders) headers
|
||||
where
|
||||
hopByHopHeaders =
|
||||
[ "Keep-Alive"
|
||||
, "Proxy-Authenticate"
|
||||
, "Proxy-Authorization"
|
||||
, "TE"
|
||||
, "Trailers"
|
||||
]
|
||||
|
||||
-- | Log incoming request
|
||||
logRequest :: Request -> IO ()
|
||||
logRequest req = do
|
||||
|
|
@ -441,12 +609,6 @@ logRequest req = do
|
|||
|
||||
putStrLn $ "[→] " ++ method' ++ " " ++ path ++ query ++ " (Host: " ++ BS8.unpack host ++ ")"
|
||||
|
||||
-- | Log response
|
||||
logResponse :: Response -> IO ()
|
||||
logResponse res = do
|
||||
let (Status code msg) = responseStatus res
|
||||
putStrLn $ "[←] " ++ show code ++ " " ++ BS8.unpack msg
|
||||
|
||||
-- Helper: zipWithM
|
||||
zipWithM :: Monad m => (a -> b -> m c) -> [a] -> [b] -> m [c]
|
||||
zipWithM f xs ys = sequence (zipWith f xs ys)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,189 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
RateLimit.hs
|
||||
-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE RecordWildCards #-}
|
||||
|
||||
module Aenebris.RateLimit
|
||||
( RateLimiter
|
||||
, RateSpec(..)
|
||||
, Decision(..)
|
||||
, Key
|
||||
, parseRateSpec
|
||||
, createRateLimiter
|
||||
, checkLimit
|
||||
, rateLimitMiddleware
|
||||
, clientIPKey
|
||||
, pathClassKey
|
||||
, defaultBurstMultiplier
|
||||
, bucketPruneThreshold
|
||||
, bucketPruneAge
|
||||
) where
|
||||
|
||||
import Control.Concurrent.STM
|
||||
( STM
|
||||
, TVar
|
||||
, atomically
|
||||
, newTVarIO
|
||||
, readTVar
|
||||
, writeTVar
|
||||
)
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString.Char8 as BS8
|
||||
import Data.Map.Strict (Map)
|
||||
import qualified Data.Map.Strict as Map
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import qualified Data.Text.Read as TR
|
||||
import Data.Time.Clock.POSIX (POSIXTime, getPOSIXTime)
|
||||
import Network.HTTP.Types (status429)
|
||||
import Network.Socket
|
||||
( HostAddress6
|
||||
, SockAddr(..)
|
||||
, hostAddress6ToTuple
|
||||
, hostAddressToTuple
|
||||
)
|
||||
import Network.Wai
|
||||
( Middleware
|
||||
, Request
|
||||
, rawPathInfo
|
||||
, remoteHost
|
||||
, responseLBS
|
||||
)
|
||||
import Numeric (showHex)
|
||||
import Text.Printf (printf)
|
||||
|
||||
secondsPerMinute :: Double
|
||||
secondsPerMinute = 60
|
||||
|
||||
secondsPerHour :: Double
|
||||
secondsPerHour = 3600
|
||||
|
||||
defaultBurstMultiplier :: Double
|
||||
defaultBurstMultiplier = 1.0
|
||||
|
||||
bucketPruneThreshold :: Int
|
||||
bucketPruneThreshold = 10000
|
||||
|
||||
bucketPruneAge :: POSIXTime
|
||||
bucketPruneAge = 300
|
||||
|
||||
data RateSpec = RateSpec
|
||||
{ rsCapacity :: !Double
|
||||
, rsRefillPerSec :: !Double
|
||||
} deriving (Show, Eq)
|
||||
|
||||
data Bucket = Bucket
|
||||
{ bkTokens :: !Double
|
||||
, bkLastRefill :: !POSIXTime
|
||||
} deriving (Show, Eq)
|
||||
|
||||
type Key = (ByteString, ByteString)
|
||||
|
||||
data RateLimiter = RateLimiter
|
||||
{ rlBuckets :: TVar (Map Key Bucket)
|
||||
, rlSpec :: !RateSpec
|
||||
}
|
||||
|
||||
data Decision
|
||||
= Allowed !Double
|
||||
| Denied !Double
|
||||
deriving (Show, Eq)
|
||||
|
||||
parseRateSpec :: Text -> Maybe RateSpec
|
||||
parseRateSpec spec =
|
||||
case T.splitOn "/" (T.strip spec) of
|
||||
[countTxt, unitTxt] -> do
|
||||
count <- parsePositiveInt countTxt
|
||||
perSec <- unitToPerSecond (T.toLower (T.strip unitTxt))
|
||||
let capacity = fromIntegral count
|
||||
Just RateSpec
|
||||
{ rsCapacity = capacity * defaultBurstMultiplier
|
||||
, rsRefillPerSec = capacity * perSec
|
||||
}
|
||||
_ -> Nothing
|
||||
where
|
||||
parsePositiveInt :: Text -> Maybe Int
|
||||
parsePositiveInt t = case TR.decimal (T.strip t) of
|
||||
Right (n, rest) | T.null rest && n > 0 -> Just n
|
||||
_ -> Nothing
|
||||
|
||||
unitToPerSecond :: Text -> Maybe Double
|
||||
unitToPerSecond u
|
||||
| u == "s" || u == "sec" || u == "second" = Just 1
|
||||
| u == "m" || u == "min" || u == "minute" = Just (1 / secondsPerMinute)
|
||||
| u == "h" || u == "hr" || u == "hour" = Just (1 / secondsPerHour)
|
||||
| otherwise = Nothing
|
||||
|
||||
createRateLimiter :: RateSpec -> IO RateLimiter
|
||||
createRateLimiter spec = do
|
||||
tv <- newTVarIO Map.empty
|
||||
pure RateLimiter { rlBuckets = tv, rlSpec = spec }
|
||||
|
||||
checkLimit :: RateLimiter -> Key -> POSIXTime -> STM Decision
|
||||
checkLimit RateLimiter{..} key now = do
|
||||
buckets <- readTVar rlBuckets
|
||||
let RateSpec cap rate = rlSpec
|
||||
current = Map.findWithDefault (Bucket cap now) key buckets
|
||||
elapsed = realToFrac (now - bkLastRefill current) :: Double
|
||||
refilled = min cap (bkTokens current + max 0 elapsed * rate)
|
||||
if refilled >= 1
|
||||
then do
|
||||
let next = Bucket (refilled - 1) now
|
||||
writeTVar rlBuckets $! pruneIfLarge now (Map.insert key next buckets)
|
||||
pure (Allowed (refilled - 1))
|
||||
else do
|
||||
let next = Bucket refilled now
|
||||
retry = if rate > 0 then (1 - refilled) / rate else 0
|
||||
writeTVar rlBuckets $! pruneIfLarge now (Map.insert key next buckets)
|
||||
pure (Denied retry)
|
||||
|
||||
pruneIfLarge :: POSIXTime -> Map Key Bucket -> Map Key Bucket
|
||||
pruneIfLarge now m
|
||||
| Map.size m < bucketPruneThreshold = m
|
||||
| otherwise = Map.filter (\b -> now - bkLastRefill b <= bucketPruneAge) m
|
||||
|
||||
rateLimitMiddleware :: RateLimiter -> Middleware
|
||||
rateLimitMiddleware rl app req respond = do
|
||||
now <- getPOSIXTime
|
||||
let key = (clientIPKey req, pathClassKey req)
|
||||
decision <- atomically (checkLimit rl key now)
|
||||
case decision of
|
||||
Allowed _remaining -> app req respond
|
||||
Denied retry -> respond $ responseLBS status429
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, ("X-RateLimit-Limit", intBS (floor (rsCapacity (rlSpec rl)) :: Int))
|
||||
, ("X-RateLimit-Remaining", "0")
|
||||
, ("X-RateLimit-Reset", intBS (ceilingInt retry))
|
||||
, ("Retry-After", intBS (ceilingInt retry))
|
||||
]
|
||||
"Too Many Requests"
|
||||
where
|
||||
ceilingInt x = max 1 (ceiling x :: Int)
|
||||
intBS n = BS8.pack (show n)
|
||||
|
||||
clientIPKey :: Request -> ByteString
|
||||
clientIPKey req = case remoteHost req of
|
||||
SockAddrInet _ ha ->
|
||||
let (a, b, c, d) = hostAddressToTuple ha
|
||||
in BS8.pack (printf "%d.%d.%d.%d" a b c d)
|
||||
SockAddrInet6 _ _ ha6 _ -> v6Bytes ha6
|
||||
SockAddrUnix p -> BS8.pack ("unix:" <> p)
|
||||
where
|
||||
v6Bytes :: HostAddress6 -> ByteString
|
||||
v6Bytes ha =
|
||||
let (a, b, c, d, e, f, g, h) = hostAddress6ToTuple ha
|
||||
parts = [a, b, c, d, e, f, g, h]
|
||||
in BS8.pack (joinColons (map (`showHex` "") parts))
|
||||
|
||||
joinColons :: [String] -> String
|
||||
joinColons [] = ""
|
||||
joinColons [x] = x
|
||||
joinColons (x : xs) = x <> ":" <> joinColons xs
|
||||
|
||||
pathClassKey :: Request -> ByteString
|
||||
pathClassKey req =
|
||||
case BS8.split '/' (rawPathInfo req) of
|
||||
(_ : seg : _) | not (BS8.null seg) -> "/" <> seg
|
||||
_ -> "/"
|
||||
|
|
@ -0,0 +1,176 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
Engine.hs
|
||||
-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
{-# LANGUAGE RecordWildCards #-}
|
||||
|
||||
module Aenebris.WAF.Engine
|
||||
( WafDecision(..)
|
||||
, MatchResult(..)
|
||||
, evaluatePhase1
|
||||
, wafMiddleware
|
||||
, wafResponseHeader
|
||||
, extractTargets
|
||||
, detectAmbiguousFraming
|
||||
, detectObsoleteLineFolding
|
||||
, detectDuplicateHost
|
||||
, runOperator
|
||||
, scoreFromMatches
|
||||
, decisionFromScore
|
||||
) where
|
||||
|
||||
import Aenebris.WAF.Rule
|
||||
( Action(..)
|
||||
, Operator(..)
|
||||
, Phase(..)
|
||||
, Rule(..)
|
||||
, RuleSet(..)
|
||||
, Severity(..)
|
||||
, Target(..)
|
||||
, runRegex
|
||||
, severityScore
|
||||
)
|
||||
import Control.Concurrent.STM (TVar, readTVarIO)
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString as BS
|
||||
import qualified Data.ByteString.Char8 as BC
|
||||
import Data.CaseInsensitive (CI)
|
||||
import qualified Data.CaseInsensitive as CI
|
||||
import Data.Char (toLower)
|
||||
import Data.Word (Word32)
|
||||
import Network.HTTP.Types (Status, mkStatus)
|
||||
import Network.HTTP.Types.URI (urlDecode)
|
||||
import Network.Wai
|
||||
( Middleware
|
||||
, Request
|
||||
, rawPathInfo
|
||||
, rawQueryString
|
||||
, requestHeaders
|
||||
, requestMethod
|
||||
, responseLBS
|
||||
)
|
||||
|
||||
data MatchResult = MatchResult
|
||||
{ mrRuleId :: !Word32
|
||||
, mrRuleName :: !ByteString
|
||||
, mrSeverity :: !Severity
|
||||
, mrAction :: !Action
|
||||
} deriving (Eq, Show)
|
||||
|
||||
data WafDecision
|
||||
= Allow
|
||||
| Deny !Int ![MatchResult]
|
||||
deriving (Eq, Show)
|
||||
|
||||
extractTargets :: Request -> Target -> [ByteString]
|
||||
extractTargets req t = case t of
|
||||
TargetMethod -> [requestMethod req]
|
||||
TargetPath -> [rawPathInfo req, urlDecode True (rawPathInfo req)]
|
||||
TargetQuery -> [rawQueryString req, urlDecode True (rawQueryString req)]
|
||||
TargetHeaderValue name ->
|
||||
[v | (n, v) <- requestHeaders req, n == CI.mk name]
|
||||
TargetAnyHeaderName ->
|
||||
[CI.original n | (n, _) <- requestHeaders req]
|
||||
TargetAnyHeaderValue ->
|
||||
[v | (_, v) <- requestHeaders req]
|
||||
TargetHost ->
|
||||
[v | (n, v) <- requestHeaders req, n == CI.mk "host"]
|
||||
TargetUserAgent ->
|
||||
[v | (n, v) <- requestHeaders req, n == CI.mk "user-agent"]
|
||||
|
||||
runOperator :: Operator -> ByteString -> Bool
|
||||
runOperator op input = case op of
|
||||
OpRegex r -> runRegex r input
|
||||
OpStreq s -> BC.map toLower input == BC.map toLower s
|
||||
OpContains s ->
|
||||
let needle = BC.map toLower s
|
||||
hay = BC.map toLower input
|
||||
in BS.isInfixOf needle hay
|
||||
OpAnyMatch ss ->
|
||||
let hay = BC.map toLower input
|
||||
in any (\s -> BS.isInfixOf (BC.map toLower s) hay) ss
|
||||
|
||||
ruleMatches :: Request -> Rule -> Bool
|
||||
ruleMatches req r =
|
||||
case ruleOp r of
|
||||
OpStreq "__synthetic__" -> syntheticMatches req r
|
||||
op ->
|
||||
let inputs = concatMap (extractTargets req) (ruleTargets r)
|
||||
in any (runOperator op) inputs
|
||||
|
||||
syntheticMatches :: Request -> Rule -> Bool
|
||||
syntheticMatches req r = case ruleName r of
|
||||
"ambiguous-framing-cl-te" -> detectAmbiguousFraming req
|
||||
"obsolete-line-folding" -> detectObsoleteLineFolding req
|
||||
"duplicate-host-header" -> detectDuplicateHost req
|
||||
_ -> False
|
||||
|
||||
detectAmbiguousFraming :: Request -> Bool
|
||||
detectAmbiguousFraming req =
|
||||
let hs = requestHeaders req
|
||||
hasCL = any ((== CI.mk "content-length") . fst) hs
|
||||
hasTE = any ((== CI.mk "transfer-encoding") . fst) hs
|
||||
in hasCL && hasTE
|
||||
|
||||
detectObsoleteLineFolding :: Request -> Bool
|
||||
detectObsoleteLineFolding req =
|
||||
let hs = requestHeaders req
|
||||
in any (containsLineFolding . snd) hs
|
||||
where
|
||||
containsLineFolding bs =
|
||||
let bytes = BS.unpack bs
|
||||
in hasFold bytes
|
||||
hasFold (a : b : c : rest)
|
||||
| a == 0x0d && b == 0x0a && (c == 0x20 || c == 0x09) = True
|
||||
| otherwise = hasFold (b : c : rest)
|
||||
hasFold (a : b : _)
|
||||
| (a == 0x0a) && (b == 0x20 || b == 0x09) = True
|
||||
hasFold _ = False
|
||||
|
||||
detectDuplicateHost :: Request -> Bool
|
||||
detectDuplicateHost req =
|
||||
let hostCount = length [n | (n, _) <- requestHeaders req, n == CI.mk "host"]
|
||||
in hostCount > 1
|
||||
|
||||
scoreFromMatches :: [MatchResult] -> Int
|
||||
scoreFromMatches = sum . map (severityScore . mrSeverity)
|
||||
|
||||
evaluatePhase1 :: RuleSet -> Request -> ([MatchResult], WafDecision)
|
||||
evaluatePhase1 rs req =
|
||||
let activeRules = filter (\r -> rulePhase r == PhaseHeaders
|
||||
&& ruleParanoia r <= rsParanoia rs)
|
||||
(rsRules rs)
|
||||
matches =
|
||||
[ MatchResult (ruleId r) (ruleName r) (ruleSeverity r) (ruleAction r)
|
||||
| r <- activeRules, ruleMatches req r
|
||||
]
|
||||
hasBlock = any ((== Block) . mrAction) matches
|
||||
score = scoreFromMatches matches
|
||||
decision = decisionFromScore hasBlock score (rsInboundThreshold rs) matches
|
||||
in (matches, decision)
|
||||
|
||||
decisionFromScore :: Bool -> Int -> Int -> [MatchResult] -> WafDecision
|
||||
decisionFromScore hasBlock score threshold matches
|
||||
| hasBlock = Deny score matches
|
||||
| score >= threshold = Deny score matches
|
||||
| otherwise = Allow
|
||||
|
||||
wafResponseHeader :: CI ByteString
|
||||
wafResponseHeader = "x-aenebris-waf"
|
||||
|
||||
status403 :: Status
|
||||
status403 = mkStatus 403 "Forbidden"
|
||||
|
||||
wafMiddleware :: TVar RuleSet -> Middleware
|
||||
wafMiddleware rsVar app req respond = do
|
||||
rs <- readTVarIO rsVar
|
||||
let (_matches, decision) = evaluatePhase1 rs req
|
||||
case decision of
|
||||
Allow -> app req respond
|
||||
Deny score _ ->
|
||||
respond $ responseLBS status403
|
||||
[ ("Content-Type", "text/plain; charset=utf-8")
|
||||
, (wafResponseHeader, "blocked score=" <> BC.pack (show score))
|
||||
]
|
||||
"403 Forbidden — request blocked by Aenebris WAF"
|
||||
|
|
@ -0,0 +1,168 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
Patterns.hs
|
||||
-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Aenebris.WAF.Patterns
|
||||
( defaultRuleSet
|
||||
, defaultRules
|
||||
, sqliRules
|
||||
, xssRules
|
||||
, traversalRules
|
||||
, crlfRules
|
||||
, protocolRules
|
||||
, ruleNamePrefix
|
||||
) where
|
||||
|
||||
import Aenebris.WAF.Rule
|
||||
( Action(..)
|
||||
, Operator(..)
|
||||
, ParanoiaLevel(..)
|
||||
, Phase(..)
|
||||
, Rule(..)
|
||||
, RuleSet(..)
|
||||
, Severity(..)
|
||||
, Target(..)
|
||||
, compileRegex
|
||||
, defaultInboundThreshold
|
||||
, defaultOutboundThreshold
|
||||
)
|
||||
import Data.ByteString (ByteString)
|
||||
import qualified Data.ByteString.Char8 as BC
|
||||
|
||||
mkRegex :: ByteString -> Operator
|
||||
mkRegex pat = case compileRegex pat of
|
||||
Right r -> OpRegex r
|
||||
Left _ -> OpStreq pat
|
||||
|
||||
sqliPatterns :: [ByteString]
|
||||
sqliPatterns =
|
||||
[ "(union[[:space:]]+(all[[:space:]]+)?select)"
|
||||
, "((select|insert|update|delete|drop|alter|create|truncate)[[:space:]]+.*[[:space:]]+(from|into|table|database))"
|
||||
, "(or[[:space:]]+1[[:space:]]*=[[:space:]]*1)"
|
||||
, "(and[[:space:]]+1[[:space:]]*=[[:space:]]*1)"
|
||||
, "(';|\";|--[[:space:]]|/\\*|\\*/)"
|
||||
, "((sleep|benchmark|pg_sleep|waitfor[[:space:]]+delay)[[:space:]]*\\()"
|
||||
, "(load_file[[:space:]]*\\(|into[[:space:]]+(out|dump)file)"
|
||||
, "(information_schema|sys\\.tables|pg_catalog|sqlite_master)"
|
||||
]
|
||||
|
||||
xssPatterns :: [ByteString]
|
||||
xssPatterns =
|
||||
[ "<[[:space:]]*script[[:space:]>]"
|
||||
, "javascript[[:space:]]*:"
|
||||
, "on(load|error|click|mouseover|focus|blur|submit|change|input)[[:space:]]*="
|
||||
, "<[[:space:]]*(iframe|embed|object|svg|math|details|marquee)[[:space:]>]"
|
||||
, "(eval|expression|alert|prompt|confirm|fromcharcode)[[:space:]]*\\("
|
||||
, "data[[:space:]]*:[[:space:]]*text/html"
|
||||
, "vbscript[[:space:]]*:"
|
||||
]
|
||||
|
||||
traversalPatterns :: [ByteString]
|
||||
traversalPatterns =
|
||||
[ "(\\.\\./|\\.\\.\\\\)"
|
||||
, "(%2e%2e[/\\\\%]|%252e%252e)"
|
||||
, "(/etc/(passwd|shadow|hosts|group)|c:\\\\windows\\\\system32)"
|
||||
, "(/proc/self/(environ|cmdline|maps)|/dev/(tcp|udp))"
|
||||
, "(\\.\\./){2,}"
|
||||
]
|
||||
|
||||
crlfPatterns :: [ByteString]
|
||||
crlfPatterns =
|
||||
[ "(%0d%0a|%0a%0d|%0a|%0d|\\r\\n|\\n\\r)"
|
||||
, "(set-cookie:|content-length:|location:|content-type:)"
|
||||
]
|
||||
|
||||
ruleNamePrefix :: ByteString -> Int -> ByteString
|
||||
ruleNamePrefix prefix n = prefix <> BC.pack (show n)
|
||||
|
||||
mkRules :: ParanoiaLevel
|
||||
-> Severity
|
||||
-> Phase
|
||||
-> [Target]
|
||||
-> ByteString
|
||||
-> [ByteString]
|
||||
-> Int
|
||||
-> [Rule]
|
||||
mkRules pl sev ph targets prefix patterns startId =
|
||||
zipWith3
|
||||
(\i name pat -> Rule
|
||||
{ ruleId = fromIntegral i
|
||||
, ruleName = name
|
||||
, rulePhase = ph
|
||||
, ruleOp = mkRegex pat
|
||||
, ruleTargets = targets
|
||||
, ruleSeverity = sev
|
||||
, ruleAction = Score
|
||||
, ruleParanoia = pl
|
||||
})
|
||||
[startId ..]
|
||||
[ruleNamePrefix prefix n | n <- [1 .. length patterns]]
|
||||
patterns
|
||||
|
||||
sqliRules :: [Rule]
|
||||
sqliRules = mkRules PL1 SevCritical PhaseHeaders
|
||||
[TargetPath, TargetQuery]
|
||||
"sqli-" sqliPatterns 1000
|
||||
|
||||
xssRules :: [Rule]
|
||||
xssRules = mkRules PL1 SevCritical PhaseHeaders
|
||||
[TargetPath, TargetQuery]
|
||||
"xss-" xssPatterns 2000
|
||||
|
||||
traversalRules :: [Rule]
|
||||
traversalRules = mkRules PL1 SevError PhaseHeaders
|
||||
[TargetPath, TargetQuery]
|
||||
"traversal-" traversalPatterns 3000
|
||||
|
||||
crlfRules :: [Rule]
|
||||
crlfRules = mkRules PL2 SevError PhaseHeaders
|
||||
[TargetPath, TargetQuery]
|
||||
"crlf-" crlfPatterns 4000
|
||||
|
||||
protocolRules :: [Rule]
|
||||
protocolRules =
|
||||
[ Rule
|
||||
{ ruleId = 9000
|
||||
, ruleName = "ambiguous-framing-cl-te"
|
||||
, rulePhase = PhaseHeaders
|
||||
, ruleOp = OpStreq "__synthetic__"
|
||||
, ruleTargets = []
|
||||
, ruleSeverity = SevCritical
|
||||
, ruleAction = Block
|
||||
, ruleParanoia = PL1
|
||||
}
|
||||
, Rule
|
||||
{ ruleId = 9001
|
||||
, ruleName = "obsolete-line-folding"
|
||||
, rulePhase = PhaseHeaders
|
||||
, ruleOp = OpStreq "__synthetic__"
|
||||
, ruleTargets = []
|
||||
, ruleSeverity = SevCritical
|
||||
, ruleAction = Block
|
||||
, ruleParanoia = PL1
|
||||
}
|
||||
, Rule
|
||||
{ ruleId = 9002
|
||||
, ruleName = "duplicate-host-header"
|
||||
, rulePhase = PhaseHeaders
|
||||
, ruleOp = OpStreq "__synthetic__"
|
||||
, ruleTargets = []
|
||||
, ruleSeverity = SevError
|
||||
, ruleAction = Block
|
||||
, ruleParanoia = PL1
|
||||
}
|
||||
]
|
||||
|
||||
defaultRules :: [Rule]
|
||||
defaultRules =
|
||||
protocolRules <> sqliRules <> xssRules <> traversalRules <> crlfRules
|
||||
|
||||
defaultRuleSet :: RuleSet
|
||||
defaultRuleSet = RuleSet
|
||||
{ rsRules = defaultRules
|
||||
, rsParanoia = PL2
|
||||
, rsInboundThreshold = defaultInboundThreshold
|
||||
, rsOutboundThreshold = defaultOutboundThreshold
|
||||
}
|
||||
|
|
@ -0,0 +1,127 @@
|
|||
{-
|
||||
©AngelaMos | 2026
|
||||
Rule.hs
|
||||
-}
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module Aenebris.WAF.Rule
|
||||
( Rule(..)
|
||||
, Operator(..)
|
||||
, Action(..)
|
||||
, Severity(..)
|
||||
, Phase(..)
|
||||
, Target(..)
|
||||
, ParanoiaLevel(..)
|
||||
, RuleSet(..)
|
||||
, severityScore
|
||||
, defaultInboundThreshold
|
||||
, defaultOutboundThreshold
|
||||
, compileRegex
|
||||
, CompiledRegex
|
||||
, runRegex
|
||||
) where
|
||||
|
||||
import Data.ByteString (ByteString)
|
||||
import Data.Word (Word32)
|
||||
import Text.Regex.TDFA (Regex)
|
||||
import Text.Regex.TDFA.ByteString (compile, execute)
|
||||
import qualified Text.Regex.TDFA as TDFA
|
||||
|
||||
data Phase
|
||||
= PhaseHeaders
|
||||
| PhaseRequestBody
|
||||
| PhaseResponseHeaders
|
||||
| PhaseResponseBody
|
||||
deriving (Eq, Show, Ord)
|
||||
|
||||
data Severity
|
||||
= SevNotice
|
||||
| SevWarning
|
||||
| SevError
|
||||
| SevCritical
|
||||
deriving (Eq, Show, Ord)
|
||||
|
||||
severityScore :: Severity -> Int
|
||||
severityScore SevNotice = 2
|
||||
severityScore SevWarning = 3
|
||||
severityScore SevError = 4
|
||||
severityScore SevCritical = 5
|
||||
|
||||
data Action
|
||||
= Block
|
||||
| Score
|
||||
| Log
|
||||
| Pass
|
||||
deriving (Eq, Show)
|
||||
|
||||
data ParanoiaLevel
|
||||
= PL1
|
||||
| PL2
|
||||
| PL3
|
||||
| PL4
|
||||
deriving (Eq, Show, Ord, Enum, Bounded)
|
||||
|
||||
data Target
|
||||
= TargetMethod
|
||||
| TargetPath
|
||||
| TargetQuery
|
||||
| TargetHeaderValue !ByteString
|
||||
| TargetAnyHeaderName
|
||||
| TargetAnyHeaderValue
|
||||
| TargetHost
|
||||
| TargetUserAgent
|
||||
deriving (Eq, Show)
|
||||
|
||||
newtype CompiledRegex = CompiledRegex { unCompiledRegex :: Regex }
|
||||
|
||||
instance Show CompiledRegex where
|
||||
show _ = "<CompiledRegex>"
|
||||
|
||||
instance Eq CompiledRegex where
|
||||
_ == _ = False
|
||||
|
||||
data Operator
|
||||
= OpRegex !CompiledRegex
|
||||
| OpStreq !ByteString
|
||||
| OpContains !ByteString
|
||||
| OpAnyMatch ![ByteString]
|
||||
deriving (Eq, Show)
|
||||
|
||||
data Rule = Rule
|
||||
{ ruleId :: !Word32
|
||||
, ruleName :: !ByteString
|
||||
, rulePhase :: !Phase
|
||||
, ruleOp :: !Operator
|
||||
, ruleTargets :: ![Target]
|
||||
, ruleSeverity :: !Severity
|
||||
, ruleAction :: !Action
|
||||
, ruleParanoia :: !ParanoiaLevel
|
||||
} deriving (Show)
|
||||
|
||||
data RuleSet = RuleSet
|
||||
{ rsRules :: ![Rule]
|
||||
, rsParanoia :: !ParanoiaLevel
|
||||
, rsInboundThreshold :: !Int
|
||||
, rsOutboundThreshold :: !Int
|
||||
} deriving (Show)
|
||||
|
||||
defaultInboundThreshold :: Int
|
||||
defaultInboundThreshold = 5
|
||||
|
||||
defaultOutboundThreshold :: Int
|
||||
defaultOutboundThreshold = 4
|
||||
|
||||
compileRegex :: ByteString -> Either String CompiledRegex
|
||||
compileRegex pat =
|
||||
case compile compOpts execOpts pat of
|
||||
Left err -> Left err
|
||||
Right r -> Right (CompiledRegex r)
|
||||
where
|
||||
compOpts = TDFA.defaultCompOpt { TDFA.caseSensitive = False }
|
||||
execOpts = TDFA.defaultExecOpt
|
||||
|
||||
runRegex :: CompiledRegex -> ByteString -> Bool
|
||||
runRegex (CompiledRegex r) input =
|
||||
case execute r input of
|
||||
Right (Just _) -> True
|
||||
_ -> False
|
||||
File diff suppressed because it is too large
Load Diff
BIN
PROJECTS/advanced/haskell-reverse-proxy/test/fixtures/geo/GeoLite2-ASN-Test.mmdb
vendored
Normal file
BIN
PROJECTS/advanced/haskell-reverse-proxy/test/fixtures/geo/GeoLite2-ASN-Test.mmdb
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
PROJECTS/advanced/haskell-reverse-proxy/test/fixtures/geo/GeoLite2-Country-Test.mmdb
vendored
Normal file
BIN
PROJECTS/advanced/haskell-reverse-proxy/test/fixtures/geo/GeoLite2-Country-Test.mmdb
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
|
|
@ -0,0 +1,49 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
██████╗ ██████╗ ██╗ ██╗████████╗ ██████╗ ██████╗ ██╗
|
||||
██╔══██╗██╔════╝ ██║ ██║╚══██╔══╝██╔═══██╗██╔═══██╗██║
|
||||
██████╔╝██║ ███╗███████║ ██║ ██║ ██║██║ ██║██║
|
||||
██╔══██╗██║ ██║╚════██║ ██║ ██║ ██║██║ ██║██║
|
||||
██████╔╝╚██████╔╝ ██║ ██║ ╚██████╔╝╚██████╔╝███████╗
|
||||
╚═════╝ ╚═════╝ ╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚══════╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://pypi.org/project/b64tool/">
|
||||
<img src="https://img.shields.io/badge/PyPI-b64tool-3775A9?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
uv tool install b64tool
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Encoding](#encoding) · [Decoding, Detection & Layer Peeling](#decoding-detection--layer-peeling)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Encoding
|
||||
|
||||
Base64, Hex, Base32, URL encoding with piped input support
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Decoding, Detection & Layer Peeling
|
||||
|
||||
File encoding, multi-format decoding, auto-detection with confidence scoring, recursive layer stripping, and multi-step encoding chains
|
||||
|
||||
|
||||
|
|
@ -16,6 +16,8 @@
|
|||
|
||||
*This is a quick overview — security theory, architecture, and full walkthroughs are in the [learn modules](#learn).*
|
||||
|
||||
**[Screenshots & demo →](DEMO.md)**
|
||||
|
||||
## What It Does
|
||||
|
||||
- Encode and decode Base64, Base64URL, Base32, Hex, and URL formats
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 101 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
|
|
@ -0,0 +1,37 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
██████╗ █████╗ ███████╗███████╗ █████╗ ██████╗
|
||||
██╔════╝██╔══██╗██╔════╝██╔════╝██╔══██╗██╔══██╗
|
||||
██║ ███████║█████╗ ███████╗███████║██████╔╝
|
||||
██║ ██╔══██║██╔══╝ ╚════██║██╔══██║██╔══██╗
|
||||
╚██████╗██║ ██║███████╗███████║██║ ██║██║ ██║
|
||||
╚═════╝╚═╝ ╚═╝╚══════╝╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://pypi.org/project/caesar-salad-cipher/">
|
||||
<img src="https://img.shields.io/badge/PyPI-caesar--salad--cipher-3775A9?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
uv tool install caesar-salad-cipher
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Encrypt, Decrypt & Crack
|
||||
|
||||
Shift-key encryption and decryption with brute-force cracking across all 26 shifts, ranked by frequency analysis scoring
|
||||
|
||||
|
||||
|
|
@ -16,6 +16,8 @@
|
|||
|
||||
*This is a quick overview — security theory, architecture, and full walkthroughs are in the [learn modules](#learn).*
|
||||
|
||||
**[Screenshots & demo →](DEMO.md)**
|
||||
|
||||
## What It Does
|
||||
|
||||
- Encrypt text using Caesar cipher with a specified shift key
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 89 KiB |
|
|
@ -0,0 +1,49 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
██████╗ ███╗ ██╗███████╗██╗ ██████╗ ██████╗ ██╗ ██╗██╗ ██╗██████╗
|
||||
██╔══██╗████╗ ██║██╔════╝██║ ██╔═══██╗██╔═══██╗██║ ██╔╝██║ ██║██╔══██╗
|
||||
██║ ██║██╔██╗ ██║███████╗██║ ██║ ██║██║ ██║█████╔╝ ██║ ██║██████╔╝
|
||||
██║ ██║██║╚██╗██║╚════██║██║ ██║ ██║██║ ██║██╔═██╗ ██║ ██║██╔═══╝
|
||||
██████╔╝██║ ╚████║███████║███████╗╚██████╔╝╚██████╔╝██║ ██╗╚██████╔╝██║
|
||||
╚═════╝ ╚═╝ ╚═══╝╚══════╝╚══════╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://pypi.org/project/dnslookup-cli/">
|
||||
<img src="https://img.shields.io/badge/PyPI-dnslookup--cli-3775A9?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
uv tool install dnslookup-cli
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Table Output](#table-output) · [JSON Export](#json-export)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Table Output
|
||||
|
||||
Full DNS record query with colored table, record type filtering, and response time tracking
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### JSON Export
|
||||
|
||||
Machine-readable JSON output for scripting and pipeline integration
|
||||
|
||||
|
||||
|
|
@ -16,6 +16,8 @@
|
|||
|
||||
*This is a quick overview — security theory, architecture, and full walkthroughs are in the [learn modules](#learn).*
|
||||
|
||||
**[Screenshots & demo →](DEMO.md)**
|
||||
|
||||
## What It Does
|
||||
|
||||
- Query A, AAAA, MX, NS, TXT, CNAME, and SOA records with colored table output
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 137 KiB |
|
|
@ -0,0 +1,49 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
██╗ ██╗ █████╗ ███████╗██╗ ██╗ ██████╗██████╗ █████╗ ██████╗██╗ ██╗███████╗██████╗
|
||||
██║ ██║██╔══██╗██╔════╝██║ ██║██╔════╝██╔══██╗██╔══██╗██╔════╝██║ ██╔╝██╔════╝██╔══██╗
|
||||
███████║███████║███████╗███████║██║ ██████╔╝███████║██║ █████╔╝ █████╗ ██████╔╝
|
||||
██╔══██║██╔══██║╚════██║██╔══██║██║ ██╔══██╗██╔══██║██║ ██╔═██╗ ██╔══╝ ██╔══██╗
|
||||
██║ ██║██║ ██║███████║██║ ██║╚██████╗██║ ██║██║ ██║╚██████╗██║ ██╗███████╗██║ ██║
|
||||
╚═╝ ╚═╝╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝ ╚═════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═════╝╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/beginner/hash-cracker">
|
||||
<img src="https://img.shields.io/badge/C++23-Multi--threaded-00599C?style=for-the-badge&logo=cplusplus&logoColor=white" alt="C++23"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
./install.sh → hashcracker --hash <hash> --wordlist <list>
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Dictionary Attack](#dictionary-attack) · [Rule-Based Mutations](#rule-based-mutations)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Dictionary Attack
|
||||
|
||||
Memory-mapped wordlist scan with auto-detected hash type, work-partitioned across all cores, with live progress bar and h/s throughput
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Rule-Based Mutations
|
||||
|
||||
Mutation rules expand a 10K wordlist into 20.1M candidates with capitalize, leet, digit-append, reverse, and toggle-case transforms applied per word
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 225 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 269 KiB |
|
|
@ -0,0 +1,49 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
██████╗██╗███████╗ █████╗ ██╗ ██╗██████╗ ██╗████████╗
|
||||
██╔════╝██║██╔════╝██╔══██╗██║ ██║██╔══██╗██║╚══██╔══╝
|
||||
██║ ██║███████╗███████║██║ ██║██║ ██║██║ ██║
|
||||
██║ ██║╚════██║██╔══██║██║ ██║██║ ██║██║ ██║
|
||||
╚██████╗██║███████║██║ ██║╚██████╔╝██████╔╝██║ ██║
|
||||
╚═════╝╚═╝╚══════╝╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/beginner/linux-cis-hardening-auditor">
|
||||
<img src="https://img.shields.io/badge/CIS_Debian_12-Benchmark_v1.1.0-4EAA25?style=for-the-badge&logo=gnubash&logoColor=white" alt="CIS Benchmark"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
./install.sh → sudo cisaudit
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Audit Report](#audit-report) · [Control Catalog](#control-catalog)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Audit Report
|
||||
|
||||
Section-scored compliance report with per-control evidence, severity, and remediation commands rendered against a fixture root requiring no privileges
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Control Catalog
|
||||
|
||||
109 registered controls across six CIS categories with section, level, and scored/non-scored classification
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.3 MiB |
|
|
@ -0,0 +1,37 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
███╗ ███╗███████╗████████╗
|
||||
████╗ ████║██╔════╝╚══██╔══╝
|
||||
██╔████╔██║███████╗ ██║
|
||||
██║╚██╔╝██║╚════██║ ██║
|
||||
██║ ╚═╝ ██║███████║ ██║
|
||||
╚═╝ ╚═╝╚══════╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://pypi.org/project/metadata-scrubber/">
|
||||
<img src="https://img.shields.io/badge/PyPI-metadata--scrubber-3775A9?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
uv tool install metadata-scrubber
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Read, Scrub & Verify
|
||||
|
||||
Metadata inspection with categorized report (device info, image data, GPS, timestamps), batch scrubbing with progress bar, and post-scrub verification
|
||||
|
||||
|
||||
|
|
@ -15,6 +15,8 @@
|
|||
> Privacy-focused CLI that strips sensitive metadata from images, PDFs, and Office documents.
|
||||
|
||||
*This is a quick overview — security theory, architecture, and full walkthroughs are in the [learn modules](#learn).*
|
||||
|
||||
**[Screenshots & demo →](DEMO.md)**
|
||||
> *Developed by [@Heritage-XioN](https://github.com/Heritage-XioN)*
|
||||
## What It Does
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
|
|
@ -0,0 +1,37 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
███╗ ██╗███████╗████████╗ █████╗ ███╗ ██╗ █████╗ ██╗
|
||||
████╗ ██║██╔════╝╚══██╔══╝██╔══██╗████╗ ██║██╔══██╗██║
|
||||
██╔██╗ ██║█████╗ ██║ ███████║██╔██╗ ██║███████║██║
|
||||
██║╚██╗██║██╔══╝ ██║ ██╔══██║██║╚██╗██║██╔══██║██║
|
||||
██║ ╚████║███████╗ ██║ ██║ ██║██║ ╚████║██║ ██║███████╗
|
||||
╚═╝ ╚═══╝╚══════╝ ╚═╝ ╚═╝ ╚═╝╚═╝ ╚═══╝╚═╝ ╚═╝╚══════╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
cd python && uv sync
|
||||
sudo netanal capture -i eth0
|
||||
```
|
||||
|
||||
```ruby
|
||||
cd cpp && ./install.sh
|
||||
just run -i eth0
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Live Capture & Analysis
|
||||
|
||||
Real-time packet capture with per-packet logging, capture summary stats, protocol distribution breakdown, and top talker ranking by bytes
|
||||
|
||||
|
||||
|
|
@ -2,6 +2,8 @@
|
|||
|
||||
Two implementations of the same network traffic analyzer — one in Python, one in C++. Both capture packets at the kernel level, parse protocol headers, and display real-time statistics.
|
||||
|
||||
**[Screenshots & demo →](DEMO.md)**
|
||||
|
||||
## Implementations
|
||||
|
||||
| Implementation | Stack | Highlights |
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 169 KiB |
|
|
@ -0,0 +1,18 @@
|
|||
# ©AngelaMos | 2026
|
||||
# .gitignore
|
||||
|
||||
build/
|
||||
*.o
|
||||
*.a
|
||||
*.so
|
||||
*.out
|
||||
CMakeCache.txt
|
||||
CMakeFiles/
|
||||
cmake_install.cmake
|
||||
compile_commands.json
|
||||
.cache/
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
.DS_Store
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
██████╗ ██████╗ ██████╗ ████████╗ ███████╗ ██████╗ █████╗ ███╗ ██╗███╗ ██╗███████╗██████╗
|
||||
██╔══██╗██╔═══██╗██╔══██╗╚══██╔══╝ ██╔════╝██╔════╝██╔══██╗████╗ ██║████╗ ██║██╔════╝██╔══██╗
|
||||
██████╔╝██║ ██║██████╔╝ ██║ ███████╗██║ ███████║██╔██╗ ██║██╔██╗ ██║█████╗ ██████╔╝
|
||||
██╔═══╝ ██║ ██║██╔══██╗ ██║ ╚════██║██║ ██╔══██║██║╚██╗██║██║╚██╗██║██╔══╝ ██╔══██╗
|
||||
██║ ╚██████╔╝██║ ██║ ██║ ███████║╚██████╗██║ ██║██║ ╚████║██║ ╚████║███████╗██║ ██║
|
||||
╚═╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚══════╝ ╚═════╝╚═╝ ╚═╝╚═╝ ╚═══╝╚═╝ ╚═══╝╚══════╝╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/beginner/simple-port-scanner">
|
||||
<img src="https://img.shields.io/badge/C++20-Boost.Asio-00599C?style=for-the-badge&logo=cplusplus&logoColor=white" alt="C++ Boost.Asio"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
mkdir build && cd build && cmake .. && make
|
||||
./simplePortScanner -i <target> -p <range>
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[SSH Discovery](#ssh-discovery) · [HTTP Discovery](#http-discovery)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### SSH Discovery
|
||||
|
||||
Async TCP scan against scanme.nmap.org with verbose service mapping showing OPEN/CLOSED/FILTERED states across the SSH well-known port range
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### HTTP Discovery
|
||||
|
||||
Concurrent scan across the HTTP port window with per-port service identification and aggregate result counts
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 148 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 152 KiB |
|
|
@ -0,0 +1,49 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
█████╗ ███╗ ██╗ ██████╗ ███████╗██╗ █████╗
|
||||
██╔══██╗████╗ ██║██╔════╝ ██╔════╝██║ ██╔══██╗
|
||||
███████║██╔██╗ ██║██║ ███╗█████╗ ██║ ███████║
|
||||
██╔══██║██║╚██╗██║██║ ██║██╔══╝ ██║ ██╔══██║
|
||||
██║ ██║██║ ╚████║╚██████╔╝███████╗███████╗██║ ██║
|
||||
╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═════╝ ╚══════╝╚══════╝╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://pkg.go.dev/github.com/CarterPerez-dev/angela">
|
||||
<img src="https://img.shields.io/badge/Go_Install-angela-00ADD8?style=for-the-badge&logo=go&logoColor=white" alt="Go install"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
go install github.com/CarterPerez-dev/angela/cmd/angela@latest
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Update Check](#update-check) · [Vulnerability Scan](#vulnerability-scan)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Update Check
|
||||
|
||||
Parallel PyPI queries with PEP 440 version parsing, surfacing major and minor upgrades across pyproject.toml dependencies
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Vulnerability Scan
|
||||
|
||||
OSV.dev integration grouping CVEs by package and severity with first-fixed version per advisory
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 189 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.0 MiB |
|
|
@ -0,0 +1,49 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
███████╗███████╗███╗ ██╗████████╗██╗███╗ ██╗███████╗██╗
|
||||
██╔════╝██╔════╝████╗ ██║╚══██╔══╝██║████╗ ██║██╔════╝██║
|
||||
███████╗█████╗ ██╔██╗ ██║ ██║ ██║██╔██╗ ██║█████╗ ██║
|
||||
╚════██║██╔══╝ ██║╚██╗██║ ██║ ██║██║╚██╗██║██╔══╝ ██║
|
||||
███████║███████╗██║ ╚████║ ██║ ██║██║ ╚████║███████╗███████╗
|
||||
╚══════╝╚══════╝╚═╝ ╚═══╝ ╚═╝ ╚═╝╚═╝ ╚═══╝╚══════╝╚══════╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://pkg.go.dev/github.com/CarterPerez-dev/sentinel">
|
||||
<img src="https://img.shields.io/badge/Go_Install-sentinel-00ADD8?style=for-the-badge&logo=go&logoColor=white" alt="Go install"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
go install github.com/CarterPerez-dev/sentinel/cmd/sentinel@latest
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Persistence Scan](#persistence-scan) · [JSON Output](#json-output)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Persistence Scan
|
||||
|
||||
17-module sweep across systemd, cron, shell profiles, ld.so.preload, and PAM with severity scoring and MITRE ATT&CK technique mapping per finding
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### JSON Output
|
||||
|
||||
Structured findings for pipeline integration with scanner attribution, severity codes, evidence strings, and aggregate severity counts
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 506 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 901 KiB |
|
|
@ -0,0 +1,81 @@
|
|||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- DEMO.md -->
|
||||
|
||||
<div align="center">
|
||||
|
||||
```ruby
|
||||
███████╗██╗███████╗███╗ ███╗
|
||||
██╔════╝██║██╔════╝████╗ ████║
|
||||
███████╗██║█████╗ ██╔████╔██║
|
||||
╚════██║██║██╔══╝ ██║╚██╔╝██║
|
||||
███████║██║███████╗██║ ╚═╝ ██║
|
||||
╚══════╝╚═╝╚══════╝╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Demo & Preview**
|
||||
|
||||
<br>
|
||||
|
||||
<a href="https://siem.carterperez-dev.com">
|
||||
<img src="https://img.shields.io/badge/▶_TRY_IT_LIVE-siem.carterperez--dev.com-DC143C?style=for-the-badge&logo=googlechrome&logoColor=white" alt="Live Demo"/>
|
||||
</a>
|
||||
|
||||
<br>
|
||||
|
||||
```ruby
|
||||
docker compose up -d → localhost:8431
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
[Dashboard](#dashboard) · [Log Viewer](#log-viewer) · [Alert Investigation](#alert-investigation) · [Correlation Rules](#correlation-rules) · [Attack Scenarios](#attack-scenarios) · [User Management](#user-management)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
### Dashboard
|
||||
|
||||
Real-time event counters, severity breakdown by donut chart, event timeline, and top source IP ranking
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Log Viewer
|
||||
|
||||
Paginated event stream filterable by source, severity, and event type with timestamp correlation
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Alert Investigation
|
||||
|
||||
Correlated alert drill-down with matched event timeline and lifecycle controls — Acknowledge, Investigate, Resolve, False Positive
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Correlation Rules
|
||||
|
||||
Threshold, Sequence, and Aggregation rule types with inline JSON condition editing and severity classification
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Attack Scenarios
|
||||
|
||||
Four MITRE ATT&CK mapped playbooks generating realistic multi-stage events — brute force with lateral movement, DNS tunneling exfiltration, phishing to C2 beaconing, privilege escalation with persistence
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### User Management
|
||||
|
||||
Role-based access control with Admin and Analyst roles, account provisioning, and status management
|
||||
|
||||
|
||||
|
|
@ -18,6 +18,8 @@
|
|||
|
||||
*This is a quick overview — security theory, architecture, and full walkthroughs are in the [learn modules](#learn).*
|
||||
|
||||
**[Screenshots & live demo →](DEMO.md)**
|
||||
|
||||
## What It Does
|
||||
|
||||
- Real-time log ingestion and event correlation with three rule types (Threshold, Sequence, Aggregation)
|
||||
|
|
|
|||
|
Before Width: | Height: | Size: 85 KiB After Width: | Height: | Size: 85 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 132 KiB After Width: | Height: | Size: 59 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Loading…
Reference in New Issue