Tier 1 (cre demo): in-memory SQLite + tempfile .env rotator. Walks
through inventory -> before -> 4-step rotation -> after -> audit chain
verify. Narrates every step event live with green/red glyphs. Runs in
under 1 second on commodity hardware, no external deps.
Tier 2 (docker/docker-compose.yml): postgres:16, localstack
(secretsmanager), hashicorp/vault dev mode, plus a fake-GitHub Flask
app that mocks /user, POST/DELETE /user/personal-access-tokens.
config/demo-full.cr.example documents env vars for the full stack.
Also fixed env_file rotator newline preservation (lines(chomp: true)
+ explicit \n join) so the rotated file is properly delimited.
Stack:
- Ansi: stdlib-only escape helpers (move, colorize, strip)
- State: rolling view of active rotations + recent events; trims to N
- Renderer: paints header / status / active / recent panels; works against
any IO so tests assert against IO::Memory
- Tui: ties State + Renderer to event bus; coalesces repaints to a tick
interval (default 200ms) to avoid flicker under burst events
Active rotation rows show step progress as a filled-bar (▰▰▰▱). Recent
events list latest 8 with timestamp + severity glyph (✓ / ⚠ / !). The
renderer's pad/center helpers correctly account for ANSI escape widths
when computing visible column width.
9 specs cover ANSI helpers (color codes, cursor move, strip), State
event-to-row mapping with completion bookkeeping and ring-buffer trim,
and Renderer output assertions across empty + active + populated states.
LogNotifier subscribes (Drop overflow) and emits stdlib Log lines with
structured kwargs (credential_id, rotation_id, severity, etc.) suitable
for downstream ingestion by journald/vector/fluentd.
Telegram is a thin HTTP::Client wrapper for sendMessage and getUpdates.
TelegramSubscriber sends emoji-prefixed alerts on RotationFailed,
DriftDetected, PolicyViolation, AlertRaised; success notifications gated
by the notify_on_success flag. Errors are swallowed so a flaky bot never
blocks the engine.
TelegramBot does long-polling getUpdates and dispatches commands:
- viewer tier: /status /queue /history /alerts /help
- operator tier: viewer + /rotate + /snooze
ACL is by chat_id allowlist (bot token + chat IDs in env vars).
/rotate publishes RotationScheduled to the bus.
11 unit specs cover send_message + error handling + getUpdates parsing,
subscriber dispatch, success-suppression flag, and bot ACL enforcement
(unauthorized / viewer-blocked-from-mutations / operator-can-rotate).
Initial tick fires immediately on start so first policy evaluation
doesn't wait for the full interval on boot. The PolicyEvaluator
subscribes to SchedulerTick events and invokes evaluate_all -> overdue
discovery -> RotationScheduled fan-out. 3 specs verify boot-tick,
periodic cadence, and clean stop().
Github::Client wraps the bearer-auth REST API with required
'X-GitHub-Api-Version: 2022-11-28' header. Implements me, create_pat
(POST /user/personal-access-tokens), delete_pat (DELETE by id).
GithubPatRotator's 4-step contract:
- generate: create_pat returns new id + token value (fresh PAT alongside old)
- apply: no-op (create_pat already active)
- verify: probe-client uses the NEW PAT to GET /user; success means it works
- commit: delete OLD PAT by id (tracked via old_pat_id tag)
- rollback_apply: delete NEW PAT
7 unit specs verify create/delete/me round-trips, full rotation path,
verify-on-401, and rollback deletion.
- New Aenebris.ML.Middleware module: Wai middleware that wires the
Aenebris.ML.Engine into the per-request pipeline. Per request:
1. Run extractFeatures (FeatureContext + Request) -> FeatureVector
2. Convert via featureVectorToVector -> VU.Vector Double
3. runEngine eng fv -> DecisionDetails
4. Route by ddDecision:
DecisionHuman -> pass through to inner Application,
optionally attach X-Aenebris-ML-Decision
and X-Aenebris-ML-Score response headers
DecisionBot -> 403 + text/plain block body
DecisionChallenge -> 403 + text/html challenge page
All response builders are operator-overridable via the
MLMiddlewareConfig record. Optional logging callback fires once
per request for structured logs / metrics integration.
- defaultBotResponse, defaultChallengeResponse: sensible defaults
matching the existing WAF middleware pattern (403 + descriptive
body). decisionResponseHeader / scoreResponseHeader are exported
so operators can build custom responses that still surface the
signal headers.
- 12 tests via Network.Wai.Test.runSession covering: pass-through
on Human, 403 on Bot, 403 on Challenge with text/html body,
signal-header attachment toggle, custom response builder override,
logging callback invocation count.
- aenebris.cabal: expose Aenebris.ML.Middleware in the library stanza.
- test/Spec.hs: add modifyTVar' to the Control.Concurrent.STM import
list for the logging-callback test.
- 354 total examples passing, 0 GHC warnings on the new module.
This closes the core Phase 2.5 ML pipeline. The full sequence
(Features -> Model -> Loader -> Inference -> Calibration -> IForest
-> Engine -> Middleware) is now fully implemented and integration-
tested. A hosted Aenebris instance can now be configured with a
trained LightGBM model + optional calibrator + optional IForest
and will block bots, challenge ambiguous traffic, and pass humans
on every HTTP request.
Rotator base exposes the four lifecycle methods as abstract; subclasses
register at compile time via 'register_as :kind' macro. REGISTRY is
populated as soon as the rotator file is required - drop a new file in
src/cre/rotators/ and the orchestrator can dispatch to it.
EnvFileRotator implements the simplest rotation:
- generate: 32 random bytes -> base64-urlsafe (no padding)
- apply: write to PATH.pending atomically (in-place line replace, 0600)
- verify: parse pending file, confirm key=value present and non-empty
- commit: rename PATH.pending -> PATH (atomic on POSIX)
- rollback_apply: unlink PATH.pending
RotationOrchestrator runs the 4-step contract with full event publishing,
state machine transitions in DB (generating -> applying -> verifying ->
committing -> completed | failed), and rollback_apply on apply/verify
failure. 8 specs cover happy path + raised-during-apply + rollback verification.
- Policy struct: matcher Proc, max_age, warn_at, enforce_action, notify_channels, triggers
- Builder validates all required fields at .build, raises BuilderError with the policy name
- DSL exposes top-level 'policy' method via 'with builder yield' so all
builder methods (description, match, max_age, enforce, notify_via, etc.)
are receiver-less inside the block
- Symbol literals autocast to Action/Channel/Trigger enums on direct calls;
Symbol overload for splat parameters (notify_via :telegram, :email)
- Evaluator subscribes to bus, fires PolicyViolation + RotationScheduled
(rotate_immediately) or AlertRaised (notify_only/quarantine) when overdue
- 15 unit specs cover Policy.matches?/overdue?/in_warning_window?,
Builder validation, DSL syntax with closures, evaluator action dispatch
- New Aenebris.ML.Engine module: pure decision pipeline composing
the previously-built Loader, Inference, Calibration, and IForest
modules into a single per-request scoring path.
Engine record holds (Ensemble, Calibrator, Maybe IForest,
EngineConfig); runEngine takes a feature vector and produces
DecisionDetails (Decision, raw proba, calibrated proba, optional
IForest score). The Decision is one of DecisionHuman, DecisionBot,
or DecisionChallenge.
- Implements the escalation-gate semantics per the 2026 research
correction (over the deprecated 0.8/0.2 weighted blend):
calibrated <= humanThreshold -> DecisionHuman
calibrated >= botThreshold -> DecisionBot
otherwise (ambiguous band):
if IForest configured and ifScore >= escalation threshold
-> DecisionBot (escalation)
else if challenges enabled
-> DecisionChallenge
else
-> DecisionHuman (fallthrough)
Defaults: 0.3 / 0.7 thresholds, 0.6 IForest escalation, challenges
on by default.
- 17 tests covering: defaultEngineConfig values, decision boundaries
with NoCalibrator and no IForest (low/mid/high leaf ensembles),
ambiguous-band escalation via low-vs-high anomaly IForests,
ambiguous-band with challenges disabled (fallthrough to Human or
escalation to Bot), Platt calibrator pulling decisions across
thresholds, and DecisionDetails field correctness.
- aenebris.cabal: expose Aenebris.ML.Engine in the library stanza.
- 342 total examples passing, 0 GHC warnings on the new module.
The Wai middleware wiring (extract features -> runEngine -> route by
Decision) is the next module (ML.Middleware); Engine intentionally
stays pure / independent of HTTP machinery so it can be unit-tested
without request fixtures.
Events form a Crystal class hierarchy so subscribers pattern-match
exhaustively. EventBus delivers to subscribers via Crystal channels with
per-subscriber overflow policy (Block for audit, Drop for best-effort).
AuditSubscriber listens for all rotation/policy/drift/alert events and
serializes them into the hash-chained audit log via AuditLog.append.
Engine wires Persistence + AuditLog + AuditSubscriber + EventBus together
so callers just .start/.stop. 7 unit specs verify boot, fanout to N
subscribers, drop-on-overflow semantics, and end-to-end event -> audit row.
Three layers of integrity:
- Hash chain (SHA-256 chained) catches silent edits
- HMAC ratchet rotates keys every N entries with HKDF-style derivation
and zeroizes old key bytes from memory
- Merkle batch sealing builds tree over content_hashes, signs root
with Ed25519 (FFI to libcrypto EVP_PKEY_ED25519 directly since stdlib
lacks the high-level OpenSSL::PKey wrapper)
AuditLog.append is mutex-guarded for serial writes; verify_chain detects
both single-row tampering and structural inconsistency. BatchSealer's
pack_message format (BE start_seq || BE end_seq || merkle_root) is shared
between signer and verifier so external auditors can validate offline.
- Random with constant-time-equal helper
- KEK loader from env (Tier 1) with InvalidKekError on bad/missing var
- AEAD via direct LibCrypto FFI bindings (Crystal 1.20 stdlib OpenSSL::Cipher
doesn't expose GCM auth_data/auth_tag, so we extend LibCrypto with
EVP_CIPHER_CTX_ctrl + EVP_aes_256_gcm and call them directly)
- Envelope: per-row DEK wrapped by KEK, AAD-bound, algorithm_id reserved
for crypto agility (0x01 = AES-256-GCM today, 0x02/0x03 reserved)
- 17 unit specs including a 100-iteration random property test
- Tag-tampered ciphertext / wrong-AAD / wrong-key all raise Aead::Error
- New Aenebris.ML.IForest module: pure Isolation Forest scorer
implementing the Liu et al. 2008 ICDM formula
anomaly_score = 2^(-E[h(x)] / c(n))
where E[h(x)] is the average path length across iTrees and
c(n) = 2*H(n-1) - 2(n-1)/n (expected unsuccessful BST search depth).
ITree ADT (ITreeLeaf size | ITreeSplit featIdx threshold left right)
with leaf-size c(n) correction added to traversal depth at every
leaf, matching the original paper. Score-only mode (accept pre-
trained forests from Python sklearn or similar); fitting is deferred
to a later phase. Default constants from Liu et al.: 100 trees,
256 subsample size, max depth ceil(log2(256)) = 8.
- 24 tests covering: harmonicNumber edge cases (H(0), H(1), H(2),
H(3), large-n asymptotic), normalizationConstant for n in {0, 1,
2, 256}, single-split and deep-tree path length traversal,
scoreIForest edge cases (empty forest, subsample 0, subsample 1),
shorter-path-equals-higher-score invariant, default constants,
Euler-Mascheroni precision.
- aenebris.cabal: expose Aenebris.ML.IForest in the library stanza.
- test/Spec.hs: import Expectation explicitly from Test.Hspec for the
shouldBeApprox helper.
- 325 total examples passing, 0 GHC warnings on the new module.
The escalation-gate composition with the calibrated GBDT score
(per docs/research/phase-2.5-ml-synthesis.md correction over the
0.8/0.2 folk-wisdom blend) is intentionally NOT in this module --
it belongs in the downstream ML.Engine module that wires Loader +
Inference + Calibration + IForest into a single decision pipeline.
- All 4 repos mirror the SQLite contracts using PG-native types
(UUID, BYTEA, JSONB, BIGSERIAL, TIMESTAMPTZ, SMALLINT)
- audit_no_modify() trigger refuses UPDATE/DELETE/TRUNCATE on audit_events
- pg_advisory_xact_lock for cross-process row locking
- Integration tests verify trigger fires correctly on tamper attempt
- Test cleanup helper temporarily disables trigger to reset between specs
- 4 integration specs all green against PostgreSQL 16
All four repos implement their abstract contracts. Migrations create
tables idempotently. Bytes round-trip through SQLite BLOB columns
correctly. RotatorKind and RotationState enums replace Symbol fields
in record types so they round-trip safely through DB string columns.
8 unit specs verify CRUD, find_by_external, in_flight filtering,
revocation, and audit append/range/latest_hash genesis behavior.
- New Aenebris.ML.Inference module: pure tree-walk over the unified Tree
SoA produced by Aenebris.ML.Loader. Mirrors LightGBM C++ NumericalDecision
exactly: NaN remapped to 0 unless MissingType=NaN; MissingType=Zero/NaN
routes via default-left flag; '<=' predicate (not '<'); categorical
bitmap test using cat_boundaries slice; average_output divisor for RF
mode; sigmoid link for binary logistic objective. kZeroThreshold = 1e-35
matches LightGBM's IsZero exactly. Defensive against NaN/Inf/negative
on categorical paths. 35 tests covering leaf-only, numerical splits,
missing-value semantics, categorical routing, multi-tree sums, sigmoid
saturation, average_output, regression vs binary objectives, and
end-to-end against the disk fixture.
- New Aenebris.ML.Calibration module: pure calibration of binary
classifier outputs. Calibrator ADT (NoCalibrator | PlattCalibrator a b
| IsotonicCalibrator bp). Platt scaling uses Newton's method with
damped line search on Platt-1999 smoothed targets (N+ + 1)/(N+ + 2)
and 1/(N- + 2); numerically stable softplus-based log-loss. Isotonic
regression uses Pool Adjacent Violators on a stack with tie pre-
grouping, sorted (raw, calibrated) breakpoints, linear interpolation,
out-of-range clamping. Sub-2-sample input returns NoCalibrator. 28
tests covering all three calibrator constructors, fit edge cases,
Platt convergence on logistic data, isotonic correction of non-
monotone data, tie handling, all-same-label degenerate cases, and
lookup behavior including clamping and interpolation.
- aenebris.cabal: expose both new modules in the library stanza.
- 301 total examples passing, 0 GHC warnings on either new module.
- shard.yml with db, pg, sqlite3, tourmaline (deps), webmock (dev)
- Makefile with build/test/format/lint/demo targets
- .gitignore, .editorconfig, LICENSE (MIT), README skeleton
- ameba lint deferred until libpcre3-dev is available on this env
- New Aenebris.ML.Loader module: pure ByteString -> Either ParseError Ensemble
parser for LightGBM v4 plain-text models. Header / tree-blocks / trailing-
section state machine; converts split-arrays + leaf-arrays into the unified
Tree SoA via unifyChild + complement; rejects non-v4, multi-class, and
is_linear=1; extracts embedded sigmoid scale and bare average_output flag;
runs validateEnsemble before returning.
- Frozen test fixtures under test/fixtures/ml/ (tiny_lgbm_v4.txt,
stump_lgbm_v4.txt) hand-crafted from the v4 spec annotated example. CI runs
against these with zero Python dependency.
- scripts/regen_lgbm_fixture.py: uv-runnable PEP 723 script that retrains a
tiny binary GBDT and writes to a separate
test/fixtures/ml/regen_sample_v4.txt so the frozen fixtures stay pristine.
- 30 new mlLoaderSpec tests covering happy path, header rejection, sigmoid
extraction, average_output, tree-level rejection, feature_names with '=',
and ParseError reporting. 238 total examples passing, 0 GHC warnings on
Loader.
- Add DEMO.md and screenshots for bug-bounty-platform, hash-cracker,
linux-cis-hardening-auditor, simple-port-scanner, simple-vulnerability-scanner,
systemd-persistence-scanner, base64-tool, caesar-cipher, dns-lookup,
metadata-scrubber-tool, network-traffic-analyzer, siem-dashboard
- Link DEMO.md from project READMEs
- Add .gitignore entries for DEMO-TRACKER and simple-port-scanner build dir
- Restructure haskell-reverse-proxy with DDoS, Fingerprint, ML, RateLimit, WAF,
Geo, and Honeypot modules; drop superseded research docs and old Makefile
- Refresh siem-dashboard dashboard.png and rename alerts.png to alert-detail.png
CarterPerez-dev
dependabot[bot]