Update all five learn documents to match the post-audit codebase:
- Replace references to the deleted server-side X3DH/Double Ratchet (`backend/app/core/encryption/`) with their client-side TypeScript counterparts (`frontend/src/crypto/{x3dh,double-ratchet}.ts`).
- Drop mentions of the removed `RatchetState` and `SkippedMessageKey` PostgreSQL models, the server-generated key paths, and the deprecated `send_encrypted_message` server-encryption method. Add an explicit "why we deleted server-side crypto" rationale in 02-ARCHITECTURE.
- Update API endpoint listings: drop `/encryption/initialize-keys`, `/encryption/rotate-signed-prekey`, `/encryption/opk-count`; add `/auth/me`, `/auth/logout`. Note that all non-auth routes require the session cookie.
- Reflect the new schema: `IdentityKey`, `SignedPrekey`, and `OneTimePrekey` now hold only public material; `User` carries a 64-byte `webauthn_user_handle`.
- Replace the `/ws?user_id=...` example with cookie-authenticated WebSocket usage.
- Remove all line-number references throughout (per the no-line-numbers feedback) — ~300 stripped via regex pass plus targeted prose rewrites.
- Update WebAuthn copy to reflect `UserVerificationRequirement.REQUIRED`, the per-credential-backup-eligibility flag, and challenge-bytes-keyed authentication storage.
- Refresh the file index and project tree to match the trimmed module layout.
This commit applies the actionable plan from docs/plans/2026-04-29-encrypted-p2p-chat-audit-fixes.md.
Backend
- Delete server-side X3DH/Double Ratchet (`app/core/encryption/*`); the server is now an opaque relay for client-encrypted messages.
- Drop `private_key` columns from IdentityKey/SignedPrekey/OneTimePrekey; prekey-bundle endpoint serves only public material.
- Remove RatchetState and SkippedMessageKey models and the deprecated server-encryption code path in message_service.
- Fix the broken `not OneTimePrekey.is_used` SQL filter (`Column.is_(False)` instead of Python `not`); repair the silent OPK-replenish amplification by deleting the server-generated path entirely.
- Stop calling `prekey_service.initialize_user_keys` from registration/login. Server never holds private key material again.
- Add session-cookie auth: opaque token issued in Redis on register/auth complete, `current_user` dependency on every protected route, `/auth/me`, `/auth/logout`. WebSocket authenticates via the same cookie.
- Add room membership checks to WS encrypted_message + typing handlers, GET/DELETE /rooms/{id}, GET /rooms/{id}/messages. broadcast_to_room now filters by participants, not presence.
- Add slowapi rate limits to register/auth/search; per-user sliding-window cap on WS messages.
- WebAuthn: replace username with a 64-byte random `webauthn_user_handle` for `user.id`; fix `backup_eligible` to read `credential_backup_eligible`; switch UserVerificationRequirement to REQUIRED; key authentication challenges by challenge bytes (no more "discoverable" collision).
- Implement GET /rooms/{id} and DELETE /rooms/{id} (cascade delete with ownership check) and fix the N+1 in list_rooms by batching User lookups.
- Remove broken/obsolete tests; add regression tests for the OPK filter bug and session-protected endpoints (401 without cookie).
- Sanitize WebSocket error responses; correlation IDs replace stack-trace leaks.
Frontend
- X3DH: prepend the 32-byte 0xff F prefix to the HKDF input per spec section 2.2.
- Double Ratchet: switch KDF_RK to spec form (HKDF salt=root_key, IKM=dh_output) and use 0x01/0x02 byte tags for KDF_CK.
- Replace base64 helpers in `crypto/primitives.ts` with the URL-safe codec from `lib/base64.ts` (single source of truth, no spread-stack hazard).
- Update auth.service to issue a session via cookies (no more dead token store), call `/auth/me` on app boot, clear crypto state on logout. Delete unused `session.store.ts`.
- Update room.service / Chat page to drop client-supplied user_id arguments; the cookie now identifies the caller.
- Better forward-secrecy UX text for messages predating the device.
- Add Vitest with X3DH and Double Ratchet round-trip + tamper-detection tests.
Tooling
- Add `slowapi>=0.1.9` to backend deps; add `vitest` to frontend devDeps with `pnpm test` script.
- justfile: `test-frontend`, `dev-reset`.
- Adjust `.gitignore` so the Python `lib/` rules no longer hide `frontend/src/lib`.
Migration: existing dev volumes hold private-key columns and KDF-incompatible ratchet states; run `just dev-reset` and re-register before hitting the new code path.
- New module src/Aenebris/Net/IP.hs exposes
sockAddrToIPBytes :: SockAddr -> ByteString.
Single canonical implementation for IPv4 dotted-decimal, IPv6
colon-hex (8 groups), and unix-socket "unix:<path>" rendering.
- Aenebris.RateLimit.clientIPKey reduces to
`sockAddrToIPBytes . remoteHost`. Removes the local v6Bytes,
the hostAddressToTuple/hostAddress6ToTuple/printf/showHex/
intercalate imports, and the duplicated implementation.
- Aenebris.DDoS.ConnLimit drops its private copy of the same
function. ipBytesFromSockAddr is kept as a thin alias for
test backward-compatibility (= sockAddrToIPBytes), so existing
callers and the connLimitSpec test continue to compile without
rename churn.
- aenebris.cabal: expose Aenebris.Net.IP in the library stanza
(now 30 modules total).
- test/Spec.hs: new netIpSpec asserts IPv4 dotted decimal,
loopback, unix-socket prefix, and IPv6 colon-separated rendering
(8 groups → 7 colons). Wires netIpSpec into main right after
geoSpec.
362 examples passing, 0 failures, 0 GHC warnings.
- Geo.hs (Finding 14): geoAsnCounts is now bounded by
defaultGeoAsnCountCap = 200_000. capAsnCounts is called inside
bumpAsnCounter; on overflow, the entry with the oldest
awWindowStart is evicted via Data.List.minimumBy + Data.Ord.comparing.
Closes the unbounded-Map memory growth path. Uses minimumBy and
comparing imports added to the existing module.
- ML/Middleware.hs, WAF/Engine.hs, Honeypot.hs (Finding 19): em
dashes (\x2014) in user-facing response bodies and the generated
robots.txt comment replaced with ASCII hyphens, per the project's
guardrail-safe terminology rule.
- aenebris.cabal (Finding 20): copyright field updated from
'2025 Carter Perez' to '2026 AngelaMos' to match the file headers.
- ML/IForest.hs (Finding 28): pathLength now respects a hard
maxIForestDepth = 64 cutoff. Beyond that depth the function
returns currentDepth + c(1) (= currentDepth) without further
recursion, so a pathological tree built by a buggy fitter cannot
blow the stack. Standard iForest depth is ceil(log2(256)) = 8, so
the cap leaves >>8 generous headroom.
- ML/Model.hs (Finding 29): validateCategoricalNode now also
rejects categorical thresholds that are not whole numbers. A
threshold of 1.5 would silently floor to 1 today; with this
change it is reported as a clear validation error instead.
Real LightGBM never writes non-integer cat indices, so this is
defense-in-depth against malformed exporters.
Build clean, 358 examples passing, 0 failures.
Brings the older pre-ML modules up to the same standard as the
newer ML pipeline. Closes Findings 7 (file headers), 8 (inline
comments), 9 (NIH reimplementations), 11 (selectWeightedRR partial
function + O(n^2)), 12 (weak LoadBalancer tests), 13 (placeholder
Backend / ConnLimit tests), and 32 (locally-redefined HTTP status
constants).
Module rewrites with new file header + zero inline comments:
- src/Aenebris/Proxy.hs
- src/Aenebris/LoadBalancer.hs
- src/Aenebris/HealthCheck.hs
- src/Aenebris/Middleware/Security.hs
- src/Aenebris/Middleware/Redirect.hs
- src/Aenebris/Config.hs
- src/Aenebris/Backend.hs
- app/Main.hs
Algorithmic + structural fixes:
- LoadBalancer.selectWeightedRR no longer uses partial (!!) and a
separate find/maximum/index pass; replaced with a single STM fold
that tags each backend with its current weight and picks via
Data.List.maximumBy + Data.Ord.comparing. O(n) instead of O(n^2),
no Maybe fallback.
- Proxy.proxyApp deduplicates the regular-HTTP code path that was
copy-pasted across the WebSocket and `_` connection-type branches
(Finding 16). Single forwardRegular helper called from both.
- Proxy.logRequest removed; per-request stdout logging was
inconsistent with the stderr error pattern used elsewhere and a
perf hazard at scale (Finding 17). Defer real structured logging
to Phase 4 per docs/status/MASTER_PLAN_2026.md.
- Backend.recordFailure now increments rbTotalFailures uniformly
across all states (Finding 22), not just Unhealthy.
- Backend.Show instance now includes weight (Finding 21).
- Connection.hs gains microsPerSecond and httpOkStatusCode named
constants. tcUpstreamReadSeconds added to TimeoutConfig.
- HealthCheck uses the shared microsPerSecond constant instead of
inline 1000000 (Finding 18). Compares status against the named
httpOkStatusCode constant.
NIH reimplementations replaced with stdlib:
- LoadBalancer: filterM, forM_ from Control.Monad; minimumBy,
maximumBy from Data.List; comparing from Data.Ord.
- Proxy: zipWithM from Control.Monad.
- HealthCheck: zipWithM_ from Control.Monad.
- Middleware.Security: catMaybes from Data.Maybe.
- Config: nub from Data.List (replaces local nubText).
- Fingerprint.JA4H, ML.Features: built-in lookup over
[(CI ByteString, ByteString)] (CI ByteString already has Eq).
- RateLimit, DDoS.ConnLimit: intercalate ":" from Data.List
(replaces duplicated joinColons in two files).
- Honeypot: status200, status404 from Network.HTTP.Types (drops
local mkStatus duplications).
- WAF.Engine: status403 from Network.HTTP.Types.
Test strengthening (test/Spec.hs):
- loadBalancerSpec: round-robin now asserts even distribution
([3,3,3] across 3 backends in 9 rounds); weighted RR asserts the
4-weight backend wins >= 35 of 50 picks; least-connections asserts
the lower-count backend is selected after manually bumping the
other to 5 active connections.
- backendSpec: "tolerates repeated failures" replaced with a real
state-transition test ('Healthy after 1 failure, Healthy after 2,
Unhealthy after 3'). 'records successes without crashing'
replaced with assertion that recordSuccess on Healthy resets
rbConsecutiveFailures to 0.
- connLimitSpec: 'release decrements counter' now reads
currentCount before and after release to assert the count
actually went 1 -> 0.
Build clean, 358 examples passing, 0 failures, 0 GHC warnings on
any touched module. Audit findings 14, 15, 18 (residual MAJOR) and
19-31 (MINOR) deferred to subsequent passes.
Closes the six CRITICAL findings from the 2026-04-29 audit:
- Tunnel.hs (Findings 1+2+4):
- connectToBackend now returns Either ConnectError Socket with
proper handling of resolution failures, connect timeouts, and
connect errors. Replaces the partial `error` call.
- parseUpgradeStatus uses safe pattern matching on BS8.lines
rather than `head $ BS8.lines` (closes the -Wx-partial warning).
- receiveUpgradeResponse is bounded at 16 KB and uses a tight
inner loop with strict accumulator (closes the unbounded-read
DoS). Adds a 30s timeout via System.Timeout.timeout.
- Adds attemptConnect with bracketOnError to ensure socket is
closed on partial-failure paths.
- File header + remove inline comments.
- TLS.hs (Findings 1+15):
- loadCredentials replaced with credentialsOrDefault that
returns empty credentials and logs to stderr on failure
rather than crashing the SNI handler with `error`.
- Cipher names updated to non-deprecated forms
(cipher13_AES_128_GCM_SHA256 etc.) to clear -Wdeprecations.
- File header + remove inline comments.
- Connection.hs (supporting Finding 3):
- File header + add microsPerSecond and httpOkStatusCode named
constants. Add tcUpstreamReadSeconds (default 30) to
TimeoutConfig.
- Proxy.hs (Findings 1+3):
- All five `error` calls replaced with hPutStrLn stderr +
exitFailure (matching Main.hs's pattern).
- forwardRequest wraps withResponse in System.Timeout.timeout
sized off Connection.tcUpstreamReadSeconds. Returns 504 when
upstream does not respond within budget.
- Main.hs (supporting Finding 3):
- HTTP client Manager now configured with managerResponseTimeout
set to tcUpstreamReadSeconds * microsPerSecond. Belt-and-
suspenders with the application-layer timeout in Proxy.hs.
- ML/Loader.hs (Finding 5):
- Added maxNumLeaves (4096) and maxNumTrees (10000) constants.
- finalizeTree rejects num_leaves <= 0 and num_leaves > maxNumLeaves
BEFORE allocating SoA arrays. Closes a crafted-model-file DoS.
- runTrees threads a tree counter and rejects when the parsed
count would exceed maxNumTrees.
- WAF/Rule.hs (Finding 6):
- CompiledRegex changed from newtype to data with an extra
!ByteString field carrying the original pattern.
- Eq instance now compares by stored pattern bytes; reflexivity
holds (x == x is True). Show instance now reveals the pattern
for debugging.
- test/Spec.hs:
- +2 WAF tests verifying Eq CompiledRegex reflexivity and
pattern-distinguishing.
- +2 ML.Loader tests verifying num_leaves bound rejection
(above maxNumLeaves and at zero).
358 total examples passing, 0 failures, 0 GHC warnings on touched
modules. Audit findings 7-32 (MAJOR systemic + remaining MAJOR +
MINOR) are deferred to subsequent passes per the audit plan.
- Switch frontend Dockerfiles to pnpm via corepack (project uses pnpm-lock.yaml)
- Pin SurrealDB to v3.0.5 and switch datastore URL from `file:` to `rocksdb:`
- Add idempotent SurrealDB schema bootstrap on connect to prevent NotFoundError on first SELECT against schemaless tables in 3.x
Walks through the full flow — passkey registration, passkey login,
new conversation lookup, empty thread, first message, end-to-end
chat, and mobile view — matching the format used by the bug-bounty
and SIEM project demos.
- New Aenebris.ML.Middleware module: Wai middleware that wires the
Aenebris.ML.Engine into the per-request pipeline. Per request:
1. Run extractFeatures (FeatureContext + Request) -> FeatureVector
2. Convert via featureVectorToVector -> VU.Vector Double
3. runEngine eng fv -> DecisionDetails
4. Route by ddDecision:
DecisionHuman -> pass through to inner Application,
optionally attach X-Aenebris-ML-Decision
and X-Aenebris-ML-Score response headers
DecisionBot -> 403 + text/plain block body
DecisionChallenge -> 403 + text/html challenge page
All response builders are operator-overridable via the
MLMiddlewareConfig record. Optional logging callback fires once
per request for structured logs / metrics integration.
- defaultBotResponse, defaultChallengeResponse: sensible defaults
matching the existing WAF middleware pattern (403 + descriptive
body). decisionResponseHeader / scoreResponseHeader are exported
so operators can build custom responses that still surface the
signal headers.
- 12 tests via Network.Wai.Test.runSession covering: pass-through
on Human, 403 on Bot, 403 on Challenge with text/html body,
signal-header attachment toggle, custom response builder override,
logging callback invocation count.
- aenebris.cabal: expose Aenebris.ML.Middleware in the library stanza.
- test/Spec.hs: add modifyTVar' to the Control.Concurrent.STM import
list for the logging-callback test.
- 354 total examples passing, 0 GHC warnings on the new module.
This closes the core Phase 2.5 ML pipeline. The full sequence
(Features -> Model -> Loader -> Inference -> Calibration -> IForest
-> Engine -> Middleware) is now fully implemented and integration-
tested. A hosted Aenebris instance can now be configured with a
trained LightGBM model + optional calibrator + optional IForest
and will block bots, challenge ambiguous traffic, and pass humans
on every HTTP request.
- New Aenebris.ML.Engine module: pure decision pipeline composing
the previously-built Loader, Inference, Calibration, and IForest
modules into a single per-request scoring path.
Engine record holds (Ensemble, Calibrator, Maybe IForest,
EngineConfig); runEngine takes a feature vector and produces
DecisionDetails (Decision, raw proba, calibrated proba, optional
IForest score). The Decision is one of DecisionHuman, DecisionBot,
or DecisionChallenge.
- Implements the escalation-gate semantics per the 2026 research
correction (over the deprecated 0.8/0.2 weighted blend):
calibrated <= humanThreshold -> DecisionHuman
calibrated >= botThreshold -> DecisionBot
otherwise (ambiguous band):
if IForest configured and ifScore >= escalation threshold
-> DecisionBot (escalation)
else if challenges enabled
-> DecisionChallenge
else
-> DecisionHuman (fallthrough)
Defaults: 0.3 / 0.7 thresholds, 0.6 IForest escalation, challenges
on by default.
- 17 tests covering: defaultEngineConfig values, decision boundaries
with NoCalibrator and no IForest (low/mid/high leaf ensembles),
ambiguous-band escalation via low-vs-high anomaly IForests,
ambiguous-band with challenges disabled (fallthrough to Human or
escalation to Bot), Platt calibrator pulling decisions across
thresholds, and DecisionDetails field correctness.
- aenebris.cabal: expose Aenebris.ML.Engine in the library stanza.
- 342 total examples passing, 0 GHC warnings on the new module.
The Wai middleware wiring (extract features -> runEngine -> route by
Decision) is the next module (ML.Middleware); Engine intentionally
stays pure / independent of HTTP machinery so it can be unit-tested
without request fixtures.
- New Aenebris.ML.IForest module: pure Isolation Forest scorer
implementing the Liu et al. 2008 ICDM formula
anomaly_score = 2^(-E[h(x)] / c(n))
where E[h(x)] is the average path length across iTrees and
c(n) = 2*H(n-1) - 2(n-1)/n (expected unsuccessful BST search depth).
ITree ADT (ITreeLeaf size | ITreeSplit featIdx threshold left right)
with leaf-size c(n) correction added to traversal depth at every
leaf, matching the original paper. Score-only mode (accept pre-
trained forests from Python sklearn or similar); fitting is deferred
to a later phase. Default constants from Liu et al.: 100 trees,
256 subsample size, max depth ceil(log2(256)) = 8.
- 24 tests covering: harmonicNumber edge cases (H(0), H(1), H(2),
H(3), large-n asymptotic), normalizationConstant for n in {0, 1,
2, 256}, single-split and deep-tree path length traversal,
scoreIForest edge cases (empty forest, subsample 0, subsample 1),
shorter-path-equals-higher-score invariant, default constants,
Euler-Mascheroni precision.
- aenebris.cabal: expose Aenebris.ML.IForest in the library stanza.
- test/Spec.hs: import Expectation explicitly from Test.Hspec for the
shouldBeApprox helper.
- 325 total examples passing, 0 GHC warnings on the new module.
The escalation-gate composition with the calibrated GBDT score
(per docs/research/phase-2.5-ml-synthesis.md correction over the
0.8/0.2 folk-wisdom blend) is intentionally NOT in this module --
it belongs in the downstream ML.Engine module that wires Loader +
Inference + Calibration + IForest into a single decision pipeline.
- New Aenebris.ML.Inference module: pure tree-walk over the unified Tree
SoA produced by Aenebris.ML.Loader. Mirrors LightGBM C++ NumericalDecision
exactly: NaN remapped to 0 unless MissingType=NaN; MissingType=Zero/NaN
routes via default-left flag; '<=' predicate (not '<'); categorical
bitmap test using cat_boundaries slice; average_output divisor for RF
mode; sigmoid link for binary logistic objective. kZeroThreshold = 1e-35
matches LightGBM's IsZero exactly. Defensive against NaN/Inf/negative
on categorical paths. 35 tests covering leaf-only, numerical splits,
missing-value semantics, categorical routing, multi-tree sums, sigmoid
saturation, average_output, regression vs binary objectives, and
end-to-end against the disk fixture.
- New Aenebris.ML.Calibration module: pure calibration of binary
classifier outputs. Calibrator ADT (NoCalibrator | PlattCalibrator a b
| IsotonicCalibrator bp). Platt scaling uses Newton's method with
damped line search on Platt-1999 smoothed targets (N+ + 1)/(N+ + 2)
and 1/(N- + 2); numerically stable softplus-based log-loss. Isotonic
regression uses Pool Adjacent Violators on a stack with tie pre-
grouping, sorted (raw, calibrated) breakpoints, linear interpolation,
out-of-range clamping. Sub-2-sample input returns NoCalibrator. 28
tests covering all three calibrator constructors, fit edge cases,
Platt convergence on logistic data, isotonic correction of non-
monotone data, tie handling, all-same-label degenerate cases, and
lookup behavior including clamping and interpolation.
- aenebris.cabal: expose both new modules in the library stanza.
- 301 total examples passing, 0 GHC warnings on either new module.
- New Aenebris.ML.Loader module: pure ByteString -> Either ParseError Ensemble
parser for LightGBM v4 plain-text models. Header / tree-blocks / trailing-
section state machine; converts split-arrays + leaf-arrays into the unified
Tree SoA via unifyChild + complement; rejects non-v4, multi-class, and
is_linear=1; extracts embedded sigmoid scale and bare average_output flag;
runs validateEnsemble before returning.
- Frozen test fixtures under test/fixtures/ml/ (tiny_lgbm_v4.txt,
stump_lgbm_v4.txt) hand-crafted from the v4 spec annotated example. CI runs
against these with zero Python dependency.
- scripts/regen_lgbm_fixture.py: uv-runnable PEP 723 script that retrains a
tiny binary GBDT and writes to a separate
test/fixtures/ml/regen_sample_v4.txt so the frozen fixtures stay pristine.
- 30 new mlLoaderSpec tests covering happy path, header rejection, sigmoid
extraction, average_output, tree-level rejection, feature_names with '=',
and ParseError reporting. 238 total examples passing, 0 GHC warnings on
Loader.
- Add DEMO.md and screenshots for bug-bounty-platform, hash-cracker,
linux-cis-hardening-auditor, simple-port-scanner, simple-vulnerability-scanner,
systemd-persistence-scanner, base64-tool, caesar-cipher, dns-lookup,
metadata-scrubber-tool, network-traffic-analyzer, siem-dashboard
- Link DEMO.md from project READMEs
- Add .gitignore entries for DEMO-TRACKER and simple-port-scanner build dir
- Restructure haskell-reverse-proxy with DDoS, Fingerprint, ML, RateLimit, WAF,
Geo, and Honeypot modules; drop superseded research docs and old Makefile
- Refresh siem-dashboard dashboard.png and rename alerts.png to alert-detail.png
Add bomber CLI tool (Go) — scans dependencies across Go/Node/Python
ecosystems, generates SPDX 2.3 and CycloneDX 1.5 SBOMs, and matches
against OSV/NVD vulnerability databases with policy engine for CI/CD.
Add file-level docstrings to ai-threat-detection, firewall-rule-engine,
hash-cracker, linux-cis-hardening-auditor, binary-analysis-tool, and
credential-enumeration.
Dashboard: stat cards, severity bar, live alert feed, top IPs/paths.
Threats: filtered paginated table with detail slide-in panel.
Models: status banner, model cards with metrics, retrain button.
Dev: Vite dev server in Node 24 container with hot reload via volume
mount and named volume for node_modules isolation.
Prod: Multi-stage build (pnpm build → nginx) serving static files with
API/WebSocket reverse proxy to backend.
Parses multi-line CSIC 2010 HTTP request files into CSICRequest
dataclasses, converts to ParsedLogEntry with synthesized defaults,
and produces 35-dim feature vectors via the existing extraction pipeline.
Add ONNX inference engine with 3-model ensemble (autoencoder, random forest,
isolation forest), ONNX export utilities, MLflow experiment tracking, and
classifier training scripts. Extend pipeline with hybrid ML/rules detection
mode, ensemble score fusion and blending, and full test coverage for all
new components. Fix Python 3 SyntaxError in parsers.py (except clause).
Complete 5-file learn documentation for the advanced encrypted P2P chat
project covering Signal Protocol (X3DH + Double Ratchet), WebAuthn/Passkeys,
and the fullstack architecture.
Ref #51
- Add headers, badges, learn module pointers, just command runner tips, and AGPLv3 licenses etc. etc.
- Add missing Justfiles in some projects
- Update pre commit hook and add TODO
Replaced security considerations with an overview of the API Rate Limiter project, detailing its features, algorithms, and importance in preventing attacks.
CarterPerez-dev
dependabot[bot]