RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs(encrypted-p2p-chat): refresh learn/ for the audit-fixed architecture 

Update all five learn documents to match the post-audit codebase:

- Replace references to the deleted server-side X3DH/Double Ratchet (`backend/app/core/encryption/`) with their client-side TypeScript counterparts (`frontend/src/crypto/{x3dh,double-ratchet}.ts`).
- Drop mentions of the removed `RatchetState` and `SkippedMessageKey` PostgreSQL models, the server-generated key paths, and the deprecated `send_encrypted_message` server-encryption method. Add an explicit "why we deleted server-side crypto" rationale in 02-ARCHITECTURE.
- Update API endpoint listings: drop `/encryption/initialize-keys`, `/encryption/rotate-signed-prekey`, `/encryption/opk-count`; add `/auth/me`, `/auth/logout`. Note that all non-auth routes require the session cookie.
- Reflect the new schema: `IdentityKey`, `SignedPrekey`, and `OneTimePrekey` now hold only public material; `User` carries a 64-byte `webauthn_user_handle`.
- Replace the `/ws?user_id=...` example with cookie-authenticated WebSocket usage.
- Remove all line-number references throughout (per the no-line-numbers feedback) — ~300 stripped via regex pass plus targeted prose rewrites.
- Update WebAuthn copy to reflect `UserVerificationRequirement.REQUIRED`, the per-credential-backup-eligibility flag, and challenge-bytes-keyed authentication storage.
- Refresh the file index and project tree to match the trimmed module layout.
This commit is contained in:
CarterPerez-dev 2026-04-29 02:57:17 -04:00
parent 6fd5f3d393
commit 5adb079e76
5 changed files with 1954 additions and 2028 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

60
PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md
View File

@ -20,19 +20,19 @@ The Signal Protocol is now the gold standard for encrypted messaging. Signal (th
### Cryptographic Protocols
- **X3DH key exchange for asynchronous session establishment.** The fundamental problem with Diffie-Hellman key exchange is that both parties need to be online at the same time. If Alice wants to send a message to Bob, but Bob is offline, classic DH cannot work. X3DH (Extended Triple Diffie-Hellman) solves this by having Bob pre-publish a bundle of public keys to the server: a long-term identity key, a medium-term signed prekey, and a set of single-use one-time prekeys. When Alice wants to start a conversation, she downloads Bob's bundle and performs three (or four, if a one-time prekey is available) DH computations to derive a shared secret. Bob can complete his half of the exchange when he comes back online. This project implements X3DH in `backend/app/core/encryption/x3dh_manager.py` on the server side and `frontend/src/crypto/x3dh.ts` on the client side, using X25519 for Diffie-Hellman and Ed25519 for prekey signatures.
- **X3DH key exchange for asynchronous session establishment.** The fundamental problem with Diffie-Hellman key exchange is that both parties need to be online at the same time. If Alice wants to send a message to Bob, but Bob is offline, classic DH cannot work. X3DH (Extended Triple Diffie-Hellman) solves this by having Bob pre-publish a bundle of public keys to the server: a long-term identity key, a medium-term signed prekey, and a set of single-use one-time prekeys. When Alice wants to start a conversation, she downloads Bob's bundle and performs three (or four, if a one-time prekey is available) DH computations to derive a shared secret. Bob can complete his half of the exchange when he comes back online. **All X3DH math runs in the browser** in `frontend/src/crypto/x3dh.ts` (`initiateX3DH` and `receiveX3DH`); the backend only stores and serves the public bundle in `backend/app/services/prekey_service.py`. X25519 is used for Diffie-Hellman, Ed25519 for prekey signatures.
- **Double Ratchet algorithm for forward secrecy and post-compromise security.** Once X3DH establishes an initial shared secret, the Double Ratchet takes over for ongoing message encryption. It combines two ratcheting mechanisms: a symmetric-key ratchet (using HMAC-based key derivation) that advances with every message, and a Diffie-Hellman ratchet that advances every time the conversation "turn" changes. The symmetric ratchet provides forward secrecy: if an attacker steals your current keys, they cannot decrypt past messages because each message key is derived from and then discarded after the previous chain key. The DH ratchet provides post-compromise security (also called "future secrecy" or "self-healing"): even if an attacker compromises your current state completely, once you send a new message with a fresh DH key pair, the attacker loses access because they do not know the new private key. The implementation lives in `backend/app/core/encryption/double_ratchet.py` and `frontend/src/crypto/double-ratchet.ts`.
- **Double Ratchet algorithm for forward secrecy and post-compromise security.** Once X3DH establishes an initial shared secret, the Double Ratchet takes over for ongoing message encryption. It combines two ratcheting mechanisms: a symmetric-key ratchet (HMAC-based) that advances with every message, and a Diffie-Hellman ratchet that advances every time the conversation "turn" changes. The symmetric ratchet provides forward secrecy: if an attacker steals your current keys, they cannot decrypt past messages because each message key is derived from and then discarded after the previous chain key. The DH ratchet provides post-compromise security (also called "future secrecy" or "self-healing"): even if an attacker compromises your current state completely, once you send a new message with a fresh DH key pair, the attacker loses access because they do not know the new private key. **The Double Ratchet runs entirely in the browser** in `frontend/src/crypto/double-ratchet.ts` (`encryptMessage` / `decryptMessage`); the backend only relays the resulting ciphertext.
- **AES-256-GCM authenticated encryption.** Each message is encrypted using AES-256-GCM (Galois/Counter Mode), which provides both confidentiality and integrity. GCM is an AEAD (Authenticated Encryption with Associated Data) cipher, meaning it produces a ciphertext and an authentication tag. If anyone tampers with the ciphertext, the tag verification fails and decryption is rejected. The "associated data" in this project is the concatenation of both users' identity public keys, binding each encrypted message to the specific conversation participants. The implementation uses 12-byte random nonces and 32-byte keys derived from the Double Ratchet chain. You will see this in the `_encrypt_with_message_key` and `_decrypt_with_message_key` methods of the `DoubleRatchet` class, and in the `aesGcmEncrypt`/`aesGcmDecrypt` functions in `frontend/src/crypto/primitives.ts`.
- **AES-256-GCM authenticated encryption.** Each message is encrypted using AES-256-GCM (Galois/Counter Mode), which provides both confidentiality and integrity. GCM is an AEAD (Authenticated Encryption with Associated Data) cipher, meaning it produces a ciphertext and an authentication tag. If anyone tampers with the ciphertext, the tag verification fails and decryption is rejected. The "associated data" in this project is the serialized message header (DH public key, message number, previous chain length), binding each ciphertext to the ratchet state that produced it. The implementation uses 12-byte random nonces and 32-byte keys derived from the Double Ratchet chain. You'll see it in `aesGcmEncrypt` / `aesGcmDecrypt` in `frontend/src/crypto/primitives.ts`, called from inside `encryptMessage` / `decryptMessage`.
- **HKDF-SHA256 key derivation.** Raw Diffie-Hellman outputs are not suitable for direct use as encryption keys. They have non-uniform distribution and may contain structural biases. HKDF (HMAC-based Key Derivation Function) extracts entropy from DH outputs and expands it into keys of the required length. This project uses HKDF-SHA256 throughout: in X3DH to derive the initial shared secret from concatenated DH outputs, and in the Double Ratchet to derive new root keys and chain keys from each DH ratchet step. The X3DH derivation uses a salt of all-zero bytes and an info string of `b'X3DH'`, prepending a block of `0xFF` bytes as specified by the Signal Protocol. You will see this in both the Python `cryptography` library's HKDF and the browser's WebCrypto `deriveBits` API with HKDF.
- **HKDF-SHA256 key derivation.** Raw Diffie-Hellman outputs are not suitable for direct use as encryption keys. They have non-uniform distribution and may contain structural biases. HKDF (HMAC-based Key Derivation Function) extracts entropy from DH outputs and expands it into keys of the required length. This project uses HKDF-SHA256 throughout: in X3DH to derive the initial shared secret from concatenated DH outputs (with the spec-mandated `0xFF * 32` F prefix and `b'X3DH'` info), and in the Double Ratchet's KDF_RK (root-key derivation) which uses the previous root key as HKDF salt and the new DH output as input keying material. The chain-key step (KDF_CK) uses HMAC-SHA256 with single-byte tags `0x01` (message key) and `0x02` (next chain key) per the Signal spec.
- **Ed25519 digital signatures and X25519 Diffie-Hellman.** The project uses two related but distinct elliptic curve operations on Curve25519. X25519 is used for Diffie-Hellman key exchange: two parties each generate a key pair, exchange public keys, and compute a shared secret. Ed25519 is used for digital signatures: the identity key owner signs their signed prekey so that other users can verify it was genuinely published by that identity, not substituted by a malicious server. The distinction matters because the mathematical operations are different (X25519 operates on the Montgomery form of the curve, Ed25519 on the Edwards form), but they share the same underlying curve. This project maintains separate X25519 and Ed25519 key pairs for each user identity, as you can see in the `generate_identity_keypair_x25519` and `generate_identity_keypair_ed25519` methods.
- **Ed25519 digital signatures and X25519 Diffie-Hellman.** The project uses two related but distinct elliptic curve operations on Curve25519. X25519 is used for Diffie-Hellman key exchange: two parties each generate a key pair, exchange public keys, and compute a shared secret. Ed25519 is used for digital signatures: the identity key owner signs their signed prekey so that other users can verify it was genuinely published by that identity, not substituted by a malicious server. The mathematical operations are different (X25519 operates on the Montgomery form of the curve, Ed25519 on the Edwards form), but they share the same underlying curve. Each user generates separate X25519 and Ed25519 identity key pairs in `generateIdentityKeyPair` (`frontend/src/crypto/x3dh.ts`); only the public halves ever leave the device.
### Authentication and Identity
- **WebAuthn/FIDO2 passkey registration and authentication.** WebAuthn is a W3C standard that allows users to authenticate using public key cryptography instead of passwords. During registration, the browser creates a new key pair on a hardware or platform authenticator (a YubiKey, a fingerprint reader, a phone's secure enclave). The public key is sent to the server. The private key never leaves the authenticator. During login, the server sends a random challenge, the authenticator signs it with the private key, and the server verifies the signature against the stored public key. No shared secret ever crosses the network. The implementation uses the `py_webauthn` library on the server side (`backend/app/core/passkey/passkey_manager.py`) and the browser's native `navigator.credentials` API on the client. Registration options include resident key requirement (enabling usernameless login) and user verification preference.
- **WebAuthn/FIDO2 passkey registration and authentication.** WebAuthn is a W3C standard that allows users to authenticate using public key cryptography instead of passwords. During registration, the browser creates a new key pair on a hardware or platform authenticator (a YubiKey, a fingerprint reader, a phone's secure enclave). The public key is sent to the server. The private key never leaves the authenticator. During login, the server sends a random challenge, the authenticator signs it with the private key, and the server verifies the signature against the stored public key. No shared secret ever crosses the network. The implementation uses the `py_webauthn` library on the server (`backend/app/core/passkey/passkey_manager.py`) and the browser's native `navigator.credentials` API on the client. Registration options require resident keys (enabling usernameless login) and **require** user verification (PIN/biometric), since this is an E2EE chat app and unattended-device access shouldn't unlock the conversation. The WebAuthn `user.id` is a 64-byte random `webauthn_user_handle` per spec, stored on the User row and never exposed to clients.
- **Public key cryptography for identity verification.** Each user in the system has two types of public keys: WebAuthn credentials (for authentication with the server) and Signal Protocol identity keys (for end-to-end encryption with other users). The WebAuthn credential proves to the server that you are who you claim to be. The Signal Protocol identity key lets other users verify your encryption keys. These are separate key pairs serving separate purposes. The identity key is an Ed25519 key pair that signs the user's medium-term signed prekeys, letting recipients verify that the prekey bundle they downloaded was genuinely published by that user and not substituted by a compromised server.
@ -40,21 +40,21 @@ The Signal Protocol is now the gold standard for encrypted messaging. Signal (th
### Fullstack Architecture
- **FastAPI async backend with SQLModel ORM.** The backend is built with FastAPI, using its native async support with `asyncio` and `async/await` throughout. Database models use SQLModel, which merges SQLAlchemy's ORM capabilities with Pydantic's data validation. The app factory pattern in `factory.py` creates the FastAPI instance with a lifespan context manager that handles startup (initializing PostgreSQL, connecting to Redis, connecting to SurrealDB) and shutdown (disconnecting cleanly). API routes are organized into four routers: `auth` (WebAuthn registration and login), `encryption` (prekey bundle management), `rooms` (chat room CRUD), and `websocket` (real-time message transport). The backend uses `ORJSONResponse` as the default response class for faster JSON serialization.
- **FastAPI async backend with SQLModel ORM.** The backend is built with FastAPI, using its native async support with `asyncio` and `async/await` throughout. Database models use SQLModel, which merges SQLAlchemy's ORM capabilities with Pydantic's data validation. The app factory pattern in `factory.py` creates the FastAPI instance with a lifespan context manager that handles startup (initializing PostgreSQL, connecting to Redis, connecting to SurrealDB) and shutdown (disconnecting cleanly). API routes are organized into four routers: `auth` (WebAuthn registration / login / `/auth/me` / `/auth/logout`), `encryption` (public prekey bundle upload and lookup), `rooms` (chat room CRUD with membership-gated access), and `websocket` (real-time message transport, authenticated via the same session cookie). All non-auth endpoints depend on `current_user` from `app/core/dependencies.py`, which resolves the session cookie against Redis. Rate limits are applied via `slowapi`. The backend uses `ORJSONResponse` as the default response class for faster JSON serialization.
- **SolidJS reactive frontend with TypeScript.** The frontend uses SolidJS, a reactive UI framework with fine-grained reactivity (no virtual DOM). State management uses Nanostores with the `@nanostores/solid` binding and `@nanostores/persistent` for localStorage persistence. Routing uses `@solidjs/router`. Server state management uses `@tanstack/solid-query` for data fetching, caching, and synchronization. The component structure is organized into domain folders: `Auth` (AuthCard, AuthForm, PasskeyButton), `Chat` (ChatInput, ChatHeader, ConversationList, MessageBubble, MessageList, EncryptionBadge, OnlineStatus, TypingIndicator, UserSearch), `Layout` (AppShell, Header, Sidebar, ProtectedRoute), and `UI` (Avatar, Badge, Button, Dropdown, Input, Modal, Spinner, Toast, Tooltip). Styling uses Tailwind CSS v4 via the Vite plugin.
- **WebSocket real-time messaging.** Messages are delivered in real time over WebSocket connections. The backend's `websocket_manager.py` maintains a connection pool and handles broadcasting. The frontend's `websocket/websocket-manager.ts` manages connection lifecycle, reconnection, and heartbeats. Message types include encrypted messages, typing indicators, presence updates, read receipts, and error notifications. The WebSocket protocol carries ciphertext between clients; the server routes messages without decrypting them. Heartbeat interval is configurable (default 30 seconds), and each user can have up to 5 concurrent connections (for multiple devices or tabs).
- **Multi-database architecture: PostgreSQL, SurrealDB, Redis.** The project uses three databases, each chosen for a specific access pattern. PostgreSQL (via `asyncpg` and SQLModel) stores authentication data, user profiles, WebAuthn credentials, and encryption key metadata (identity keys, signed prekeys, one-time prekeys, ratchet state). This data is relational, requires ACID transactions, and benefits from SQL's querying power. SurrealDB stores real-time chat data: messages, room membership, and presence information. Its live query feature enables real-time push notifications without polling. Redis stores ephemeral state: WebAuthn challenge tokens (with TTL-based expiration), rate limiting counters, and session caches. Each database has its own manager class (`Base.py` for PostgreSQL, `surreal_manager.py` for SurrealDB, `redis_manager.py` for Redis) handling connection lifecycle and operations.
- **Multi-database architecture: PostgreSQL, SurrealDB, Redis.** The project uses three databases, each chosen for a specific access pattern. PostgreSQL (via `asyncpg` and SQLModel) stores authentication data, user profiles, WebAuthn credentials, and **public** prekey material (identity public keys, signed prekey publics + signatures, unused one-time prekey publics). No private key ever lives server-side. SurrealDB stores real-time chat data: messages (ciphertext only), room membership, and presence. Its live query feature enables real-time push notifications without polling. Redis stores ephemeral state: WebAuthn registration context (challenge + user handle + display name), authentication challenges (keyed by challenge bytes for replay-safety), session tokens (`session:<token> -> user_id` with 24h TTL), and `slowapi` rate-limit counters. Each database has its own manager class (`Base.py` for PostgreSQL, `surreal_manager.py` for SurrealDB, `redis_manager.py` for Redis).
- **Docker containerization with Nginx reverse proxy.** The entire application runs in Docker containers orchestrated by Docker Compose. The development compose file (`dev.compose.yml`) defines six services: PostgreSQL 16 Alpine, SurrealDB (latest), Redis 8 Alpine, the FastAPI backend, the Vite dev server for the frontend, and Nginx Alpine as a reverse proxy. Each database has health checks to ensure the backend only starts after its dependencies are ready. The Nginx configuration routes HTTP traffic and proxies WebSocket connections. There are separate Dockerfiles for development (with hot-reload via volume mounts) and production (with optimized builds) in `conf/docker/dev/` and `conf/docker/prod/`.
### Security Engineering
- **Zero-knowledge server design.** The server in this application is designed so that it cannot read messages even if it wanted to. Private keys are generated on client devices and stored in the browser's IndexedDB. The server stores only public keys and encrypted message blobs. When Alice sends a message to Bob, the client-side crypto layer encrypts the plaintext using the Double Ratchet, sends the ciphertext over the WebSocket, and the server forwards it to Bob. At no point does the server possess any decryption key. The associated data binding (concatenation of both users' identity public keys) ensures that even if the server tried to relay a message to the wrong recipient, decryption would fail because the associated data would not match. This is a meaningful defense against a compromised server attempting message misdirection.
- **Zero-knowledge server design.** The server in this application is designed so that it cannot read messages even if it wanted to. Private keys are generated on client devices (`generateIdentityKeyPair` in `frontend/src/crypto/x3dh.ts`) and stored in the browser's IndexedDB (`frontend/src/crypto/key-store.ts`). The server stores only public keys (`backend/app/services/prekey_service.py:store_client_keys` writes only the public halves; the schema columns for private keys were dropped) and ciphertext blobs in SurrealDB. When Alice sends a message to Bob, `crypto-service.ts:encrypt` runs the Double Ratchet locally, hands the ciphertext to the WebSocket, and the server's `message_service.store_encrypted_message` does a pure pass-through write — there is no `decrypt` code path on the backend. The header serialization that becomes the AES-GCM associated data binds each ciphertext to the ratchet state that produced it, so a compromised server cannot redirect ciphertext between sessions without breaking authentication.
- **Key rotation and lifecycle management.** Cryptographic keys in this system have defined lifecycles. One-time prekeys are single-use: once consumed during an X3DH exchange, they are deleted from the server. The system maintains a pool of 100 one-time prekeys and replenishes them when the count drops. Signed prekeys rotate every 48 hours, with a 7-day retention period for old keys to handle in-flight messages. Identity keys are long-term but can be rotated (configurable, default 90 days). Ratchet state is per-conversation and advances with every message. Skipped message keys (for out-of-order delivery) are cached up to a configurable limit (default 2000) with the oldest keys evicted first. These lifecycle parameters are centralized in `config.py` as named constants.
- **Key rotation and lifecycle management.** Cryptographic keys in this system have defined lifecycles, all driven by the client. One-time prekeys are single-use: when a sender fetches Bob's bundle, the chosen OPK is marked `is_used=True` server-side (so it can't be handed out twice), but the private OPK lives only in Bob's IndexedDB. The client maintains a pool of 100 OPKs and replenishes (and re-uploads the new public OPKs) when the unused count drops below half (`crypto-service.ts:replenishOneTimePreKeys`). Signed prekeys rotate every 48 hours; the client generates a new SPK locally and re-runs `uploadKeys` to publish the new public + signature. Identity keys are long-term and intentionally don't rotate — they're the trust anchor that signs the SPK. Ratchet state is per-conversation, advances with every message, and is serialized into the IndexedDB ratchet store after every encrypt/decrypt. Skipped message keys (for out-of-order delivery) live in the in-memory ratchet state, capped at `MAX_SKIP_MESSAGE_KEYS`.
- **Threat modeling for messaging applications.** Building this project exposes you to the specific threat model that the Signal Protocol is designed to address. The primary threats include: a compromised server attempting to read or modify messages (mitigated by E2E encryption and associated data binding), an attacker who steals a user's current session keys (mitigated by forward secrecy from the symmetric ratchet), an attacker who fully compromises a device and then loses access (mitigated by post-compromise security from the DH ratchet), a man-in-the-middle attempting to substitute public keys (mitigated by identity key signing and optional safety number verification), and an attacker who intercepts one-time prekeys (mitigated by the three base DH computations that provide security even without the one-time prekey). Understanding which threats each mechanism addresses is the core of cryptographic protocol design.
@ -158,34 +158,30 @@ encrypted-p2p-chat/
| | | +-- websocket.py # WebSocket connection and message routing
| | |
| | |-- core/
| | | |-- encryption/
| | | | |-- x3dh_manager.py # X3DH protocol: key generation, exchange, verification
| | | | +-- double_ratchet.py # Double Ratchet: symmetric + DH ratchet, encrypt/decrypt
| | | |-- passkey/
| | | | +-- passkey_manager.py # WebAuthn credential registration and verification
| | | |-- dependencies.py # current_user, issue_session, revoke_session
| | | |-- websocket_manager.py # Connection pool, broadcasting, heartbeat management
| | | |-- surreal_manager.py # SurrealDB client: connect, query, live subscriptions
| | | |-- redis_manager.py # Redis client: challenges, rate limits, ephemeral cache
| | | |-- redis_manager.py # Redis client: sessions, challenges, ephemeral cache
| | | |-- enums.py # Shared enumerations (message types, status codes)
| | | |-- exceptions.py # Domain-specific exception classes
| | | +-- exception_handlers.py # FastAPI exception handler registration
| | |
| | |-- models/ # SQLModel database models (PostgreSQL)
| | | |-- Base.py # SQLAlchemy engine, session factory, init_db
| | | |-- User.py # User account with username and display name
| | | |-- Credential.py # WebAuthn credential storage (public key, counter)
| | | |-- IdentityKey.py # Long-term X25519 + Ed25519 identity key pairs
| | | |-- SignedPrekey.py # Medium-term signed prekeys (rotate every 48h)
| | | |-- OneTimePrekey.py # Single-use prekeys for X3DH (pool of 100)
| | | |-- RatchetState.py # Per-conversation Double Ratchet state blob
| | | +-- SkippedMessageKey.py # Cached keys for out-of-order message delivery
| | | |-- User.py # User account + 64-byte WebAuthn user_handle
| | | |-- Credential.py # WebAuthn credential storage (public key, counter, BE/BS flags)
| | | |-- IdentityKey.py # Long-term X25519 + Ed25519 identity public keys (no privates)
| | | |-- SignedPrekey.py # Client-uploaded signed prekey publics + signatures
| | | +-- OneTimePrekey.py # Single-use prekey publics (consumed via is_used flag)
| | |
| | |-- services/ # Business logic layer
| | | |-- auth_service.py # Registration and authentication orchestration
| | | |-- prekey_service.py # Key bundle management and rotation scheduling
| | | |-- message_service.py # Encryption/decryption coordination
| | | |-- auth_service.py # Registration / authentication / search (no key gen)
| | | |-- prekey_service.py # Public-only prekey bundle storage + lookup
| | | |-- message_service.py # Pass-through ciphertext writer (server never decrypts)
| | | |-- presence_service.py # Online/offline status tracking via SurrealDB
| | | +-- websocket_service.py # WebSocket message routing and type dispatch
| | | +-- websocket_service.py # WebSocket message routing + membership check + per-user rate cap
| | |
| | |-- schemas/ # Pydantic request/response models
| | | |-- auth.py # Registration/login options and verification schemas
@ -199,11 +195,10 @@ encrypted-p2p-chat/
| | +-- main.py # Uvicorn entry point
| |
| |-- tests/ # pytest test suite
| | |-- conftest.py # Shared fixtures (async engine, test client)
| | |-- test_auth_service.py # Authentication flow tests
| | |-- test_encryption.py # Double Ratchet encrypt/decrypt round-trip tests
| | |-- test_message_service.py # Message service integration tests
| | +-- test_x3dh.py # X3DH key exchange protocol tests
| | |-- conftest.py # Shared fixtures (in-memory async SQLite, test users)
| | |-- test_auth_service.py # User CRUD and lookup
| | |-- test_prekey_service.py # Bundle/upload regression tests (incl. broken-filter regression)
| | +-- test_session_auth.py # Protected endpoints return 401 without session cookie
| |
| |-- alembic/ # Database migration scripts
| | +-- env.py # Alembic environment configuration
@ -267,8 +262,7 @@ encrypted-p2p-chat/
| | | +-- room.service.ts # Room and message API calls
| | |
| | |-- stores/ # Nanostores state management
| | | |-- auth.store.ts # Authentication state (user, token)
| | | |-- session.store.ts # Session lifecycle management
| | | |-- auth.store.ts # Current user + persistent userId hint
| | | |-- rooms.store.ts # Room list and active room state
| | | |-- messages.store.ts # Message history per conversation
| | | |-- presence.store.ts # User online status tracking
@ -408,7 +402,7 @@ Check that ports 5432, 8001, 6379, 8000, 5173, and 80 are not already in use. No
WebAuthn requires either `localhost` or a valid HTTPS origin. The default configuration uses `RP_ID=localhost` and `RP_ORIGIN=http://localhost` in `.env.example`. Make sure your `.env` file has these values set correctly. If you are accessing the frontend at `http://localhost:5173`, then `RP_ORIGIN` must match that exact origin (scheme + host + port). If you are testing from a LAN IP address or a non-localhost hostname, you will need a TLS certificate because WebAuthn will refuse to create credentials on non-secure origins other than localhost.
**"No identity key" error when starting a chat:**
After WebAuthn registration, the client must generate and upload encryption keys (identity key pair, signed prekey, one-time prekeys) to the server. This happens in `crypto-service.ts`. If this step fails silently, the user will have an account but no encryption keys, and other users will not be able to initiate encrypted sessions with them. Check the browser console for errors during the key generation step. Common causes include IndexedDB access being blocked (private browsing mode in some browsers restricts IndexedDB), WebCrypto API not being available (requires a secure context), or network errors during the key upload POST request to the `/api/encryption/keys` endpoint.
After WebAuthn registration, the client must generate and upload **public** encryption keys (identity key pair publics, signed prekey + signature, one-time prekey publics) to the server. This happens in `crypto-service.ts:initialize`, which then calls `uploadPublicKeys` → `POST /encryption/upload-keys/{user_id}`. If this step fails silently, the user will have an account but no published prekey bundle, and other users will get a 400 "User has not uploaded keys yet" when trying to initiate an X3DH session. Check the browser console for errors during the key generation step. Common causes include IndexedDB access being blocked (private browsing mode in some browsers restricts IndexedDB), WebCrypto API not being available (requires a secure context), or the session cookie not being sent (verify the request has `credentials: 'include'`, which the shared `api-client.ts` already sets).
**Database migration errors:**
If you see schema mismatch errors on startup, run migrations first. For Docker: `just migrate head`. For local development: `just migrate-local head`. If you need to start fresh, `just dev-down` followed by removing the Docker volumes will give you a clean database.

183
PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md
View File

@ -107,22 +107,22 @@ Double Ratchet on ciphertext, nonce, Double Ratchet on
Alice's device and header verbatim Bob's device
```
On the server side, `message_service.py:269-314` implements the
`store_encrypted_message` method. Look at the docstring on line 280:
On the server side, `backend/app/services/message_service.py` implements the
`store_encrypted_message` method. Look at the docstring :
`"Stores client-encrypted message in SurrealDB (pass-through, no server encryption)"`.
The function receives `ciphertext`, `nonce`, and `header` as strings
from the client, and stores them directly in SurrealDB at lines 292-302
from the client, and stores them directly in SurrealDB
without any decryption or re-encryption step. The server is a
passthrough. It writes what it receives and reads what it wrote. At no
point does it import any cryptographic key or call any decryption
function on these message parameters.
On the client side, `crypto-service.ts:217-249` implements the
On the client side, `frontend/src/crypto/crypto-service.ts` implements the
`encrypt` method. The client calls `encryptMessage` from
`double-ratchet.ts` at line 229, receives ciphertext and nonce, then
`double-ratchet.ts` , receives ciphertext and nonce, then
sends these as base64-encoded strings to the server. The plaintext
never leaves the client's process. The actual symmetric encryption
happens in `primitives.ts:224-259` using the WebCrypto API's AES-GCM
happens in `frontend/src/crypto/primitives.ts` using the WebCrypto API's AES-GCM
implementation, which runs in the browser's native cryptographic module
rather than in JavaScript. This means the plaintext is never even
accessible to JavaScript debugging tools during the encryption
@ -227,9 +227,9 @@ cryptographic identity. It is generated once and kept for the lifetime
of the account. This project generates two identity keypairs per user:
- An X25519 keypair for Diffie-Hellman operations
(ref: `x3dh_manager.py:60-86`)
(ref: `frontend/src/crypto/x3dh.ts`)
- An Ed25519 keypair for digital signatures
(ref: `x3dh_manager.py:88-114`)
(ref: `frontend/src/crypto/x3dh.ts`)
The X25519 keypair participates directly in the DH calculations. The
Ed25519 keypair signs the signed prekey to prove it belongs to the same
@ -244,10 +244,10 @@ their identity.
The signed prekey is an X25519 keypair that rotates periodically. In
this project, rotation happens every 48 hours as configured at
`config.py:76` (`SIGNED_PREKEY_ROTATION_HOURS=48`).
`config.py` (`SIGNED_PREKEY_ROTATION_HOURS=48`).
When a new SPK is generated (`x3dh_manager.py:116-152`), the public
key is signed using the Ed25519 identity key at line 141:
When a new SPK is generated (`frontend/src/crypto/x3dh.ts`), the public
key is signed using the Ed25519 identity key :
`signature = identity_private.sign(spk_public_bytes)`. This signature
proves that the SPK was created by the holder of the identity key. When
Alice downloads Bob's prekey bundle, she verifies this signature before
@ -260,24 +260,23 @@ computed with them become unrecoverable). Longer rotation means fewer
key management operations and less complexity. The 48-hour window used
here is consistent with Signal's recommendation.
Old signed prekeys are retained for 7 days after deactivation
(`config.py:77`, `SIGNED_PREKEY_RETENTION_DAYS=7`) to allow
messages-in-flight that were encrypted against the old SPK to still be
decrypted. After that retention period,
`prekey_service.py:428-465` (`cleanup_old_signed_prekeys`) deletes them
permanently.
Old signed prekeys are kept around (with `is_active = False`) so
messages-in-flight that were encrypted against the previous SPK can
still complete their initial X3DH on the receiver. There is no
automated reaper in this codebase; pruning very old inactive SPKs is a
suggested extension in `04-CHALLENGES.md`.
**One-Time Prekey (OPK) -- Single use, consumed and deleted**
One-time prekeys are X25519 keypairs that are used exactly once and then
discarded. They are generated in batches
(ref: `x3dh_manager.py:154-174` for generation,
`prekey_service.py:363-407` for batch replenishment) and uploaded to the
(ref: `frontend/src/crypto/x3dh.ts` for generation,
`backend/app/services/prekey_service.py` for batch replenishment) and uploaded to the
server.
When Alice initiates a conversation with Bob, the server gives Alice one
of Bob's unused OPKs and marks it as consumed
(`prekey_service.py:331-338`). This OPK participates in the fourth DH
(`backend/app/services/prekey_service.py`). This OPK participates in the fourth DH
operation (DH4) of the X3DH handshake. Because the OPK is used only
once and then deleted, it provides an additional layer of forward
secrecy specifically for the initial message of a conversation.
@ -287,7 +286,7 @@ other users initiating conversations), X3DH falls back to three DH
operations instead of four. The protocol still works, but the initial
message has slightly weaker forward secrecy because the fourth DH
operation is skipped. The system generates 100 OPKs initially
(`config.py:75`, `DEFAULT_ONE_TIME_PREKEY_COUNT=100`) and replenishes
(`config.py`, `DEFAULT_ONE_TIME_PREKEY_COUNT=100`) and replenishes
them when the supply drops below half.
**Ephemeral Key (EK) -- Generated per session, never stored**
@ -297,11 +296,11 @@ sender) at the moment she initiates a conversation. It is used in DH2,
DH3, and DH4 of the X3DH handshake. It is never stored on disk; it
exists only in memory for the duration of the key exchange computation.
In the code, it is generated at `x3dh_manager.py:227-228`:
In the code, it is generated at `frontend/src/crypto/x3dh.ts`:
```python
alice_ek_private = X25519PrivateKey.generate()
alice_ek_public = alice_ek_private.public_key()
alice_ek_private = X25519PrivateKey.generate
alice_ek_public = alice_ek_private.public_key
```
After the shared secret is computed, Alice sends the ephemeral public
@ -346,7 +345,7 @@ Key Material Derivation:
SK = HKDF-SHA256(salt, input, info, length=32)
```
In the codebase, the sender side is at `x3dh_manager.py:241-264`:
In the codebase, the sender side is at `frontend/src/crypto/x3dh.ts`:
- Line 241: `dh1 = alice_ik_private.exchange(bob_spk_public)` -- DH1
- Line 242: `dh2 = alice_ek_private.exchange(bob_ik_public)` -- DH2
@ -355,15 +354,15 @@ In the codebase, the sender side is at `x3dh_manager.py:241-264`:
- Line 252: `key_material = dh1 + dh2 + dh3 + dh4` -- concatenation
- Lines 257-264: HKDF derivation with `0xFF * 32` prefix and `b'X3DH'` info string
The receiver side at `x3dh_manager.py:308-333` performs the same
The receiver side at `frontend/src/crypto/x3dh.ts` performs the same
operations but with the roles reversed. DH1 becomes
`bob_spk_private.exchange(alice_ik_public)` (line 308), because Bob has
`bob_spk_private.exchange(alice_ik_public)` , because Bob has
the SPK private key and Alice's IK public key. The property of
Diffie-Hellman guarantees that `X25519(a_priv, B_pub)` produces the
same result as `X25519(b_priv, A_pub)`, so both sides compute identical
shared secrets.
The `0xFF * 32` prefix prepended at line 257 (`f = b'\xff' * X25519_KEY_SIZE`)
The `0xFF * 32` prefix prepended (`f = b'\xff' * X25519_KEY_SIZE`)
is a fixed padding specified by the X3DH standard. It ensures the HKDF
input is at least 32 bytes long even in edge cases and provides domain
separation from other uses of the same keys.
@ -422,7 +421,7 @@ Before performing any DH operations, Alice must verify that Bob's signed
prekey actually belongs to Bob. A malicious server could substitute its
own SPK and intercept communications.
The verification happens at `x3dh_manager.py:217-220` inside
The verification happens at `frontend/src/crypto/x3dh.ts` inside
`perform_x3dh_sender`:
```python
@ -432,10 +431,10 @@ if not self.verify_signed_prekey(bob_bundle.signed_prekey,
raise ValueError("Invalid signed prekey signature")
```
The `verify_signed_prekey` method at `x3dh_manager.py:176-206` uses
The `verify_signed_prekey` method at `frontend/src/crypto/x3dh.ts` uses
Ed25519 signature verification. It decodes the SPK public key bytes, the
signature bytes, and the Ed25519 identity public key bytes, then calls
`identity_public.verify(signature_bytes, spk_public_bytes)` at line 196.
`identity_public.verify(signature_bytes, spk_public_bytes)` .
Ed25519 verification either succeeds or raises `InvalidSignature`. If
verification fails, the entire X3DH handshake is aborted.
@ -515,12 +514,12 @@ The root chain advances during a DH ratchet step. It takes the current
root key and a fresh DH output (from a new DH key exchange) and
produces a new root key and a new chain key.
Reference: `double_ratchet.py:79-94`
Reference: `frontend/src/crypto/double-ratchet.ts`
```python
def _kdf_rk(self, root_key: bytes, dh_output: bytes) -> tuple[bytes, bytes]:
hkdf = HKDF(
algorithm = hashes.SHA256(),
algorithm = hashes.SHA256,
length = HKDF_OUTPUT_SIZE * 2, # 64 bytes total
salt = root_key, # current root key as salt
info = b'',
@ -543,17 +542,17 @@ and the message chains.
The symmetric chains advance with every message. Each step takes the
current chain key and produces the next chain key and a message key.
Reference: `double_ratchet.py:96-109`
Reference: `frontend/src/crypto/double-ratchet.ts`
```python
def _kdf_ck(self, chain_key: bytes) -> tuple[bytes, bytes]:
h_chain = hmac.HMAC(chain_key, hashes.SHA256())
h_chain = hmac.HMAC(chain_key, hashes.SHA256)
h_chain.update(b'\x01')
next_chain_key = h_chain.finalize()
next_chain_key = h_chain.finalize
h_message = hmac.HMAC(chain_key, hashes.SHA256())
h_message = hmac.HMAC(chain_key, hashes.SHA256)
h_message.update(b'\x02')
message_key = h_message.finalize()
message_key = h_message.finalize
return next_chain_key, message_key
```
@ -618,17 +617,17 @@ even if an attacker had compromised all previous key material, the new
DH exchange produces a shared secret they cannot predict, and all
subsequent keys are secure again.
The implementation spans `double_ratchet.py:155-213`:
The implementation spans `frontend/src/crypto/double-ratchet.ts`:
- `_dh_ratchet_send` (lines 155-176): Called when the sender needs to
advance the ratchet. Generates a new DH keypair at line 159, performs
DH with the peer's public key at line 169, and derives new root and
sending chain keys at lines 171-174.
- `_dh_ratchet_send` : Called when the sender needs to
advance the ratchet. Generates a new DH keypair , performs
DH with the peer's public key , and derives new root and
sending chain keys .
- `_dh_ratchet_receive` (lines 178-213): Called when a received message
contains a new DH public key. Updates the peer public key at line 189,
- `_dh_ratchet_receive` : Called when a received message
contains a new DH public key. Updates the peer public key ,
performs DH with the existing private key to derive a new receiving
chain key at lines 195-198, then generates a new private key at line
chain key , then generates a new private key at line
200 and performs another DH to derive a new sending chain key at lines
208-211. This double DH step on the receiver side ensures both
receiving and sending chains are updated.
@ -644,9 +643,9 @@ Internet messages can arrive out of order. If Alice sends messages 1, 2,
4. Derive mk3 and decrypt message 3
The skipped message key mechanism handles this. Reference:
`double_ratchet.py:215-277`.
`frontend/src/crypto/double-ratchet.ts`.
`_store_skipped_message_keys` (lines 215-244) is called when the
`_store_skipped_message_keys` is called when the
received message number is greater than the expected message number. It
iterates through the gap, deriving and caching each skipped message key:
@ -663,23 +662,23 @@ message_number)`. This tuple key is necessary because message numbers
reset with each DH ratchet step: message 0 under DH key A1 is different
from message 0 under DH key A2.
`_try_skipped_message_key` (lines 260-277) checks whether a received
`_try_skipped_message_key` checks whether a received
message matches a previously cached skipped key. If it does, the cached
key is used for decryption and then removed from the cache (it is
consumed by `dict.pop()` at line 270).
consumed by `dict.pop` ).
Security limits prevent abuse. An attacker who sends messages with
enormous message numbers could force the ratchet to derive and store
millions of keys, exhausting memory. Two limits are enforced:
- `MAX_SKIP_MESSAGE_KEYS = 1000` (`config.py:73`): No more than 1000
- `MAX_SKIP_MESSAGE_KEYS = 1000` (`config.py`): No more than 1000
message keys can be skipped in a single gap. If a message arrives
claiming to be message number 5000 when we expect message 0, the
decryption is rejected at lines 226-230.
decryption is rejected .
- `MAX_CACHED_MESSAGE_KEYS = 2000` (`config.py:74`): The total number
- `MAX_CACHED_MESSAGE_KEYS = 2000` (`config.py`): The total number
of cached skipped keys across all ratchet epochs. If the cache is
full, the oldest keys are evicted at lines 232-234 via
full, the oldest keys are evicted via
`_evict_oldest_skipped_keys`.
### Forward Secrecy Proof
@ -759,23 +758,23 @@ The encryption flow is:
5. The output is ciphertext + a 16-byte authentication tag (GCM appends
the tag to the ciphertext)
Backend implementation at `double_ratchet.py:111-130`:
Backend implementation at `frontend/src/crypto/double-ratchet.ts`:
```python
def _encrypt_with_message_key(self, message_key, plaintext, associated_data):
aesgcm = AESGCM(message_key)
nonce = os.urandom(AES_GCM_NONCE_SIZE) # 12 bytes from config.py:69
nonce = os.urandom(AES_GCM_NONCE_SIZE) # 12 bytes from config.py
ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data)
return nonce, ciphertext
```
Backend decryption at `double_ratchet.py:132-153` catches `InvalidTag`
exceptions (line 151), which indicate that the ciphertext was tampered
Backend decryption at `frontend/src/crypto/double-ratchet.ts` catches `InvalidTag`
exceptions , which indicate that the ciphertext was tampered
with, the wrong key was used, or the associated data does not match.
The error is re-raised as `ValueError("Message tampered or corrupted")`
at line 153.
.
Frontend implementation at `primitives.ts:224-259` uses the WebCrypto
Frontend implementation at `frontend/src/crypto/primitives.ts` uses the WebCrypto
API:
```typescript
@ -791,7 +790,7 @@ const ciphertext = await subtle.encrypt(
)
```
Frontend decryption at `primitives.ts:261-292` mirrors this with
Frontend decryption at `frontend/src/crypto/primitives.ts` mirrors this with
`subtle.decrypt`. The WebCrypto API throws a `DOMException` if
authentication fails, which is functionally equivalent to the Python
`InvalidTag` exception.
@ -817,12 +816,12 @@ security reason to prefer CBC over GCM for new implementations.
### Parameters
As defined in `config.py:68-70`:
As defined in `config.py`:
```
AES_GCM_KEY_SIZE = 32 (256 bits) line 68
AES_GCM_NONCE_SIZE = 12 (96 bits) line 69
HKDF_OUTPUT_SIZE = 32 (256 bits) line 70
AES_GCM_KEY_SIZE = 32 (256 bits)
AES_GCM_NONCE_SIZE = 12 (96 bits)
HKDF_OUTPUT_SIZE = 32 (256 bits)
```
The 12-byte (96-bit) nonce is the recommended size for GCM. Longer
@ -941,16 +940,16 @@ Step 3: Client sends attestation for verification
- Private key STAYS on authenticator -- server never sees it
```
Implementation reference: `passkey_manager.py:55-94`
(`generate_registration_options`). At line 65, a 32-byte challenge is
Implementation reference: `backend/app/core/passkey/passkey_manager.py`
(`generate_registration_options`). a 32-byte challenge is
generated: `challenge = secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)`.
At lines 74-87, the WebAuthn options are constructed with RP
configuration, user information, and authenticator requirements. The
authenticator selection at lines 82-85 specifies
authenticator selection specifies
`ResidentKeyRequirement.REQUIRED`, which forces creation of a
discoverable credential (passkey).
Registration verification at `passkey_manager.py:96-130`
Registration verification at `backend/app/core/passkey/passkey_manager.py`
(`verify_registration`) calls `verify_registration_response` at lines
105-110, which validates the attestation object, checks the challenge,
verifies the RP ID, and confirms the origin.
@ -987,12 +986,12 @@ Step 3: Client sends assertion for verification
- Returns authenticated session
```
Implementation reference: `passkey_manager.py:132-160`
(`generate_authentication_options`). At line 139, a fresh challenge is
Implementation reference: `backend/app/core/passkey/passkey_manager.py`
(`generate_authentication_options`). a fresh challenge is
generated. At lines 148-153, WebAuthn authentication options are
constructed.
Authentication verification at `passkey_manager.py:162-207`
Authentication verification at `backend/app/core/passkey/passkey_manager.py`
(`verify_authentication`). At lines 173-180, the assertion is verified
against the expected challenge, RP ID, origin, and stored credential
public key. The critical clone detection check follows.
@ -1015,7 +1014,7 @@ increased (or has decreased), it indicates one of two things:
Both scenarios are security incidents that warrant blocking
authentication and alerting the user.
Reference: `passkey_manager.py:184-193`:
Reference: `backend/app/core/passkey/passkey_manager.py`:
```python
if (credential_current_sign_count != 0 and new_sign_count != 0
@ -1043,7 +1042,7 @@ support signature counters.
## Constant-Time Comparison
A detail worth calling out: `primitives.ts:388-397` implements a
A detail worth calling out: `frontend/src/crypto/primitives.ts` implements a
constant-time byte array comparison:
```typescript
@ -1141,11 +1140,11 @@ The layers interact in a strict top-down sequence for new conversations:
2. X3DH establishes a shared secret with the peer (even if the peer is
offline)
3. The shared secret initializes the Double Ratchet
(`double_ratchet.py:279-302`)
(`frontend/src/crypto/double-ratchet.ts`)
4. Each message is encrypted with a unique AES-256-GCM key derived from
the ratchet (`double_ratchet.py:323-362`)
the ratchet (`frontend/src/crypto/double-ratchet.ts`)
5. The encrypted message is transmitted via WebSocket and stored in
SurrealDB (`message_service.py:269-314`)
SurrealDB (`backend/app/services/message_service.py`)
For ongoing conversations, only steps 4 and 5 repeat. The X3DH
handshake happens once per conversation. The Double Ratchet then runs
@ -1224,11 +1223,11 @@ well-tested implementations.
112 bits for use through 2030+ (NIST SP 800-57 Part 1).
**CWE-330: Use of Insufficiently Random Values** -- Randomness comes
from two sources: `os.urandom()` on the backend (which reads from the
from two sources: `os.urandom` on the backend (which reads from the
operating system's CSPRNG -- `/dev/urandom` on Linux) and
`crypto.getRandomValues()` on the frontend (which uses the browser's
`crypto.getRandomValues` on the frontend (which uses the browser's
CSPRNG). Both are cryptographically secure. Nonce generation at
`double_ratchet.py:122` and `primitives.ts:294-297` use these
`frontend/src/crypto/double-ratchet.ts` and `frontend/src/crypto/primitives.ts` use these
exclusively.
**CWE-311: Missing Encryption of Sensitive Data** -- All message content
@ -1268,20 +1267,20 @@ cryptography textbook.
server never generates or holds encryption keys for message content.
Key generation happens in two places:
1. On the backend, `x3dh_manager.py:60-86` generates X25519 identity
keypairs using `X25519PrivateKey.generate()`, which calls into
1. On the backend, `frontend/src/crypto/x3dh.ts` generates X25519 identity
keypairs using `X25519PrivateKey.generate`, which calls into
OpenSSL's random number generator. These keys are for the X3DH
protocol, and the private keys are stored in the database for the
server-side key exchange path.
2. On the frontend, `primitives.ts:15-24` generates X25519 keypairs
2. On the frontend, `frontend/src/crypto/primitives.ts` generates X25519 keypairs
using the WebCrypto API (`subtle.generateKey`), which uses the
browser's hardware-backed CSPRNG. In the client-side encryption
model, these keys never leave the browser.
The `store_encrypted_message` function (`message_service.py:269-314`)
The `store_encrypted_message` function (`backend/app/services/message_service.py`)
receives pre-encrypted ciphertext from the client and stores it directly
in SurrealDB at lines 292-302 without any server-side decryption. The
in SurrealDB without any server-side decryption. The
server's role is explicitly that of a blind relay. Even if the entire
server infrastructure were compromised, the attacker would obtain only
encrypted blobs with no corresponding decryption keys.
@ -1302,7 +1301,7 @@ Ratchet combination used in this project. Each message gets a unique
AES key through the ratchet mechanism, providing forward secrecy across
billions of daily messages. WhatsApp's implementation stores prekey
bundles on their servers (just as this project does via
`prekey_service.py:293-361`), allowing asynchronous session
`backend/app/services/prekey_service.py`), allowing asynchronous session
establishment. The X3DH handshake is performed when a user initiates a
new conversation, and the Double Ratchet runs continuously thereafter.
@ -1315,12 +1314,12 @@ same situation occurred with FBI requests in the United States and
government demands in India and the UK.
**Connection to this project.** The X3DH implementation at
`x3dh_manager.py:208-281` and the Double Ratchet at
`double_ratchet.py:279-416` implement the same cryptographic operations
`frontend/src/crypto/x3dh.ts` and the Double Ratchet at
`frontend/src/crypto/double-ratchet.ts` implement the same cryptographic operations
described in the Signal Protocol specification. The same four DH
operations (lines 241-252), the same HKDF derivation (lines 257-264),
the same KDF chain operations (lines 79-109 of double_ratchet.py), and
the same skipped message key mechanism (lines 215-277). The protocol
operations , the same HKDF derivation ,
the same KDF chain operations (lines 79-109 of frontend/src/crypto/double-ratchet.ts), and
the same skipped message key mechanism . The protocol
specifications are public, the formal security proofs are published, and
the implementation follows them directly.
@ -1351,7 +1350,7 @@ authentication, even when the passwords are used to derive encryption
keys.
In this project's WebAuthn implementation
(`passkey_manager.py:55-94`), there is no password. The user
(`backend/app/core/passkey/passkey_manager.py`), there is no password. The user
authenticates with a biometric or PIN on their authenticator device. The
authenticator holds an ECDSA or EdDSA private key that is hardware-bound
and never exported. If the server database is fully compromised, the

523
PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md
View File

File diff suppressed because it is too large Load Diff

318
PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md
View File

@ -15,26 +15,22 @@ backend/
rooms.py # Chat room creation and listing
websocket.py # WebSocket endpoint (receives raw frames)
core/
encryption/
x3dh_manager.py # X3DH key exchange: generate, sign, exchange
double_ratchet.py # Double Ratchet: KDF chains, encrypt, decrypt
passkey/
passkey_manager.py # WebAuthn: registration, auth, clone detection
dependencies.py # current_user, issue_session, revoke_session
websocket_manager.py # Connection pool, heartbeats, live query sub
surreal_manager.py # SurrealDB client (messages, live queries)
redis_manager.py # Redis client (challenges, presence)
redis_manager.py # Redis client (sessions, challenges)
exceptions.py # Custom exception hierarchy
exception_handlers.py # FastAPI exception-to-HTTP mappings
enums.py # Presence status enum
models/
Base.py # SQLModel engine, async session maker
User.py # User table (id, username, display_name)
Credential.py # WebAuthn credential storage
IdentityKey.py # X25519 + Ed25519 identity keypairs
SignedPrekey.py # Signed prekeys with rotation/expiry
OneTimePrekey.py # Single-use one-time prekeys
RatchetState.py # Serialized Double Ratchet state per conversation
SkippedMessageKey.py # Persisted skipped message keys
User.py # User table (id, username, webauthn_user_handle, ...)
Credential.py # WebAuthn credential storage (BE/BS, sign_count)
IdentityKey.py # X25519 + Ed25519 identity public keys (no privates)
SignedPrekey.py # Signed prekey publics + signature
OneTimePrekey.py # Single-use prekey publics (is_used flag)
schemas/
auth.py # Pydantic schemas for registration/auth flows
websocket.py # Pydantic schemas for WS message types
@ -42,11 +38,11 @@ backend/
rooms.py # Pydantic schemas for room operations
common.py # Shared response schemas
services/
auth_service.py # User creation, credential management
prekey_service.py # Key initialization, rotation, bundle retrieval
message_service.py # Conversation init, encrypt, store, decrypt
websocket_service.py # WS message routing (type dispatch)
presence_service.py # Online/offline/away status via Redis
auth_service.py # User creation, credential management, search
prekey_service.py # Public-only prekey storage + bundle lookup
message_service.py # Pass-through ciphertext writer
websocket_service.py # WS routing + per-user rate cap + membership
presence_service.py # Online/offline/away status via SurrealDB
config.py # All constants + Settings (env vars)
factory.py # FastAPI app factory with middleware
main.py # Uvicorn entry point
@ -111,17 +107,17 @@ X3DH (Extended Triple Diffie-Hellman) solves a specific problem: Alice wants to
### Step 1: Key Generation
The foundation of X3DH is generating two types of identity keypairs per user. Here is the X25519 identity key generation from `x3dh_manager.py:60-86`:
The foundation of X3DH is generating two types of identity keypairs per user. Here is the X25519 identity key generation from `frontend/src/crypto/x3dh.ts`:
```python
def generate_identity_keypair_x25519(self) -> tuple[str, str]:
private_key = X25519PrivateKey.generate()
public_key = private_key.public_key()
private_key = X25519PrivateKey.generate
public_key = private_key.public_key
private_bytes = private_key.private_bytes(
encoding = serialization.Encoding.Raw,
format = serialization.PrivateFormat.Raw,
encryption_algorithm = serialization.NoEncryption()
encryption_algorithm = serialization.NoEncryption
)
public_bytes = public_key.public_bytes(
encoding = serialization.Encoding.Raw,
@ -134,7 +130,7 @@ def generate_identity_keypair_x25519(self) -> tuple[str, str]:
)
```
There is also `generate_identity_keypair_ed25519` at `x3dh_manager.py:88-114`, which follows the exact same pattern but uses `Ed25519PrivateKey.generate()` instead.
There is also `generate_identity_keypair_ed25519` at `frontend/src/crypto/x3dh.ts`, which follows the exact same pattern but uses `Ed25519PrivateKey.generate` instead.
**Why Raw encoding?** PEM and DER formats include metadata headers, algorithm identifiers, and ASN.1 structure. That is wasted bytes when you already know the algorithm on both sides. Raw encoding gives you exactly 32 bytes for X25519 and 32 bytes for Ed25519. That is the minimum footprint, and it maps directly to what the WebCrypto API expects on the frontend when importing with `importKey("raw", ...)`.
@ -144,20 +140,20 @@ There is also `generate_identity_keypair_ed25519` at `x3dh_manager.py:88-114`, w
### Step 2: Signed Prekey Creation
The Signed Prekey (SPK) is a semi-static X25519 key that gets rotated every 48 hours. Here is `generate_signed_prekey` from `x3dh_manager.py:116-152`:
The Signed Prekey (SPK) is a semi-static X25519 key that gets rotated every 48 hours. Here is `generate_signed_prekey` from `frontend/src/crypto/x3dh.ts`:
```python
def generate_signed_prekey(self,
identity_private_key_ed25519: str) -> tuple[str,
str,
str]:
spk_private = X25519PrivateKey.generate()
spk_public = spk_private.public_key()
spk_private = X25519PrivateKey.generate
spk_public = spk_private.public_key
spk_private_bytes = spk_private.private_bytes(
encoding = serialization.Encoding.Raw,
format = serialization.PrivateFormat.Raw,
encryption_algorithm = serialization.NoEncryption()
encryption_algorithm = serialization.NoEncryption
)
spk_public_bytes = spk_public.public_bytes(
encoding = serialization.Encoding.Raw,
@ -182,11 +178,11 @@ def generate_signed_prekey(self,
**What does the signature prove?** It proves exactly one thing: "The entity controlling the Ed25519 private key corresponding to Bob's published Ed25519 public key chose to sign this specific SPK public key." That binds the SPK to Bob's identity.
**Why not self-sign with X25519?** X25519 is a DH function, not a signature scheme. There is no `sign()` method on an X25519 key. You could theoretically convert between Curve25519 and Ed25519 key formats (they share the same underlying curve), but doing that conversion is a footgun. It is error-prone, not all libraries support it, and it violates the principle of key separation. Using a dedicated Ed25519 signing key is simpler and safer.
**Why not self-sign with X25519?** X25519 is a DH function, not a signature scheme. There is no `sign` method on an X25519 key. You could theoretically convert between Curve25519 and Ed25519 key formats (they share the same underlying curve), but doing that conversion is a footgun. It is error-prone, not all libraries support it, and it violates the principle of key separation. Using a dedicated Ed25519 signing key is simpler and safer.
### Step 3: Prekey Bundle Assembly
When Alice wants to message Bob, the server assembles Bob's prekey bundle. This happens in `prekey_service.py:293-361`:
When Alice wants to message Bob, the server assembles Bob's prekey bundle. This happens in `backend/app/services/prekey_service.py`:
```python
async def get_prekey_bundle(
@ -201,7 +197,7 @@ async def get_prekey_bundle(
spk_statement = select(SignedPrekey).where(
SignedPrekey.user_id == user_id,
SignedPrekey.is_active
).order_by(SignedPrekey.created_at.desc())
).order_by(SignedPrekey.created_at.desc)
...
if not signed_prekey:
signed_prekey = await self.rotate_signed_prekey(session, user_id)
@ -217,15 +213,15 @@ async def get_prekey_bundle(
...
```
**Why auto-rotate?** If Bob's signed prekey expired and there are no active ones, the bundle fetch would fail. Auto-rotation at `prekey_service.py:316-321` ensures there is always an active SPK. The rotation generates a new X25519 keypair, signs it with Bob's Ed25519 identity key, marks old SPKs as inactive, and sets a 48-hour expiry (`SIGNED_PREKEY_ROTATION_HOURS = 48` in `config.py:76`).
**Why auto-rotate?** If Bob's signed prekey expired and there are no active ones, the bundle fetch would fail. Auto-rotation at `backend/app/services/prekey_service.py` ensures there is always an active SPK. The rotation generates a new X25519 keypair, signs it with Bob's Ed25519 identity key, marks old SPKs as inactive, and sets a 48-hour expiry (`SIGNED_PREKEY_ROTATION_HOURS = 48` in `config.py`).
**Why mark OPK as used?** One-Time Prekeys provide forward secrecy for the initial handshake. Each OPK is consumed exactly once. When Alice fetches Bob's bundle, the server marks that OPK as `is_used = True` and commits the change. If a second person also tries to start a conversation with Bob around the same time, they get a different OPK (or none at all). This single-use guarantee means that even if Bob's long-term keys are later compromised, an attacker cannot retroactively compute the shared secret for conversations that used an OPK, because the OPK private key was consumed and the DH output from it cannot be reconstructed from the other keys alone.
**What happens if no OPK is available?** The protocol still works. The `PreKeyBundle` dataclass has `one_time_prekey: str | None = None`. When the bundle is assembled without an OPK, the X3DH sender side simply computes `dh1 + dh2 + dh3` instead of `dh1 + dh2 + dh3 + dh4`. This gives slightly weaker forward secrecy for the initial handshake (it relies on the signed prekey not being compromised), but the Double Ratchet provides forward secrecy for all subsequent messages regardless. The system compensates by calling `replenish_one_time_prekeys` periodically to maintain a pool of 100 OPKs (`DEFAULT_ONE_TIME_PREKEY_COUNT = 100` in `config.py:75`).
**What happens if no OPK is available?** The protocol still works. The `PreKeyBundle` dataclass has `one_time_prekey: str | None = None`. When the bundle is assembled without an OPK, the X3DH sender side simply computes `dh1 + dh2 + dh3` instead of `dh1 + dh2 + dh3 + dh4`. This gives slightly weaker forward secrecy for the initial handshake (it relies on the signed prekey not being compromised), but the Double Ratchet provides forward secrecy for all subsequent messages regardless. The system compensates by calling `replenish_one_time_prekeys` periodically to maintain a pool of 100 OPKs (`DEFAULT_ONE_TIME_PREKEY_COUNT = 100` in `config.py`).
### Step 4: X3DH Sender Side
This is the core of the initial key agreement. Here is the DH operations from `x3dh_manager.py:241-264`:
This is the core of the initial key agreement. Here is the DH operations from `frontend/src/crypto/x3dh.ts`:
```python
dh1 = alice_ik_private.exchange(bob_spk_public)
@ -244,7 +240,7 @@ else:
f = b'\xff' * X25519_KEY_SIZE
hkdf = HKDF(
algorithm = hashes.SHA256(),
algorithm = hashes.SHA256,
length = X25519_KEY_SIZE,
salt = b'\x00' * X25519_KEY_SIZE,
info = b'X3DH',
@ -252,7 +248,7 @@ hkdf = HKDF(
shared_key = hkdf.derive(f + key_material)
```
Before this code runs, the sender verifies the signed prekey signature at `x3dh_manager.py:217-220`. If verification fails, a `ValueError` is raised and the exchange aborts. This is the MITM protection.
Before this code runs, the sender verifies the signed prekey signature at `frontend/src/crypto/x3dh.ts`. If verification fails, a `ValueError` is raised and the exchange aborts. This is the MITM protection.
**Each DH operation explained:**
@ -267,11 +263,11 @@ Before this code runs, the sender verifies the signed prekey signature at `x3dh_
**Why info="X3DH"?** The `info` parameter in HKDF is a domain separation string. It ensures that the derived key is bound to this specific protocol. If the same key material were accidentally reused in a different context (say, a TLS handshake), the info string would produce a different derived key. This prevents cross-protocol attacks.
**Associated data: IK_A || IK_B.** After deriving the shared key, the code computes `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` at `x3dh_manager.py:272`. This associated data is later passed to AES-GCM as the AAD (Additional Authenticated Data). It binds the encrypted messages to the specific pair of identity keys that performed the handshake. If an attacker tries to redirect messages between different users, the AAD check will fail during decryption.
**Associated data: IK_A || IK_B.** After deriving the shared key, the code computes `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` at `frontend/src/crypto/x3dh.ts`. This associated data is later passed to AES-GCM as the AAD (Additional Authenticated Data). It binds the encrypted messages to the specific pair of identity keys that performed the handshake. If an attacker tries to redirect messages between different users, the AAD check will fail during decryption.
### Step 5: X3DH Receiver Side
The receiver-side logic lives at `x3dh_manager.py:283-350`. The operations are mirrored:
The receiver-side logic lives at `frontend/src/crypto/x3dh.ts`. The operations are mirrored:
```python
dh1 = bob_spk_private.exchange(alice_ik_public) # mirrors alice_ik.exchange(bob_spk)
@ -282,7 +278,7 @@ dh4 = bob_opk_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(
This works because of how Diffie-Hellman is defined: `A_private.exchange(B_public) == B_private.exchange(A_public)`. The math is commutative. Both sides compute the same four 32-byte DH outputs, concatenate them in the same order, and derive the same shared key through the same HKDF parameters (zero salt, `X3DH` info, `0xFF` padding).
The receiver also constructs `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` in the same order (Alice's IK first, Bob's IK second) at `x3dh_manager.py:341`. Order matters. If Alice put hers first on the sender side but Bob reversed the order on the receiver side, the associated data would not match and decryption would fail.
The receiver also constructs `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` in the same order (Alice's IK first, Bob's IK second) at `frontend/src/crypto/x3dh.ts`. Order matters. If Alice put hers first on the sender side but Bob reversed the order on the receiver side, the associated data would not match and decryption would fail.
### Common Mistakes
@ -291,7 +287,7 @@ The receiver also constructs `associated_data = alice_ik_public_bytes + bob_ik_p
```python
# WRONG - breaks compatibility between sender and receiver
salt = os.urandom(32)
hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=salt, info=b'X3DH')
hkdf = HKDF(algorithm=hashes.SHA256, length=32, salt=salt, info=b'X3DH')
```
If you use a random salt, the sender and receiver will derive different shared keys. The sender would need to transmit the salt alongside the ephemeral key, and at that point you have added complexity for no security gain since the DH outputs already provide the randomness.
@ -306,7 +302,7 @@ def perform_x3dh_sender(self, alice_identity_private_x25519, bob_bundle, ...):
...
```
Without signature verification, an attacker who controls the network can replace Bob's SPK public key with their own. Alice would compute a shared secret with the attacker. The attacker then performs X3DH with Bob using Bob's real SPK, and relays messages between them. Neither Alice nor Bob detects the interception. The fix is on `x3dh_manager.py:217-220`: verify before exchanging.
Without signature verification, an attacker who controls the network can replace Bob's SPK public key with their own. Alice would compute a shared secret with the attacker. The attacker then performs X3DH with Bob using Bob's real SPK, and relays messages between them. Neither Alice nor Bob detects the interception. The fix is on `frontend/src/crypto/x3dh.ts`: verify before exchanging.
**BAD: Reusing one-time prekeys**
@ -317,11 +313,11 @@ opk = get_opk_for_user(user_id)
# another conversation uses the same OPK
```
If two conversations use the same OPK, an attacker who later compromises Bob's OPK private key can compute the DH4 output for both conversations. That defeats the purpose of single-use keys. The fix is at `prekey_service.py:332`: `one_time_prekey.is_used = True` immediately upon retrieval, followed by `session.commit()`.
If two conversations use the same OPK, an attacker who later compromises Bob's OPK private key can compute the DH4 output for both conversations. That defeats the purpose of single-use keys. The fix is at `backend/app/services/prekey_service.py`: `one_time_prekey.is_used = True` immediately upon retrieval, followed by `session.commit`.
### Frontend X3DH Implementation
The frontend mirrors the backend X3DH in TypeScript using the WebCrypto API. The client-side sender logic lives in `x3dh.ts:115-173`:
The frontend mirrors the backend X3DH in TypeScript using the WebCrypto API. The client-side sender logic lives in `x3dh.ts`:
```typescript
export async function initiateX3DH(
@ -344,7 +340,7 @@ export async function initiateX3DH(
const sharedKey = await hkdfDerive(concatenated, EMPTY_SALT, X3DH_INFO, 32)
```
The HKDF derivation used here is from `primitives.ts:166-192`:
The HKDF derivation used here is from `frontend/src/crypto/primitives.ts`:
```typescript
export async function hkdfDerive(
@ -371,7 +367,7 @@ There is an important difference between the backend and frontend HKDF calls. Th
The WebCrypto API's `subtle.deriveBits` with the `HKDF` algorithm handles the extract-then-expand steps internally. You provide the input key material, a salt, an info string, and the desired output length in bits (not bytes, which is why `outputLength * 8` appears). The `importKey` call with `{ name: "HKDF" }` creates a non-extractable key object suitable only for derivation, which prevents the raw key material from being accidentally exported or misused.
The receiver-side X3DH in `x3dh.ts:175-215` mirrors the sender with the same key swaps as the Python version. The commutative property of ECDH ensures both sides derive the same 32-byte shared key.
The receiver-side X3DH in `x3dh.ts` mirrors the sender with the same key swaps as the Python version. The commutative property of ECDH ensures both sides derive the same 32-byte shared key.
---
@ -381,58 +377,60 @@ The Double Ratchet provides ongoing forward secrecy and break-in recovery after
### Initialization
The sender initializes at `double_ratchet.py:279-302`:
The sender initializes via `initializeRatchetSender` in `frontend/src/crypto/double-ratchet.ts`:
```python
def initialize_sender(
self,
shared_key: bytes,
peer_public_key: bytes
) -> DoubleRatchetState:
dh_private = X25519PrivateKey.generate()
peer_public = X25519PublicKey.from_public_bytes(peer_public_key)
dh_output = dh_private.exchange(peer_public)
```ts
export async function initializeRatchetSender(
peerId: string,
sharedKey: Uint8Array,
peerPublicKey: Uint8Array,
): Promise<DoubleRatchetState> {
const dhKeyPair = await generateX25519KeyPair
const dhPublicKey = await exportPublicKey(dhKeyPair.publicKey)
const peerKey = await importX25519PublicKey(peerPublicKey)
const dhOutput = await x25519DeriveSharedSecret(dhKeyPair.privateKey, peerKey)
root_key, sending_chain_key = self._kdf_rk(shared_key, dh_output)
// KDF_RK per spec: HKDF(salt = previous_root_key, IKM = dh_output, info = "DoubleRatchet")
const derivedKeys = await hkdfDerive(dhOutput, sharedKey, RATCHET_INFO, 64)
const rootKey = derivedKeys.slice(0, 32)
const sendingChainKey = derivedKeys.slice(32, 64)
state = DoubleRatchetState(
root_key = root_key,
sending_chain_key = sending_chain_key,
receiving_chain_key = b'\x00' * HKDF_OUTPUT_SIZE,
dh_private_key = dh_private,
dh_peer_public_key = peer_public_key
)
return state
return { peer_id: peerId, root_key: rootKey, sending_chain_key: sendingChainKey, /* ... */ }
}
```
The receiver initializes at `double_ratchet.py:304-321`:
The receiver initializes via `initializeRatchetReceiver`:
```python
def initialize_receiver(
self,
shared_key: bytes,
own_private_key: X25519PrivateKey
) -> DoubleRatchetState:
state = DoubleRatchetState(
root_key = shared_key,
sending_chain_key = b'\x00' * HKDF_OUTPUT_SIZE,
receiving_chain_key = b'\x00' * HKDF_OUTPUT_SIZE,
dh_private_key = own_private_key,
dh_peer_public_key = None
)
return state
```ts
export async function initializeRatchetReceiver(
peerId: string,
sharedKey: Uint8Array,
dhKeyPair: CryptoKeyPair,
): Promise<DoubleRatchetState> {
const dhPublicKey = await exportPublicKey(dhKeyPair.publicKey)
return {
peer_id: peerId,
root_key: sharedKey,
sending_chain_key: new Uint8Array(0),
receiving_chain_key: null,
dh_private_key: dhKeyPair,
dh_public_key: dhPublicKey,
dh_peer_public_key: null,
/* ... */
}
}
```
**Why does the sender do an extra DH step?** The sender generates a fresh DH keypair and immediately performs a DH exchange with Bob's SPK public key. This produces a `dh_output` which gets fed into `_kdf_rk` along with the X3DH `shared_key` to derive the first `root_key` and `sending_chain_key`. This extra step means the first message already has a DH ratchet contribution beyond the X3DH output. The receiver, by contrast, does not perform this step during initialization. It simply stores the X3DH shared key as the root key and waits for the first message, which will carry the sender's new DH public key and trigger the receiver's first ratchet step.
### KDF Chain Operations
The root key chain derivation at `double_ratchet.py:79-94`:
The root key chain derivation at `frontend/src/crypto/double-ratchet.ts`:
```python
def _kdf_rk(self, root_key: bytes, dh_output: bytes) -> tuple[bytes, bytes]:
hkdf = HKDF(
algorithm = hashes.SHA256(),
algorithm = hashes.SHA256,
length = HKDF_OUTPUT_SIZE * 2,
salt = root_key,
info = b'',
@ -447,30 +445,30 @@ def _kdf_rk(self, root_key: bytes, dh_output: bytes) -> tuple[bytes, bytes]:
The empty `info` parameter is intentional. The Signal spec does not use an info string for the root chain KDF. Domain separation happens implicitly because the root chain HKDF always uses the root key as salt, whereas the X3DH HKDF uses a zero salt. Different salts produce different outputs even with the same input, so there is no cross-contamination.
The chain key derivation at `double_ratchet.py:96-109`:
The chain key derivation at `frontend/src/crypto/double-ratchet.ts`:
```python
def _kdf_ck(self, chain_key: bytes) -> tuple[bytes, bytes]:
h_chain = hmac.HMAC(chain_key, hashes.SHA256())
h_chain = hmac.HMAC(chain_key, hashes.SHA256)
h_chain.update(b'\x01')
next_chain_key = h_chain.finalize()
next_chain_key = h_chain.finalize
h_message = hmac.HMAC(chain_key, hashes.SHA256())
h_message = hmac.HMAC(chain_key, hashes.SHA256)
h_message.update(b'\x02')
message_key = h_message.finalize()
message_key = h_message.finalize
return next_chain_key, message_key
```
**Why HMAC with 0x01 and 0x02?** The chain key needs to produce two outputs: the next chain key and the message key. Using HMAC with different single-byte constants is the simplest way to derive two independent values from one input. `0x01` produces the next chain key, `0x02` produces the message key. The constants are arbitrary but fixed by convention (the Signal spec uses these exact values). The important thing is that they are different, so the two HMAC outputs are cryptographically independent.
**Why two separate HMACs?** You cannot reuse an HMAC object after calling `finalize()` in the Python `cryptography` library. Each HMAC computation is a fresh instance. Even if you could reuse it, producing two different outputs from the same key requires two different inputs, which means two operations.
**Why two separate HMACs?** You cannot reuse an HMAC object after calling `finalize` in the Python `cryptography` library. Each HMAC computation is a fresh instance. Even if you could reuse it, producing two different outputs from the same key requires two different inputs, which means two operations.
**What is the one-way property?** Given `next_chain_key = HMAC(chain_key, 0x01)`, you cannot compute `chain_key` from `next_chain_key`. HMAC is a one-way function. This means that if an attacker obtains a message key for message N, they cannot derive the chain key that produced it, and therefore cannot derive message keys for messages 0 through N-1. This is forward secrecy within a single chain.
### Message Encryption
Here is the encrypt path from `double_ratchet.py:323-362`:
Here is the encrypt path from `frontend/src/crypto/double-ratchet.ts`:
```python
def encrypt_message(self, state, plaintext, associated_data):
@ -478,7 +476,7 @@ def encrypt_message(self, state, plaintext, associated_data):
nonce, ciphertext = self._encrypt_with_message_key(message_key, plaintext, associated_data)
if state.dh_private_key:
dh_public = state.dh_private_key.public_key()
dh_public = state.dh_private_key.public_key
dh_public_bytes = dh_public.public_bytes(
encoding = serialization.Encoding.Raw,
format = serialization.PublicFormat.Raw
@ -497,7 +495,7 @@ def encrypt_message(self, state, plaintext, associated_data):
return encrypted_msg
```
And the underlying AES-GCM encryption at `double_ratchet.py:111-130`:
And the underlying AES-GCM encryption at `frontend/src/crypto/double-ratchet.ts`:
```python
def _encrypt_with_message_key(self, message_key, plaintext, associated_data):
@ -513,7 +511,7 @@ def _encrypt_with_message_key(self, message_key, plaintext, associated_data):
### Message Decryption
The decrypt path at `double_ratchet.py:364-416` handles three distinct cases:
The decrypt path at `frontend/src/crypto/double-ratchet.ts` handles three distinct cases:
```python
def decrypt_message(self, state, encrypted_msg, associated_data):
@ -558,7 +556,7 @@ def decrypt_message(self, state, encrypted_msg, associated_data):
### Out-of-Order Handling
The skip mechanism at `double_ratchet.py:215-244`:
The skip mechanism at `frontend/src/crypto/double-ratchet.ts`:
```python
def _store_skipped_message_keys(self, state, until_message_number, dh_public_key):
@ -581,9 +579,9 @@ def _store_skipped_message_keys(self, state, until_message_number, dh_public_key
state.receiving_chain_key = chain_key
```
**MAX_SKIP = 1000** (`config.py:73`). If an attacker sends a message claiming `message_number = 999999999`, the receiver would need to compute 999999999 HMAC operations to derive all the intermediate keys. That is a denial-of-service attack. The limit of 1000 means the receiver will reject any message that would require skipping more than 1000 positions. In practice, even aggressive out-of-order delivery rarely exceeds a few dozen skipped messages.
**MAX_SKIP = 1000** (`config.py`). If an attacker sends a message claiming `message_number = 999999999`, the receiver would need to compute 999999999 HMAC operations to derive all the intermediate keys. That is a denial-of-service attack. The limit of 1000 means the receiver will reject any message that would require skipping more than 1000 positions. In practice, even aggressive out-of-order delivery rarely exceeds a few dozen skipped messages.
**MAX_CACHE = 2000** (`config.py:74`). Even with the skip limit, a determined attacker could slowly accumulate skipped keys by sending many small gaps. The cache limit prevents unbounded memory growth. When the cache is full, `_evict_oldest_skipped_keys` removes the oldest entries to make room. Oldest entries are the ones most likely to be genuinely lost (if a message has not arrived after 2000 other skipped keys were stored, it is probably never coming).
**MAX_CACHE = 2000** (`config.py`). Even with the skip limit, a determined attacker could slowly accumulate skipped keys by sending many small gaps. The cache limit prevents unbounded memory growth. When the cache is full, `_evict_oldest_skipped_keys` removes the oldest entries to make room. Oldest entries are the ones most likely to be genuinely lost (if a message has not arrived after 2000 other skipped keys were stored, it is probably never coming).
### Common Mistakes
@ -591,7 +589,7 @@ def _store_skipped_message_keys(self, state, until_message_number, dh_public_key
**Not evicting old skipped keys (memory exhaustion).** Without eviction, the `skipped_message_keys` dict grows without bound. An attacker can cause OOM by sending many messages with incrementing but non-sequential message numbers across multiple ratchet steps.
**Reusing nonces (catastrophic for AES-GCM).** AES-GCM uses a 12-byte nonce. If the same nonce is used twice with the same key, an attacker can XOR the two ciphertexts to eliminate the keystream and recover both plaintexts. The code uses `os.urandom(AES_GCM_NONCE_SIZE)` at `double_ratchet.py:122` for a fresh random nonce every time. Since each message key is used exactly once (each chain key advances after derivation), and each message key gets a random nonce, the probability of a collision is negligible. But if someone modified the code to reuse message keys, the random nonce would be the only defense, and with 12 bytes, birthday collisions become likely after about 2^48 messages under the same key.
**Reusing nonces (catastrophic for AES-GCM).** AES-GCM uses a 12-byte nonce. If the same nonce is used twice with the same key, an attacker can XOR the two ciphertexts to eliminate the keystream and recover both plaintexts. The code uses `os.urandom(AES_GCM_NONCE_SIZE)` at `frontend/src/crypto/double-ratchet.ts` for a fresh random nonce every time. Since each message key is used exactly once (each chain key advances after derivation), and each message key gets a random nonce, the probability of a collision is negligible. But if someone modified the code to reuse message keys, the random nonce would be the only defense, and with 12 bytes, birthday collisions become likely after about 2^48 messages under the same key.
---
@ -599,7 +597,7 @@ def _store_skipped_message_keys(self, state, until_message_number, dh_public_key
### Registration Flow
Registration is handled in `passkey_manager.py:55-94`. The server generates registration options:
Registration is handled in `backend/app/core/passkey/passkey_manager.py`. The server generates registration options:
```python
def generate_registration_options(
@ -624,21 +622,21 @@ def generate_registration_options(
**ResidentKeyRequirement.REQUIRED** means the authenticator must store the credential on the device itself (a "discoverable credential" or "passkey"). This allows usernameless login: the user just taps their security key or uses biometrics, and the authenticator presents the credential without the user typing a username. Without REQUIRED, the server would need to provide a credential ID hint during authentication, which means the user must provide their username first.
**Challenge generation** uses `secrets.token_bytes(32)` at `passkey_manager.py:65`. The `secrets` module is Python's CSPRNG interface, equivalent to `/dev/urandom`. The challenge must be unpredictable to prevent replay attacks. The challenge is stored in Redis with a TTL of 600 seconds (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600` in `config.py:84`), so it expires before an attacker could realistically capture and replay it.
**Challenge generation** uses `secrets.token_bytes(32)` at `backend/app/core/passkey/passkey_manager.py`. The `secrets` module is Python's CSPRNG interface, equivalent to `/dev/urandom`. The challenge must be unpredictable to prevent replay attacks. The challenge is stored in Redis with a TTL of 600 seconds (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600` in `config.py`), so it expires before an attacker could realistically capture and replay it.
**Attestation verification** at `passkey_manager.py:96-130` calls `verify_registration_response` from py_webauthn. This checks that the authenticator's response was correctly signed, that the RP ID matches, that the origin matches, and that the challenge matches. The `expected_origin = self.rp_origin` check at `passkey_manager.py:109` is critical: it ensures the registration happened from the legitimate frontend, not a phishing page on a different domain.
**Attestation verification** at `backend/app/core/passkey/passkey_manager.py` calls `verify_registration_response` from py_webauthn. This checks that the authenticator's response was correctly signed, that the RP ID matches, that the origin matches, and that the challenge matches. The `expected_origin = self.rp_origin` check at `backend/app/core/passkey/passkey_manager.py` is critical: it ensures the registration happened from the legitimate frontend, not a phishing page on a different domain.
### Authentication Flow
Authentication follows the same challenge-response pattern. The server generates options at `passkey_manager.py:132-160`, sends the challenge to the client, the client's authenticator signs the challenge with its private key, and the server verifies at `passkey_manager.py:162-207`.
Authentication follows the same challenge-response pattern. The server generates options at `backend/app/core/passkey/passkey_manager.py`, sends the challenge to the client, the client's authenticator signs the challenge with its private key, and the server verifies at `backend/app/core/passkey/passkey_manager.py`.
**Why challenge-response?** Without a fresh challenge, an attacker who intercepts one authentication response could replay it forever. The challenge ensures each authentication is unique. The authenticator signs the challenge (along with other data like the RP ID and origin) with its private key. The server verifies the signature using the stored public key. Even if the signed response is intercepted, it cannot be reused because the next authentication will have a different challenge.
**Why check origin?** The origin check (`expected_origin = self.rp_origin`) at `passkey_manager.py:177` prevents credential phishing. If an attacker sets up a fake site at `evil-chat.com` and tricks the user into authenticating, the authenticator will include `evil-chat.com` as the origin in the signed data. When the attacker tries to forward that signed response to the real server, the origin check fails because the server expects `http://localhost:3000` (or whatever the production origin is). The authenticator binds the signature to the origin, making phishing structurally impossible.
**Why check origin?** The origin check (`expected_origin = self.rp_origin`) at `backend/app/core/passkey/passkey_manager.py` prevents credential phishing. If an attacker sets up a fake site at `evil-chat.com` and tricks the user into authenticating, the authenticator will include `evil-chat.com` as the origin in the signed data. When the attacker tries to forward that signed response to the real server, the origin check fails because the server expects `http://localhost:3000` (or whatever the production origin is). The authenticator binds the signature to the origin, making phishing structurally impossible.
### Clone Detection
From `passkey_manager.py:184-193`:
From `backend/app/core/passkey/passkey_manager.py`:
```python
if (credential_current_sign_count != 0 and new_sign_count != 0
@ -665,16 +663,16 @@ if (credential_current_sign_count != 0 and new_sign_count != 0
### Connection Management
From `websocket_manager.py:43-95`:
From `websocket_manager.py`:
```python
async def connect(self, websocket, user_id):
await websocket.accept()
await websocket.accept
if user_id not in self.active_connections:
self.active_connections[user_id] = []
if len(self.active_connections[user_id]) >= WS_MAX_CONNECTIONS_PER_USER:
await self._send_error(websocket, "max_connections", ...)
await websocket.close()
await websocket.close
return False
self.active_connections[user_id].append(websocket)
await presence_service.set_user_online(user_id)
@ -685,13 +683,13 @@ async def connect(self, websocket, user_id):
**Why per-user connection lists?** A user might have the chat open on their phone and their laptop simultaneously. Each device creates its own WebSocket connection. The `active_connections` dict maps `UUID -> list[WebSocket]`, so when a message arrives for a user, it gets sent to all of their active connections. Without this, multi-device support would not work.
**Why max 5?** `WS_MAX_CONNECTIONS_PER_USER = 5` (from `config.py:142`). Without a limit, a malicious client could open thousands of WebSocket connections and exhaust server memory. Five is generous enough for legitimate multi-device usage (phone, tablet, laptop, desktop, maybe one more) while capping resource consumption. The specific number is configurable via environment variable.
**Why max 5?** `WS_MAX_CONNECTIONS_PER_USER = 5` (from `config.py`). Without a limit, a malicious client could open thousands of WebSocket connections and exhaust server memory. Five is generous enough for legitimate multi-device usage (phone, tablet, laptop, desktop, maybe one more) while capping resource consumption. The specific number is configurable via environment variable.
**Why heartbeat?** WebSocket connections can silently die. A user closes their laptop lid, their WiFi drops, or a NAT gateway times out the connection. The TCP stack might not detect the dead connection for minutes or hours. The heartbeat loop at `websocket_manager.py:177-201` sends a ping every `WS_HEARTBEAT_INTERVAL` (30 seconds by default). If the send fails, it means the connection is dead, and `disconnect` cleans it up. This keeps the `active_connections` dict accurate and ensures presence status reflects reality.
**Why heartbeat?** WebSocket connections can silently die. A user closes their laptop lid, their WiFi drops, or a NAT gateway times out the connection. The TCP stack might not detect the dead connection for minutes or hours. The heartbeat loop at `websocket_manager.py` sends a ping every `WS_HEARTBEAT_INTERVAL` (30 seconds by default). If the send fails, it means the connection is dead, and `disconnect` cleans it up. This keeps the `active_connections` dict accurate and ensures presence status reflects reality.
### Live Query Subscription
From `websocket_manager.py:203-223`:
From `websocket_manager.py`:
```python
async def _subscribe_to_messages(self, user_id):
@ -705,13 +703,13 @@ async def _subscribe_to_messages(self, user_id):
self.live_query_ids[user_id] = live_id
```
**SurrealDB pushes new messages to the server, server forwards to client. No polling needed.** SurrealDB has a native live query feature. When you register a live query like `LIVE SELECT * FROM messages WHERE recipient_id = $user_id`, SurrealDB pushes every new matching record to the callback in real time. The server does not need to poll the database on a timer. When a new message is created in SurrealDB (by `message_service.store_encrypted_message`), the live query fires, `_handle_live_message` at `websocket_manager.py:225-251` picks it up, wraps it in a WebSocket message schema, and sends it to all of that user's active connections.
**SurrealDB pushes new messages to the server, server forwards to client. No polling needed.** SurrealDB has a native live query feature. When you register a live query like `LIVE SELECT * FROM messages WHERE recipient_id = $user_id`, SurrealDB pushes every new matching record to the callback in real time. The server does not need to poll the database on a timer. When a new message is created in SurrealDB (by `message_service.store_encrypted_message`), the live query fires, `_handle_live_message` at `websocket_manager.py` picks it up, wraps it in a WebSocket message schema, and sends it to all of that user's active connections.
This is more efficient than polling because there is zero latency (the callback fires the instant the record is created) and zero wasted queries (no empty poll cycles when no messages are pending).
### Dead Connection Cleanup
The `disconnect` method at `websocket_manager.py:97-133` handles cleanup:
The `disconnect` method at `websocket_manager.py` handles cleanup:
```python
async def disconnect(self, websocket, user_id):
@ -726,7 +724,7 @@ async def disconnect(self, websocket, user_id):
# Cancel heartbeat task
```
Dead connections are also detected during `send_message` at `websocket_manager.py:135-153`. If sending to a WebSocket raises an exception, that WebSocket is added to a `dead_connections` list, and each dead connection is cleaned up via `disconnect` after the send loop completes. This lazy cleanup means the system self-heals: even if a heartbeat fails to detect a dead connection, the next message send attempt will catch it.
Dead connections are also detected during `send_message` at `websocket_manager.py`. If sending to a WebSocket raises an exception, that WebSocket is added to a `dead_connections` list, and each dead connection is cleaned up via `disconnect` after the send loop completes. This lazy cleanup means the system self-heals: even if a heartbeat fails to detect a dead connection, the next message send attempt will catch it.
The cleanup only sets the user offline and kills the live query when the last connection for that user is removed. If they still have another tab open, their other connection keeps working and they stay online.
@ -736,7 +734,7 @@ The cleanup only sets the user offline and kills the live query when the last co
### Server-Side Storage (Passthrough)
From `message_service.py:269-314`:
From `backend/app/services/message_service.py`:
```python
async def store_encrypted_message(
@ -762,7 +760,7 @@ This is the correct E2E approach. Many "encrypted" chat systems encrypt on the s
### Conversation Initialization
From `message_service.py:48-166`:
From `backend/app/services/message_service.py`:
```python
async def initialize_conversation(self, session, sender_id, recipient_id):
@ -770,7 +768,7 @@ async def initialize_conversation(self, session, sender_id, recipient_id):
# Fetch sender's identity key
# Fetch recipient's prekey bundle
# Perform X3DH sender-side
x3dh_result = x3dh_manager.perform_x3dh_sender(
x3dh_result = initiateX3DH (frontend/src/crypto/x3dh.ts)(
alice_identity_private_x25519 = sender_ik.private_key,
bob_bundle = recipient_bundle,
bob_identity_public_ed25519 = recipient_ik.public_key_ed25519
@ -785,7 +783,7 @@ async def initialize_conversation(self, session, sender_id, recipient_id):
This is the bridge between key exchange and ongoing encryption. The method orchestrates the entire flow: check if a conversation already exists (idempotency), fetch both users' keys from PostgreSQL, perform the X3DH exchange, initialize the Double Ratchet with the resulting shared key, serialize the ratchet state, and persist it. After this method completes, `encrypt_message` and `decrypt_message` can operate using the stored ratchet state.
The check at `message_service.py:60-73` is important: if a ratchet state already exists for this sender-recipient pair, the method returns it without doing anything. This prevents accidental re-initialization, which would generate a new shared secret and break the existing conversation.
The check at `backend/app/services/message_service.py` is important: if a ratchet state already exists for this sender-recipient pair, the method returns it without doing anything. This prevents accidental re-initialization, which would generate a new shared secret and break the existing conversation.
---
@ -793,7 +791,7 @@ The check at `message_service.py:60-73` is important: if a ratchet state already
### Constant-Time Comparison
From `primitives.ts:388-397`:
From `frontend/src/crypto/primitives.ts`:
```typescript
export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean {
@ -814,9 +812,9 @@ The length check at the top does leak whether the lengths are equal. This is gen
### Random Number Generation
On the Python backend, `os.urandom(AES_GCM_NONCE_SIZE)` at `double_ratchet.py:122` reads from the operating system's cryptographically secure random number generator (CSPRNG). On Linux, this is backed by `/dev/urandom`, which draws entropy from hardware events (disk timing, network interrupts, etc.) and maintains a CSPRNG state that is computationally indistinguishable from true randomness.
On the Python backend, `os.urandom(AES_GCM_NONCE_SIZE)` at `frontend/src/crypto/double-ratchet.ts` reads from the operating system's cryptographically secure random number generator (CSPRNG). On Linux, this is backed by `/dev/urandom`, which draws entropy from hardware events (disk timing, network interrupts, etc.) and maintains a CSPRNG state that is computationally indistinguishable from true randomness.
On the browser frontend, the `generateRandomBytes` function at `primitives.ts:294-298`:
On the browser frontend, the `generateRandomBytes` function at `frontend/src/crypto/primitives.ts`:
```typescript
export function generateRandomBytes(length: number): Uint8Array {
@ -828,21 +826,21 @@ export function generateRandomBytes(length: number): Uint8Array {
This calls the WebCrypto API's CSPRNG. Every major browser implements this using the OS CSPRNG under the hood. Chrome uses BoringSSL's CSPRNG, Firefox uses NSS, Safari uses CommonCrypto. All ultimately draw from the kernel's entropy pool.
The same function is used for nonce generation in `aesGcmEncrypt` at `primitives.ts:243`: `const nonce = generateRandomBytes(AES_GCM_NONCE_SIZE)`. It is also used for WebAuthn challenge generation on the server side via `secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)` at `passkey_manager.py:65`, which is Python's wrapper around `os.urandom`.
The same function is used for nonce generation in `aesGcmEncrypt` at `frontend/src/crypto/primitives.ts`: `const nonce = generateRandomBytes(AES_GCM_NONCE_SIZE)`. It is also used for WebAuthn challenge generation on the server side via `secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)` at `backend/app/core/passkey/passkey_manager.py`, which is Python's wrapper around `os.urandom`.
**Why CSPRNG?** Nonces, challenges, and ephemeral keys must be unpredictable. If an attacker can predict the next nonce, they can precompute the keystream for AES-GCM and decrypt messages in real time. If they can predict the next ephemeral key, they can compute the DH shared secret. If they can predict the next WebAuthn challenge, they can pre-sign a response and replay it.
**Why not Math.random()?** `Math.random()` uses a PRNG (pseudorandom number generator) seeded from a low-entropy source (often just the current time or a simple counter). Most JavaScript engines use xorshift128+ or a similar fast PRNG. The internal state is typically 128 bits, and it can be reconstructed by observing as few as 4-5 consecutive outputs. An attacker who knows the internal state of `Math.random()` can predict every future output, which would compromise every nonce and every ephemeral key generated by the application. In 2015, researchers demonstrated recovering the full xorshift128+ state from V8's `Math.random()` by observing just a few outputs. `Math.random()` is fine for shuffling a deck of cards in a UI animation. It is catastrophic for cryptography.
**Why not Math.random?** `Math.random` uses a PRNG (pseudorandom number generator) seeded from a low-entropy source (often just the current time or a simple counter). Most JavaScript engines use xorshift128+ or a similar fast PRNG. The internal state is typically 128 bits, and it can be reconstructed by observing as few as 4-5 consecutive outputs. An attacker who knows the internal state of `Math.random` can predict every future output, which would compromise every nonce and every ephemeral key generated by the application. In 2015, researchers demonstrated recovering the full xorshift128+ state from V8's `Math.random` by observing just a few outputs. `Math.random` is fine for shuffling a deck of cards in a UI animation. It is catastrophic for cryptography.
### Key Serialization
All keys are serialized as base64url-encoded raw bytes. For example, in `x3dh_manager.py:67-85`:
All keys are serialized as base64url-encoded raw bytes. For example, in `frontend/src/crypto/x3dh.ts`:
```python
private_bytes = private_key.private_bytes(
encoding = serialization.Encoding.Raw,
format = serialization.PrivateFormat.Raw,
encryption_algorithm = serialization.NoEncryption()
encryption_algorithm = serialization.NoEncryption
)
...
return (
@ -863,21 +861,21 @@ Here is what happens when Alice types "Hello Bob" and it reaches Bob's screen. E
**1. Alice types "Hello Bob" in Chat.tsx.** The `ChatInput` component captures the text in a form submit handler. The handler creates a temporary message ID (a random string for optimistic UI updates) and calls into the message sending logic. The plaintext exists only in the browser's JavaScript heap at this point.
**2. crypto-service.ts `encrypt("bob-uuid", "Hello Bob")`.** The `CryptoService.encrypt` method at `crypto-service.ts:217-249` is invoked. It first checks whether a ratchet session exists with Bob.
**2. crypto-service.ts `encrypt("bob-uuid", "Hello Bob")`.** The `CryptoService.encrypt` method at `frontend/src/crypto/crypto-service.ts` is invoked. It first checks whether a ratchet session exists with Bob.
**3. Load ratchet state from IndexedDB.** `getRatchetState(peerId)` at `crypto-service.ts:303-315` checks the in-memory `Map<string, DoubleRatchetState>` first (keyed by peer ID). If the state is not in memory, it queries IndexedDB via `key-store.ts`, deserializes the stored JSON back into a `DoubleRatchetState` object (reconstructing CryptoKey objects from base64-encoded bytes), and caches it. If no state exists at all, `establishSession(peerId)` triggers the full X3DH flow: fetch Bob's prekey bundle from the server via `api.encryption.getPrekeyBundle`, perform client-side X3DH via `x3dh.ts:initiateX3DH`, and initialize the sending ratchet via `double-ratchet.ts:initializeRatchetSender`.
**3. Load ratchet state from IndexedDB.** `getRatchetState(peerId)` at `frontend/src/crypto/crypto-service.ts` checks the in-memory `Map<string, DoubleRatchetState>` first (keyed by peer ID). If the state is not in memory, it queries IndexedDB via `key-store.ts`, deserializes the stored JSON back into a `DoubleRatchetState` object (reconstructing CryptoKey objects from base64-encoded bytes), and caches it. If no state exists at all, `establishSession(peerId)` triggers the full X3DH flow: fetch Bob's prekey bundle from the server via `api.encryption.getPrekeyBundle`, perform client-side X3DH via `x3dh.ts:initiateX3DH`, and initialize the sending ratchet via `double-ratchet.ts:initializeRatchetSender`.
**4. double-ratchet.ts `encryptMessage`: advance chain key via HMAC.** At `double-ratchet.ts:167`, `deriveMessageKey(state.sending_chain_key)` computes two values:
**4. double-ratchet.ts `encryptMessage`: advance chain key via HMAC.** At `double-ratchet.ts`, `deriveMessageKey(state.sending_chain_key)` computes two values:
- `messageKey = HMAC-SHA-256(chainKey, "MessageKey")` - the key that will encrypt this specific message
- `nextChainKey = HMAC-SHA-256(chainKey, "ChainKey")` - the chain key for the next message
The old `sending_chain_key` is immediately overwritten with `nextChainKey`. The `messageKey` is used once and never stored. This is the forward secrecy mechanism: after encryption, the message key cannot be recovered from the updated chain state.
**5. primitives.ts `aesGcmEncrypt` with message key.** At `double-ratchet.ts:178`, the 32-byte `messageKey` is imported as an AES-GCM CryptoKey. The `plaintext` bytes (`TextEncoder.encode("Hello Bob")` = 9 bytes) are encrypted. A 12-byte random nonce is generated via `crypto.getRandomValues`. The message header (DH public key, message number, previous chain length) is serialized to JSON and used as Additional Authenticated Data (AAD). AES-GCM produces the ciphertext (9 bytes of encrypted data + 16 bytes of authentication tag = 25 bytes total).
**5. primitives.ts `aesGcmEncrypt` with message key.** At `double-ratchet.ts`, the 32-byte `messageKey` is imported as an AES-GCM CryptoKey. The `plaintext` bytes (`TextEncoder.encode("Hello Bob")` = 9 bytes) are encrypted. A 12-byte random nonce is generated via `crypto.getRandomValues`. The message header (DH public key, message number, previous chain length) is serialized to JSON and used as Additional Authenticated Data (AAD). AES-GCM produces the ciphertext (9 bytes of encrypted data + 16 bytes of authentication tag = 25 bytes total).
**6. Return {ciphertext, nonce, header}.** The `EncryptedMessage` object at `double-ratchet.ts:183-187` contains raw `Uint8Array` ciphertext and nonce, plus a typed `MessageHeader` object. Back in `crypto-service.ts:244-248`, the ciphertext and nonce are base64-encoded to strings, and the header (including any pending X3DH header for first messages) is JSON-stringified into a single string. The result is three strings: `ciphertext`, `nonce`, and `header`. All three are opaque to anyone without the message key.
**6. Return {ciphertext, nonce, header}.** The `EncryptedMessage` object at `double-ratchet.ts` contains raw `Uint8Array` ciphertext and nonce, plus a typed `MessageHeader` object. Back in `frontend/src/crypto/crypto-service.ts`, the ciphertext and nonce are base64-encoded to strings, and the header (including any pending X3DH header for first messages) is JSON-stringified into a single string. The result is three strings: `ciphertext`, `nonce`, and `header`. All three are opaque to anyone without the message key.
**7. WebSocket sends JSON payload.** `websocket-manager.ts:108-124` calls `sendEncryptedMessage`, which constructs the outgoing message:
**7. WebSocket sends JSON payload.** `websocket-manager.ts` calls `sendEncryptedMessage`, which constructs the outgoing message:
```json
{
@ -893,11 +891,11 @@ The old `sending_chain_key` is immediately overwritten with `nextChainKey`. The
This JSON is sent via `ws.send(JSON.stringify(message))`. If the WebSocket is not connected, the message is queued in `messageQueue` and sent when the connection is restored.
**8. websocket.py:46-54 receives.** The `websocket_endpoint` at `websocket.py:46` calls `data = await websocket.receive_text()`, parses the JSON with `json.loads(data)`, and passes the resulting dict to `websocket_service.route_message`. At this point the server has the encrypted blob. It cannot decrypt it. It does not have the message key, the chain key, or any ratchet state for Alice and Bob's conversation (in the E2E model).
**8. websocket.py receives.** The `websocket_endpoint` at `websocket.py` calls `data = await websocket.receive_text`, parses the JSON with `json.loads(data)`, and passes the resulting dict to `websocket_service.route_message`. At this point the server has the encrypted blob. It cannot decrypt it. It does not have the message key, the chain key, or any ratchet state for Alice and Bob's conversation (in the E2E model).
**9. websocket_service.py routes to handler.** At `websocket_service.py:63`, the message type `"encrypted_message"` dispatches to `handle_encrypted_message` at `websocket_service.py:87-179`. The handler extracts `recipient_id`, `room_id`, `ciphertext`, `nonce`, and `header` from the message dict.
**9. websocket_service.py routes to handler.** At `backend/app/services/websocket_service.py`, the message type `"encrypted_message"` dispatches to `handle_encrypted_message` at `backend/app/services/websocket_service.py`. The handler extracts `recipient_id`, `room_id`, `ciphertext`, `nonce`, and `header` from the message dict.
**10. message_service.py:269-314 stores in SurrealDB.** The handler calls `message_service.store_encrypted_message`, which constructs a SurrealDB document:
**10. backend/app/services/message_service.py stores in SurrealDB.** The handler calls `message_service.store_encrypted_message`, which constructs a SurrealDB document:
```python
surreal_message = {
@ -908,8 +906,8 @@ surreal_message = {
"nonce": nonce, # still base64
"header": header, # still JSON string
"sender_username": sender_user.username,
"created_at": now.isoformat(),
"updated_at": now.isoformat(),
"created_at": now.isoformat,
"updated_at": now.isoformat,
}
```
@ -917,30 +915,30 @@ The document is written to SurrealDB via `surreal_db.create_message(surreal_mess
**11. SurrealDB live query fires.** The instant the message record is created, SurrealDB pushes a `CREATE` event to any registered live query matching `recipient_id = bob_uuid`. This is a push, not a pull. There is no polling interval. The latency from write to callback is measured in single-digit milliseconds.
**12. websocket_manager.py:225-251 picks up the update.** The `_handle_live_message` callback at `websocket_manager.py:225` receives the `LiveMessageUpdate`. It checks that `update.action == "CREATE"` (ignoring updates and deletes). It wraps the message data in an `EncryptedMessageWS` Pydantic schema and calls `self.send_message(user_id, ws_message.model_dump(mode="json"))`.
**12. websocket_manager.py picks up the update.** The `_handle_live_message` callback at `websocket_manager.py` receives the `LiveMessageUpdate`. It checks that `update.action == "CREATE"` (ignoring updates and deletes). It wraps the message data in an `EncryptedMessageWS` Pydantic schema and calls `self.send_message(user_id, ws_message.model_dump(mode="json"))`.
**13. ConnectionManager sends to Bob's WebSocket(s).** At `websocket_manager.py:135-153`, the manager iterates over all of Bob's active connections (he might have the app open on his phone and laptop) and calls `websocket.send_json(message)` on each. If any send fails, that connection is added to a `dead_connections` list and cleaned up after the loop.
**13. ConnectionManager sends to Bob's WebSocket(s).** At `websocket_manager.py`, the manager iterates over all of Bob's active connections (he might have the app open on his phone and laptop) and calls `websocket.send_json(message)` on each. If any send fails, that connection is added to a `dead_connections` list and cleaned up after the loop.
**14. Bob's websocket-manager.ts receives.** The `handleMessage` method at `websocket-manager.ts:166-178` parses the JSON from `event.data`. The `routeMessage` method at `websocket-manager.ts:180-201` runs through type guards. `isEncryptedMessageWS(message)` matches because the message has `type: "encrypted_message"` and the required fields. The message is dispatched to `handleWSMessage` in `message-handlers.ts`.
**14. Bob's websocket-manager.ts receives.** The `handleMessage` method at `websocket-manager.ts` parses the JSON from `event.data`. The `routeMessage` method at `websocket-manager.ts` runs through type guards. `isEncryptedMessageWS(message)` matches because the message has `type: "encrypted_message"` and the required fields. The message is dispatched to `handleWSMessage` in `message-handlers.ts`.
**15. crypto-service.ts `decrypt(alice_id, ciphertext, nonce, header)`.** The message handler calls `CryptoService.decrypt` at `crypto-service.ts:251-301`. The `header` string is parsed to extract the ratchet header (DH public key, message number, previous chain length) and optionally an X3DH header (if this is the first message in a new conversation).
**15. crypto-service.ts `decrypt(alice_id, ciphertext, nonce, header)`.** The message handler calls `CryptoService.decrypt` at `frontend/src/crypto/crypto-service.ts`. The `header` string is parsed to extract the ratchet header (DH public key, message number, previous chain length) and optionally an X3DH header (if this is the first message in a new conversation).
**16. Load ratchet state from IndexedDB.** Same mechanism as step 3, but for Bob's side. If no ratchet state exists for Alice, and the message includes an X3DH header, `handleIncomingSession` at `crypto-service.ts:168-215` triggers:
**16. Load ratchet state from IndexedDB.** Same mechanism as step 3, but for Bob's side. If no ratchet state exists for Alice, and the message includes an X3DH header, `handleIncomingSession` at `frontend/src/crypto/crypto-service.ts` triggers:
- Look up Alice's identity key from the X3DH header
- Look up Bob's signed prekey private key from IndexedDB
- Look up the consumed one-time prekey if one was used
- Call `receiveX3DH` from `x3dh.ts:175-215` to compute the same shared secret Alice computed
- Call `initializeRatchetReceiver` from `double-ratchet.ts:68-88` with the shared secret and Bob's signed prekey pair
- Call `receiveX3DH` from `x3dh.ts` to compute the same shared secret Alice computed
- Call `initializeRatchetReceiver` from `double-ratchet.ts` with the shared secret and Bob's signed prekey pair
**17. Check skipped keys, advance receiving chain.** `decryptMessage` at `double-ratchet.ts:190-238` runs through the three-case logic:
**17. Check skipped keys, advance receiving chain.** `decryptMessage` at `double-ratchet.ts` runs through the three-case logic:
- Check if this message's key was previously stored (skipped key cache hit)
- Check if Alice's DH public key in the header differs from the last known key (DH ratchet step needed)
- Check if the message number is ahead of the current position (skip intermediate keys)
Then derive the message key: `messageKey = HMAC-SHA-256(receivingChainKey, "MessageKey")`.
**18. primitives.ts `aesGcmDecrypt`.** At `double-ratchet.ts:232`, AES-256-GCM decryption is performed. The 32-byte message key is imported as a CryptoKey. The 12-byte nonce from the message is used as the IV. The header JSON is reconstructed and used as AAD. `subtle.decrypt` verifies the 16-byte authentication tag and produces the 9 plaintext bytes. If the tag does not match (tampered message, wrong key, wrong nonce, wrong AAD), the WebCrypto API throws an `OperationError`.
**18. primitives.ts `aesGcmDecrypt`.** At `double-ratchet.ts`, AES-256-GCM decryption is performed. The 32-byte message key is imported as a CryptoKey. The 12-byte nonce from the message is used as the IV. The header JSON is reconstructed and used as AAD. `subtle.decrypt` verifies the 16-byte authentication tag and produces the 9 plaintext bytes. If the tag does not match (tampered message, wrong key, wrong nonce, wrong AAD), the WebCrypto API throws an `OperationError`.
**19. Return "Hello Bob".** The `Uint8Array` plaintext bytes are decoded via `new TextDecoder().decode(plaintextBytes)` at `crypto-service.ts:300`, producing the string `"Hello Bob"`. The ratchet state is serialized and saved back to IndexedDB.
**19. Return "Hello Bob".** The `Uint8Array` plaintext bytes are decoded via `new TextDecoder.decode(plaintextBytes)` at `frontend/src/crypto/crypto-service.ts`, producing the string `"Hello Bob"`. The ratchet state is serialized and saved back to IndexedDB.
**20. Chat.tsx renders plaintext.** The decrypted message is added to the message store. The reactive SolidJS component re-renders the message list. Bob sees "Hello Bob" on screen. The entire journey from Alice's keypress to Bob's screen took:
- ~1ms for client-side encryption (HMAC + AES-GCM)
@ -958,15 +956,15 @@ The plaintext "Hello Bob" existed in memory on Alice's device and Bob's device.
### Encryption Failures
When encryption fails (corrupted ratchet state, invalid key, etc.), an `EncryptionError` is raised at `message_service.py:366`. The exception hierarchy in `exceptions.py:73-76` derives from `AppException`. The exception handler returns the error to the client via the WebSocket error message type. The message is never stored, because the encryption failed before the store call.
When encryption fails (corrupted ratchet state, invalid key, etc.), an `EncryptionError` is raised at `backend/app/services/message_service.py`. The exception hierarchy in `exceptions.py` derives from `AppException`. The exception handler returns the error to the client via the WebSocket error message type. The message is never stored, because the encryption failed before the store call.
### Database Transaction Failures
`IntegrityError` is caught in every service method that writes to the database. For example, at `message_service.py:161-164`:
`IntegrityError` is caught in every service method that writes to the database. For example, at `backend/app/services/message_service.py`:
```python
except IntegrityError as e:
await session.rollback()
await session.rollback
raise DatabaseError("Failed to initialize conversation") from e
```
@ -974,7 +972,7 @@ The pattern is consistent: catch the SQLAlchemy integrity error, roll back the s
### WebSocket Disconnection
When a WebSocket disconnects (user closes the tab, network drops), `WebSocketDisconnect` is caught at `websocket.py:79`. The `finally` block at `websocket.py:84` always runs `connection_manager.disconnect(websocket, user_uuid)`, which:
When a WebSocket disconnects (user closes the tab, network drops), `WebSocketDisconnect` is caught at `websocket.py`. The `finally` block at `websocket.py` always runs `connection_manager.disconnect(websocket, user_uuid)`, which:
- Removes the WebSocket from the connection pool
- If it was the last connection for that user: kills the SurrealDB live query, cancels the heartbeat task, and sets presence to offline via Redis
- If other connections remain: just removes this one and logs the remaining count
@ -985,7 +983,7 @@ When a WebSocket disconnects (user closes the tab, network drops), `WebSocketDis
### Unit Tests
**Test X3DH key agreement.** Generate identity keys for Alice and Bob. Generate a signed prekey for Bob. Generate an OPK for Bob. Assemble Bob's prekey bundle. Perform sender-side X3DH as Alice via `x3dh_manager.perform_x3dh_sender`. Perform receiver-side X3DH as Bob via `x3dh_manager.perform_x3dh_receiver`, passing Alice's ephemeral public key and identity key from the sender result. Assert that `alice_result.shared_key == bob_result.shared_key`. This verifies the commutative DH math. Also assert that `alice_result.associated_data == bob_result.associated_data`, confirming both sides compute the same AAD.
**Test X3DH key agreement.** Generate identity keys for Alice and Bob. Generate a signed prekey for Bob. Generate an OPK for Bob. Assemble Bob's prekey bundle. Perform sender-side X3DH as Alice via `initiateX3DH (frontend/src/crypto/x3dh.ts)`. Perform receiver-side X3DH as Bob via `receiveX3DH (frontend/src/crypto/x3dh.ts)`, passing Alice's ephemeral public key and identity key from the sender result. Assert that `alice_result.shared_key == bob_result.shared_key`. This verifies the commutative DH math. Also assert that `alice_result.associated_data == bob_result.associated_data`, confirming both sides compute the same AAD.
Test the OPK-absent case separately: set `bob_bundle.one_time_prekey = None`, pass `bob_one_time_prekey_private = None` on the receiver side. Verify the shared keys still match (they will, because both sides fall back to the 3-DH variant).
@ -1009,15 +1007,15 @@ For clone detection, call `verify_authentication` with `credential_current_sign_
### Integration Tests
**Full message flow.** Register two users with WebAuthn mocks. Initialize encryption keys for both via `prekey_service.initialize_user_keys`. Fetch user B's prekey bundle. Initialize a conversation from A to B via `message_service.initialize_conversation`. Send an encrypted message from A. On user B's side, load the ratchet state (performing receiver-side X3DH), and decrypt the message. Verify the plaintext matches. This exercises the entire pipeline: key generation, prekey bundle assembly, X3DH, Double Ratchet initialization, encryption, SurrealDB storage, and decryption.
**Full message flow (frontend).** Stand up two clean clients in `vitest`, run `generateIdentityKeyPair` / `generateSignedPreKey` / `generateOneTimePreKeys` for each, hand Alice Bob's bundle, run `initiateX3DH` and `initializeRatchetSender`, encrypt a message, then on Bob's side run `receiveX3DH` + `initializeRatchetReceiver` and `decryptMessage`. Assert the plaintext matches. The existing `frontend/src/crypto/x3dh.test.ts` and `frontend/src/crypto/double-ratchet.test.ts` already cover this end-to-end.
**WebSocket lifecycle.** Start the FastAPI test server. Connect two test WebSocket clients (simulating Alice and Bob). Send a typed message from Alice via `ws.send_json({"type": "encrypted_message", ...})`. Assert Bob's WebSocket receives the forwarded message. Send a typing indicator from Alice. Assert Bob receives it. Disconnect Alice. Assert Bob receives a presence update (or verify via the presence service that Alice's status is now offline). Reconnect Alice. Verify the heartbeat starts and presence is restored.
**WebSocket lifecycle.** Spin up the FastAPI test app, register two users via the auth flow (which sets a session cookie), open two WebSocket connections carrying the cookies, and exercise the full message round-trip. The cookie-authenticated WebSocket is the only auth path — there is no `?user_id=` query string anymore.
**Database rollback on failure.** Create an async session. Start an operation that will fail (e.g., insert a `RatchetState` with a `user_id` that violates a foreign key constraint). Assert that the session rolls back cleanly and raises `DatabaseError`. Verify that no partial data was committed by querying the table afterward.
**Database rollback on failure.** Create an async session. Start an operation that will fail (for example, inserting an `IdentityKey` row whose `user_id` doesn't exist). Assert that the session rolls back cleanly and raises `DatabaseError`. Verify that no partial data was committed by querying the table afterward.
**Key rotation.** Initialize keys for a user. Verify they have an active signed prekey. Call `prekey_service.rotate_signed_prekey`. Verify the old SPK is now inactive. Verify the new SPK is active. Fetch the prekey bundle. Verify the bundle contains the new SPK, not the old one. Verify the signature on the new SPK validates against the user's Ed25519 identity key.
**Key rotation.** Generate a fresh signed prekey on the client (`generateSignedPreKey`), call `uploadKeys` again with it, then re-fetch the bundle as a peer. Verify the bundle contains the new SPK and that its signature validates against the user's Ed25519 identity key. Confirm that the previously-active SPK row is now `is_active = False` in PostgreSQL.
**OPK exhaustion.** Initialize a user with 2 OPKs. Fetch the prekey bundle twice (consuming both OPKs). Fetch a third time. Verify the bundle is returned with `one_time_prekey = None`. Verify X3DH still succeeds without the OPK (the 3-DH fallback).
**OPK exhaustion.** Upload a user with 2 OPKs. Fetch their prekey bundle twice (consuming both, marking them `is_used = True`). Fetch a third time and verify the bundle returns with `one_time_prekey = None`. Verify X3DH still succeeds without the OPK (the 3-DH fallback).
---
@ -1029,7 +1027,7 @@ For clone detection, call `verify_authentication` with `credential_current_sign_
**Cause:** Reusing the same nonce with the same key completely breaks GCM's security. AES-GCM uses the nonce to generate a unique keystream via CTR mode. If two messages share the same nonce and key, XORing the two ciphertexts cancels out the keystream and produces the XOR of the two plaintexts. An attacker can then use frequency analysis or known-plaintext attacks to recover both messages. Additionally, the authentication tag becomes forgeable.
**Fix:** Always use `os.urandom(12)` for a fresh nonce, as done at `double_ratchet.py:122`. Since each message key is derived from a unique chain key position, nonce reuse across different message keys is not a concern. The risk would only arise if the same message key were used twice, which the chain advancement prevents.
**Fix:** Always use `os.urandom(12)` for a fresh nonce, as done at `frontend/src/crypto/double-ratchet.ts`. Since each message key is derived from a unique chain key position, nonce reuse across different message keys is not a concern. The risk would only arise if the same message key were used twice, which the chain advancement prevents.
### Pitfall 2: Not Verifying Signed Prekey
@ -1037,7 +1035,7 @@ For clone detection, call `verify_authentication` with `credential_current_sign_
**Cause:** Skipping signature verification allows an attacker to substitute their own SPK into Bob's prekey bundle. Alice computes a shared secret with the attacker. The attacker independently computes a shared secret with Bob using Bob's real SPK. The attacker can now read and modify all messages.
**Fix:** Always verify before exchange, as done at `x3dh_manager.py:217-220`:
**Fix:** Always verify before exchange, as done at `frontend/src/crypto/x3dh.ts`:
```python
if not self.verify_signed_prekey(bob_bundle.signed_prekey,
@ -1052,7 +1050,7 @@ if not self.verify_signed_prekey(bob_bundle.signed_prekey,
**Cause:** If private key material is accessible to the server, anyone who compromises the server (SQL injection, SSH access, backup leak) can decrypt every message ever sent.
**Fix:** Client-side key generation and IndexedDB storage. The server only stores public keys. Look at `prekey_service.py:45-150`: the `store_client_keys` method receives only public keys from the client. The `private_key` fields on `IdentityKey` and `SignedPrekey` models are set to empty strings `""` when storing client-uploaded keys. Private keys live exclusively in the browser's IndexedDB, which is sandboxed per origin.
**Fix:** Client-side key generation and IndexedDB storage. The server only stores public keys. Look at `backend/app/services/prekey_service.py`: the `store_client_keys` method receives only public keys from the client. The `private_key` fields on `IdentityKey` and `SignedPrekey` models are set to empty strings `""` when storing client-uploaded keys. Private keys live exclusively in the browser's IndexedDB, which is sandboxed per origin.
### Pitfall 4: Non-Constant-Time Comparison
@ -1060,7 +1058,7 @@ if not self.verify_signed_prekey(bob_bundle.signed_prekey,
**Cause:** Using `===` or `==` for byte array comparison in JavaScript short-circuits on the first mismatch. An attacker who can measure response times with sufficient precision (microsecond resolution is achievable over a network) can determine how many leading bytes are correct and brute-force the rest.
**Fix:** XOR-based comparison as in `primitives.ts:388-397`:
**Fix:** XOR-based comparison as in `frontend/src/crypto/primitives.ts`:
```typescript
export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean {
@ -1079,19 +1077,19 @@ export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean {
The codebase follows a strict layering that separates concerns and minimizes the blast radius of bugs.
- **Encryption logic in `core/encryption/`** (never in the API layer). The `x3dh_manager.py` and `double_ratchet.py` files have zero knowledge of HTTP, WebSocket, or databases. They take bytes in and return bytes out. `X3DHManager` does not import SQLModel. `DoubleRatchet` does not import FastAPI. This makes them independently testable: you can instantiate `X3DHManager()` in a test file, call `generate_identity_keypair_x25519()`, and verify the output without setting up a database, a Redis connection, or a web server. It also makes them reusable: if you wanted to use the same X3DH implementation in a CLI tool or a different server framework, you could import `core/encryption/` without pulling in any web dependencies.
- **Encryption logic in `frontend/src/crypto/`** (never in the API layer). The `frontend/src/crypto/x3dh.ts` and `frontend/src/crypto/double-ratchet.ts` files have zero knowledge of HTTP, WebSocket, or databases. They take bytes in and return bytes out. `X3DHManager` does not import SQLModel. `DoubleRatchet` does not import FastAPI. This makes them independently testable: you can instantiate `X3DHManager` in a test file, call `generate_identity_keypair_x25519`, and verify the output without setting up a database, a Redis connection, or a web server. It also makes them reusable: if you wanted to use the same X3DH implementation in a CLI tool or a different server framework, you could import `frontend/src/crypto/` without pulling in any web dependencies.
- **Database operations in `services/`** (never in core). The `prekey_service.py`, `message_service.py`, and `auth_service.py` files handle all SQLModel queries and SurrealDB operations. They receive an `AsyncSession` as a parameter (dependency injection) and call into `core/encryption/` for crypto operations. The crypto layer never sees a session object, and the service layer never calls OpenSSL directly. This means a database schema change only affects service files and models, not the encryption logic.
- **Database operations in `services/`** (never in core). The `prekey_service.py`, `message_service.py`, and `auth_service.py` files handle all SQLModel queries and SurrealDB operations. They receive an `AsyncSession` as a parameter (dependency injection) and call into `frontend/src/crypto/` for crypto operations. The crypto layer never sees a session object, and the service layer never calls OpenSSL directly. This means a database schema change only affects service files and models, not the encryption logic.
- **Pydantic schemas for ALL API boundaries.** Every WebSocket message type has a corresponding schema in `schemas/websocket.py` (e.g., `EncryptedMessageWS`, `TypingIndicatorWS`, `ErrorMessageWS`). Every HTTP request/response has a schema in `schemas/auth.py`, `schemas/rooms.py`, etc. This provides runtime validation: if a client sends a malformed payload (missing fields, wrong types, oversized strings), Pydantic rejects it before the data reaches service logic. For example, `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` in `config.py:44` caps the ciphertext field length, preventing a client from sending a 10GB ciphertext that would consume server memory.
- **Pydantic schemas for ALL API boundaries.** Every WebSocket message type has a corresponding schema in `schemas/websocket.py` (e.g., `EncryptedMessageWS`, `TypingIndicatorWS`, `ErrorMessageWS`). Every HTTP request/response has a schema in `schemas/auth.py`, `schemas/rooms.py`, etc. This provides runtime validation: if a client sends a malformed payload (missing fields, wrong types, oversized strings), Pydantic rejects it before the data reaches service logic. For example, `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` in `config.py` caps the ciphertext field length, preventing a client from sending a 10GB ciphertext that would consume server memory.
- **Module-level singletons for stateless managers.** At the bottom of `x3dh_manager.py:353`: `x3dh_manager = X3DHManager()`. At the bottom of `double_ratchet.py:419`: `double_ratchet = DoubleRatchet()`. At the bottom of `websocket_manager.py:296`: `connection_manager = ConnectionManager()`. These classes either hold no per-request state (X3DHManager, DoubleRatchet) or manage global connection state (ConnectionManager). A single instance per process is safe because: (a) stateless managers have no mutable state, so concurrent async tasks cannot interfere with each other; (b) the ConnectionManager's mutable state (`active_connections` dict) is accessed within a single event loop, so there are no threading concerns.
- **Module-level singletons for stateless managers.** At the bottom of `backend/app/core/websocket_manager.py`: `connection_manager = ConnectionManager`. At the bottom of `backend/app/core/redis_manager.py` and `backend/app/core/surreal_manager.py`: `redis_manager` and `surreal_db`. The X3DH and Double Ratchet implementations live in the browser as plain TypeScript modules with exported functions, so there's no equivalent server-side singleton for them. The Python singletons either manage shared connection pools or are stateless wrappers; they're safe to share across the process because (a) stateless managers have no mutable state, so concurrent async tasks cannot interfere with each other, and (b) `connection_manager.active_connections` is touched only inside the single event loop.
- **Async all the way through (no blocking I/O).** Every database query uses `await session.execute(...)`. Every WebSocket operation uses `await websocket.send_json(...)`. Every Redis call uses async methods. Every SurrealDB call uses async methods. The only synchronous operations are the CPU-bound cryptographic computations (HMAC, HKDF, AES-GCM, X25519 DH exchange), which complete in microseconds and do not benefit from async. If a crypto operation ever became slow (e.g., Argon2 password hashing taking 500ms), it would need to be offloaded to a thread pool via `asyncio.to_thread()` to avoid blocking the event loop. But the operations used here are all sub-millisecond.
- **Async all the way through (no blocking I/O).** Every database query uses `await session.execute(...)`. Every WebSocket operation uses `await websocket.send_json(...)`. Every Redis call uses async methods. Every SurrealDB call uses async methods. The only synchronous operations are the CPU-bound cryptographic computations (HMAC, HKDF, AES-GCM, X25519 DH exchange), which complete in microseconds and do not benefit from async. If a crypto operation ever became slow (e.g., Argon2 password hashing taking 500ms), it would need to be offloaded to a thread pool via `asyncio.to_thread` to avoid blocking the event loop. But the operations used here are all sub-millisecond.
- **Custom exception hierarchy in `exceptions.py`.** Every error type has its own exception class (`EncryptionError`, `DecryptionError`, `KeyExchangeError`, `DatabaseError`, `UserNotFoundError`, etc.), all inheriting from `AppException`. This allows `exception_handlers.py` to map each exception type to an appropriate HTTP status code and error response. A `UserNotFoundError` returns 404. An `EncryptionError` returns 500. A `ChallengeExpiredError` returns 401. Without the hierarchy, every error would be a generic 500, and the client would have no idea what went wrong.
- **Frontend mirrors the backend structure.** The frontend's `crypto/` directory parallels the backend's `core/encryption/`. `primitives.ts` provides the same low-level operations as Python's `cryptography` library. `x3dh.ts` parallels `x3dh_manager.py`. `double-ratchet.ts` parallels `double_ratchet.py`. `crypto-service.ts` parallels `message_service.py` (the orchestration layer). This symmetry makes it easier to trace a bug: if decryption fails, you compare the frontend and backend implementations step by step, checking that the same constants, the same byte orderings, and the same HKDF parameters are used on both sides.
- **Frontend mirrors the backend structure.** The frontend's `crypto/` directory parallels the backend's `frontend/src/crypto/`. `primitives.ts` provides the same low-level operations as Python's `cryptography` library. `x3dh.ts` parallels `frontend/src/crypto/x3dh.ts`. `double-ratchet.ts` parallels `frontend/src/crypto/double-ratchet.ts`. `crypto-service.ts` parallels `message_service.py` (the orchestration layer). This symmetry makes it easier to trace a bug: if decryption fails, you compare the frontend and backend implementations step by step, checking that the same constants, the same byte orderings, and the same HKDF parameters are used on both sides.
---
@ -1103,7 +1101,7 @@ The codebase follows a strict layering that separates concerns and minimizes the
- **py_webauthn:** Implements the WebAuthn/FIDO2 server-side protocol. Handles the complex ASN.1 parsing of attestation objects, CBOR decoding of authenticator data, signature verification across multiple algorithm families (ES256, RS256, EdDSA), and challenge lifecycle management. The WebAuthn specification is over 200 pages. Writing a compliant implementation from scratch would take months and invite subtle bugs in the CBOR parsing alone. py_webauthn also provides convenient helpers like `bytes_to_base64url` and `base64url_to_bytes` that we use throughout the encryption code.
- **sqlmodel:** Combines SQLAlchemy's query builder with Pydantic's validation. You define a model once (like `RatchetState` or `IdentityKey`) and get both a database table schema and a Pydantic serialization schema. The `AsyncSession` support via `sqlmodel.ext.asyncio.session` integrates cleanly with FastAPI's async request handlers. The alternative would be using raw SQLAlchemy models plus separate Pydantic schemas, which doubles the model definitions and introduces synchronization risk (changing a column in the SQLAlchemy model but forgetting to update the Pydantic schema).
- **sqlmodel:** Combines SQLAlchemy's query builder with Pydantic's validation. You define a model once (like `User` or `IdentityKey`) and get both a database table schema and a Pydantic serialization schema. The `AsyncSession` support via `sqlmodel.ext.asyncio.session` integrates cleanly with FastAPI's async request handlers. The alternative would be using raw SQLAlchemy models plus separate Pydantic schemas, which doubles the model definitions and introduces synchronization risk (changing a column in the SQLAlchemy model but forgetting to update the Pydantic schema).
- **surrealdb:** The Python client for SurrealDB's WebSocket protocol. The key feature is live query support: `LIVE SELECT * FROM messages WHERE recipient_id = $user_id` creates a persistent subscription that pushes new matching records to a callback. This eliminates the need for a separate pub/sub system (like Redis Pub/Sub or RabbitMQ) for real-time message delivery. SurrealDB serves as both the message store and the real-time event bus.

220
PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md
View File

@ -25,10 +25,10 @@ Each challenge references actual files in this project. The line numbers point y
The backend already has the scaffolding for this. Look at these files:
- `backend/app/config.py:54` defines `WS_MESSAGE_TYPE_RECEIPT = "receipt"` already
- `backend/app/services/websocket_service.py:69-70` already routes receipt messages to `handle_read_receipt`
- `backend/app/services/websocket_service.py:272-314` has a working `handle_read_receipt` implementation
- `backend/app/schemas/websocket.py:62-69` defines `ReadReceiptWS` with `message_id`, `user_id`, and `read_at`
- `backend/app/config.py` defines `WS_MESSAGE_TYPE_RECEIPT = "receipt"` already
- `backend/app/services/backend/app/services/websocket_service.py` already routes receipt messages to `handle_read_receipt`
- `backend/app/services/backend/app/services/websocket_service.py` has a working `handle_read_receipt` implementation
- `backend/app/schemas/websocket.py` defines `ReadReceiptWS` with `message_id`, `user_id`, and `read_at`
So the backend is done. The work is on the frontend.
@ -79,11 +79,11 @@ So the backend is done. The work is on the frontend.
This feature is substantially built already. Study what exists:
- `backend/app/config.py:52` defines `WS_MESSAGE_TYPE_TYPING = "typing"`
- `backend/app/services/websocket_service.py:65-66` routes typing messages to `handle_typing_indicator`
- `backend/app/services/websocket_service.py:181-224` broadcasts typing events to the room via `connection_manager.broadcast_to_room`
- `backend/app/schemas/websocket.py:42-49` defines `TypingIndicatorWS` with `user_id`, `room_id`, `is_typing`
- `frontend/src/stores/typing.store.ts:1-113` is a complete typing store with auto-clear timeouts (5 seconds, line 10)
- `backend/app/config.py` defines `WS_MESSAGE_TYPE_TYPING = "typing"`
- `backend/app/services/backend/app/services/websocket_service.py` routes typing messages to `handle_typing_indicator`
- `backend/app/services/backend/app/services/websocket_service.py` broadcasts typing events to the room via `connection_manager.broadcast_to_room`
- `backend/app/schemas/websocket.py` defines `TypingIndicatorWS` with `user_id`, `room_id`, `is_typing`
- `frontend/src/stores/typing.store.ts` is a complete typing store with auto-clear timeouts (5 seconds,)
- `frontend/src/components/Chat/TypingIndicator.tsx` already renders the typing indicator UI
The backend and most of the frontend state management are done. What is missing:
@ -113,7 +113,7 @@ No input for 3s -> send "typing: false"
- Open two tabs as different users in the same room
- Start typing in Tab A
- Verify "typing..." appears in Tab B within 1 second
- Stop typing and verify the indicator disappears after 5 seconds (the `TYPING_TIMEOUT_MS` constant at `typing.store.ts:10`)
- Stop typing and verify the indicator disappears after 5 seconds (the `TYPING_TIMEOUT_MS` constant at `typing.store.ts`)
- Type rapidly and verify only one WebSocket event per 3-second window (check WS frames in DevTools)
- Switch rooms and verify typing state resets
@ -123,7 +123,7 @@ No input for 3s -> send "typing: false"
**What to build:** Display human-readable timestamps on messages ("2:34 PM", "Yesterday", "Jan 15") and ensure messages display in correct chronological order even when they arrive out of order over the WebSocket.
**Why it matters:** The Double Ratchet protocol handles cryptographic out-of-order delivery (see `double_ratchet.py:215-244` for the skipped message key logic), but the UI also needs to handle display ordering correctly. This is a common source of bugs in messaging apps. If Alice sends message A then message B, but B arrives at Bob first over the WebSocket, the UI must still display A before B.
**Why it matters:** The Double Ratchet protocol handles cryptographic out-of-order delivery (see `frontend/src/crypto/double-ratchet.ts` for the skipped message key logic), but the UI also needs to handle display ordering correctly. This is a common source of bugs in messaging apps. If Alice sends message A then message B, but B arrives at Bob first over the WebSocket, the UI must still display A before B.
**What you will learn:**
- Client-side message sorting with stable ordering
@ -132,14 +132,14 @@ No input for 3s -> send "typing: false"
**Where to start reading:**
- `backend/app/schemas/websocket.py:26-39` shows `EncryptedMessageWS` includes a `timestamp` field
- `frontend/src/crypto/message-store.ts:95-119` already sorts messages by `created_at` (line 106-107)
- `backend/app/schemas/websocket.py` shows `EncryptedMessageWS` includes a `timestamp` field
- `frontend/src/crypto/message-store.ts` already sorts messages by `created_at`
- `frontend/src/lib/date.ts` is the existing date utility file
- `frontend/src/components/Chat/MessageBubble.tsx` renders individual messages
**Implementation approach:**
1. The message store at `message-store.ts:106-107` already sorts by `created_at`. Verify this works when messages arrive out of order by checking that new messages insert at the correct position, not just appended at the end.
1. The message store at `message-store.ts` already sorts by `created_at`. Verify this works when messages arrive out of order by checking that new messages insert at the correct position, not just appended at the end.
2. Create a timestamp formatter. Use `Intl.RelativeTimeFormat` for recent messages and `Intl.DateTimeFormat` for older ones:
- Under 1 minute: "Just now"
@ -175,20 +175,20 @@ No input for 3s -> send "typing: false"
The backend infrastructure is already built:
- `backend/app/core/websocket_manager.py:74` calls `presence_service.set_user_online(user_id)` on WebSocket connect
- `backend/app/core/websocket_manager.py:108` calls `presence_service.set_user_offline(user_id)` on disconnect (only when the last connection for that user closes)
- `backend/app/services/presence_service.py:22-35` sets online status in SurrealDB
- `backend/app/services/presence_service.py:37-50` sets offline status
- `backend/app/services/websocket_service.py:226-270` handles presence update messages
- `backend/app/core/websocket_manager.py` calls `presence_service.set_user_online(user_id)` on WebSocket connect
- `backend/app/core/websocket_manager.py` calls `presence_service.set_user_offline(user_id)` on disconnect (only when the last connection for that user closes)
- `backend/app/services/presence_service.py` sets online status in SurrealDB
- `backend/app/services/presence_service.py` sets offline status
- `backend/app/services/backend/app/services/websocket_service.py` handles presence update messages
The frontend store is also ready:
- `frontend/src/stores/presence.store.ts:1-72` has `$presenceByUser` map, `setUserPresence`, `getUserStatus`, `isUserOnline`
- `frontend/src/stores/presence.store.ts` has `$presenceByUser` map, `setUserPresence`, `getUserStatus`, `isUserOnline`
- `frontend/src/components/Chat/OnlineStatus.tsx` already renders the status indicator
**Implementation approach:**
1. When a user connects via WebSocket, the backend already updates SurrealDB presence. What is missing is broadcasting that change to the user's contacts. In `websocket_manager.py`, after `presence_service.set_user_online(user_id)` succeeds (line 74), broadcast a presence update to all rooms that user belongs to.
1. When a user connects via WebSocket, the backend already updates SurrealDB presence. What is missing is broadcasting that change to the user's contacts. In `websocket_manager.py`, after `presence_service.set_user_online(user_id)` succeeds , broadcast a presence update to all rooms that user belongs to.
2. Add a `PresenceUpdateWS` handler in the frontend WebSocket message handler. When a `"presence"` message arrives, call `setUserPresence` from the presence store.
@ -223,30 +223,30 @@ Signal allows searching decrypted messages on-device. WhatsApp backs up messages
**Where to start reading:**
- `frontend/src/crypto/message-store.ts:1-211` is the existing IndexedDB message store with `decrypted_messages` object store
- `frontend/src/crypto/message-store.ts:34-39` creates indexes on `room_id`, `created_at`, and a compound `room_created` index
- `frontend/src/crypto/message-store.ts:95-119` shows how messages are queried by room
- `frontend/src/crypto/key-store.ts:392-412` shows `clearAllKeys()` which wipes all crypto state on logout
- `frontend/src/crypto/message-store.ts` is the existing IndexedDB message store with `decrypted_messages` object store
- `frontend/src/crypto/message-store.ts` creates indexes on `room_id`, `created_at`, and a compound `room_created` index
- `frontend/src/crypto/message-store.ts` shows how messages are queried by room
- `frontend/src/crypto/key-store.ts` shows `clearAllKeys` which wipes all crypto state on logout
**Implementation approach:**
1. Decrypted messages are already stored in IndexedDB at `message-store.ts:61-67` (`saveDecryptedMessage`). The plaintext is already persisted locally. You do not need to add a separate plaintext store.
1. Decrypted messages are already stored in IndexedDB at `message-store.ts` (`saveDecryptedMessage`). The plaintext is already persisted locally. You do not need to add a separate plaintext store.
2. Add a `searchMessages` function to `message-store.ts`. IndexedDB does not support full-text search natively. You have two options:
Option A (simple): Load all messages for the user, filter with `String.prototype.includes()` or a regex. This works for small message histories (under 10,000 messages).
Option A (simple): Load all messages for the user, filter with `String.prototype.includes` or a regex. This works for small message histories (under 10,000 messages).
Option B (performant): Build a simple inverted index in a separate IndexedDB object store. On each message save, tokenize the plaintext, store `{token: string, message_id: string}` entries. Search queries look up tokens in the index. This scales to hundreds of thousands of messages.
3. Build a search UI. Add a search input to the sidebar. Results should show message previews with the matching text highlighted. Clicking a result should navigate to that message in the conversation and scroll it into view.
4. Clear the search index on logout. Add this to whatever logout function already calls `clearAllKeys()` and `clearAllMessages()`.
4. Clear the search index on logout. Add this to whatever logout function already calls `clearAllKeys` and `clearAllMessages`.
**Security considerations:**
The local plaintext store is a security tradeoff. If the device is compromised, the search history exposes message content. Signal handles this by only searching messages from the current app session (not backed by a persistent index). Consider offering a setting: "Enable message search (stores decrypted messages locally)" vs. "Maximum security (no local message storage)".
Also consider: IndexedDB data survives browser cache clears in some browsers. Use `indexedDB.deleteDatabase()` on logout, not just clearing object stores.
Also consider: IndexedDB data survives browser cache clears in some browsers. Use `indexedDB.deleteDatabase` on logout, not just clearing object stores.
**How to test:**
- Send several messages containing known keywords between two users
@ -272,15 +272,15 @@ Also consider: IndexedDB data survives browser cache clears in some browsers. Us
**Where to start reading:**
- `frontend/src/crypto/primitives.ts:224-259` is the `aesGcmEncrypt` function. It accepts `Uint8Array` for plaintext, which means binary data works directly with no conversion needed.
- `frontend/src/crypto/primitives.ts:261-292` is `aesGcmDecrypt`
- `backend/app/services/message_service.py:269-314` is `store_encrypted_message` which stores to SurrealDB
- `backend/app/config.py:44` defines `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` (this limits inline message content, not file uploads)
- `frontend/src/crypto/frontend/src/crypto/primitives.ts` is the `aesGcmEncrypt` function. It accepts `Uint8Array` for plaintext, which means binary data works directly with no conversion needed.
- `frontend/src/crypto/frontend/src/crypto/primitives.ts` is `aesGcmDecrypt`
- `backend/app/services/backend/app/services/message_service.py` is `store_encrypted_message` which stores to SurrealDB
- `backend/app/config.py` defines `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` (this limits inline message content, not file uploads)
**Implementation approach:**
1. **Client: Read and encrypt the file**
- Use `FileReader.readAsArrayBuffer()` to get the raw bytes
- Use `FileReader.readAsArrayBuffer` to get the raw bytes
- For small files (under 1MB): encrypt the entire file as one AES-256-GCM operation using a message key from the Double Ratchet
- For large files (over 1MB): chunk into 1MB segments. Encrypt each chunk with a derived key: `chunk_key = HKDF(message_key, salt=chunk_index)`
- Each chunk gets its own nonce (never reuse nonces)
@ -371,12 +371,12 @@ Each approach has different properties for security, UX, and complexity. This ch
**Where to start reading:**
- `backend/app/core/websocket_manager.py:39` stores `active_connections: dict[UUID, list[WebSocket]]` which already supports multiple connections per user
- `backend/app/core/websocket_manager.py:52` enforces `WS_MAX_CONNECTIONS_PER_USER` (default 5, see `config.py:141`)
- `backend/app/core/websocket_manager.py:145-153` iterates over ALL connections for a user when sending a message
- `backend/app/services/prekey_service.py:152-219` initializes user keys with identity key, signed prekey, and one-time prekeys
- `backend/app/core/websocket_manager.py` stores `active_connections: dict[UUID, list[WebSocket]]` which already supports multiple connections per user
- `backend/app/core/websocket_manager.py` enforces `WS_MAX_CONNECTIONS_PER_USER` (default 5, see `config.py`)
- `backend/app/core/websocket_manager.py` iterates over ALL connections for a user when sending a message
- `backend/app/services/backend/app/services/prekey_service.py` initializes user keys with identity key, signed prekey, and one-time prekeys
- `backend/app/models/IdentityKey.py` stores one identity key per user (this needs to change)
- `backend/app/services/message_service.py:48-166` `initialize_conversation` creates one ratchet per user pair (this also needs to change)
- `backend/app/services/backend/app/services/message_service.py` `initialize_conversation` creates one ratchet per user pair (this also needs to change)
**Architecture change:**
@ -398,20 +398,19 @@ If Alice has 2 devices and Bob has 3 devices, there are 2 x 3 = 6 ratchet sessio
**Implementation phases:**
**Phase 1: Database schema changes**
- Add a `device_id` column to the `IdentityKey` model. Change the unique constraint from `(user_id)` to `(user_id, device_id)`.
- Add a `device_id` column to `RatchetState`. Change the unique constraint to `(user_id, peer_user_id, peer_device_id)`.
- Add a `device_id` column to the `IdentityKey`, `SignedPrekey`, and `OneTimePrekey` models. Change the unique constraint on `IdentityKey` from `(user_id)` to `(user_id, device_id)`.
- Add a `devices` table: `device_id`, `user_id`, `device_name`, `created_at`, `last_active`.
- Client-side ratchet state in `frontend/src/crypto/key-store.ts` becomes keyed on `(peer_user_id, peer_device_id)` instead of just `peer_id`.
**Phase 2: Device registration**
- When a user registers a new device (via WebAuthn), generate a new identity key pair for that device.
- Each device uploads its own prekey bundle (identity key, signed prekey, one-time prekeys).
- The prekey bundle endpoint (`prekey_service.py:293-361`) needs a `device_id` parameter.
- The prekey bundle endpoint (`backend/app/services/prekey_service.py`) needs a `device_id` parameter.
**Phase 3: Message fan-out**
- When Alice sends to Bob, the backend looks up all of Bob's devices.
- For each device, it checks if a ratchet session exists. If not, it performs X3DH with that device's prekey bundle.
- Alice's client encrypts the message once per recipient device.
- The WebSocket forward logic at `websocket_service.py:87-179` sends the appropriate ciphertext to each device.
- Alice's client fetches all of Bob's per-device prekey bundles, then runs X3DH and encrypts once per device.
- The WebSocket payload from Alice to the server now carries `device_id` alongside `recipient_id`.
- The forwarding logic in `backend/app/services/websocket_service.py:handle_encrypted_message` routes each ciphertext to the recipient device's connection.
**Phase 4: Device linking**
- A new device should NOT just register independently. It should be "linked" by an existing device.
@ -422,13 +421,13 @@ If Alice has 2 devices and Bob has 3 devices, there are 2 x 3 = 6 ratchet sessio
When Alice sends a message to Bob (who has 3 devices), the flow is:
1. Alice's client fetches prekey bundles for ALL of Bob's devices: `GET /api/encryption/prekeys/{bob_id}?all_devices=true`
1. Alice's client fetches prekey bundles for ALL of Bob's devices: `GET /encryption/prekey-bundle/{bob_id}?all_devices=true`
2. Alice performs X3DH separately with each device's prekey bundle (3 separate key exchanges)
3. Alice encrypts the message 3 times (once per ratchet session)
4. Alice sends 3 encrypted payloads to the server, each tagged with `(recipient_id, device_id)`
5. The server forwards each payload to the correct device
This means the `store_encrypted_message` function at `message_service.py:269-314` needs a `device_id` field. The SurrealDB message schema changes from `{sender_id, recipient_id, ciphertext}` to `{sender_id, sender_device_id, recipient_id, recipient_device_id, ciphertext}`.
This means the `store_encrypted_message` function at `backend/app/services/message_service.py` needs a `device_id` field. The SurrealDB message schema changes from `{sender_id, recipient_id, ciphertext}` to `{sender_id, sender_device_id, recipient_id, recipient_device_id, ciphertext}`.
**Device verification:**
@ -479,10 +478,10 @@ Signal uses Sender Keys for group chats. WhatsApp adopted the same approach. The
**Where to start reading:**
- `backend/app/api/rooms.py:38-132` creates rooms with participants. This is where group support starts.
- `backend/app/core/encryption/double_ratchet.py:96-109` has the chain key derivation (`_kdf_ck`) which is the same pattern used in Sender Keys (single-direction chain)
- `backend/app/core/websocket_manager.py:155-175` `broadcast_to_room` sends to all room members
- `frontend/src/crypto/primitives.ts:224-259` AES-GCM encryption works for sender key messages too
- `backend/app/api/rooms.py` creates rooms with participants. This is where group support starts.
- `frontend/src/crypto/double-ratchet.ts` has the chain key derivation (`_kdf_ck`) which is the same pattern used in Sender Keys (single-direction chain)
- `backend/app/core/websocket_manager.py` `broadcast_to_room` sends to all room members
- `frontend/src/crypto/frontend/src/crypto/primitives.ts` AES-GCM encryption works for sender key messages too
**Implementation phases:**
@ -592,19 +591,19 @@ The threat model that motivates this is "harvest now, decrypt later." An adversa
**Where to start reading:**
- `backend/app/core/encryption/x3dh_manager.py:208-281` is `perform_x3dh_sender`. This is the exact function you will modify.
- `backend/app/core/encryption/x3dh_manager.py:241-255` is where the DH shared secrets are combined:
- `frontend/src/crypto/x3dh.ts` is `perform_x3dh_sender`. This is the exact function you will modify.
- `frontend/src/crypto/x3dh.ts` is where the DH shared secrets are combined:
```python
dh1 = alice_ik_private.exchange(bob_spk_public) # line 241
dh2 = alice_ek_private.exchange(bob_ik_public) # line 242
dh3 = alice_ek_private.exchange(bob_spk_public) # line 243
# dh4 = alice_ek_private.exchange(bob_opk_public) # line 251 (optional)
dh1 = alice_ik_private.exchange(bob_spk_public)
dh2 = alice_ek_private.exchange(bob_ik_public)
dh3 = alice_ek_private.exchange(bob_spk_public)
# dh4 = alice_ek_private.exchange(bob_opk_public) (optional)
key_material = dh1 + dh2 + dh3 [+ dh4]
```
- `backend/app/core/encryption/x3dh_manager.py:257-264` is where HKDF derives the shared key:
- `frontend/src/crypto/x3dh.ts` is where HKDF derives the shared key:
```python
f = b'\xff' * X25519_KEY_SIZE
hkdf = HKDF(algorithm=hashes.SHA256(), length=X25519_KEY_SIZE,
hkdf = HKDF(algorithm=hashes.SHA256, length=X25519_KEY_SIZE,
salt=b'\x00' * X25519_KEY_SIZE, info=b'X3DH')
shared_key = hkdf.derive(f + key_material)
```
@ -640,27 +639,27 @@ Understand the difference between a KEM (Key Encapsulation Mechanism) and a DH e
**Phase 2: Prekey bundle extension**
1. Add a `kyber_public_key` field to the `PreKeyBundle` dataclass at `x3dh_manager.py:34-42`
2. When generating prekeys (`prekey_service.py:152-219`), also generate a Kyber keypair using `liboqs-python`:
1. Add a `kyber_public_key` field to the `PreKeyBundle` dataclass at `frontend/src/crypto/x3dh.ts`
2. When generating prekeys (`backend/app/services/prekey_service.py`), also generate a Kyber keypair using `liboqs-python`:
```python
import oqs
kem = oqs.KeyEncapsulation("Kyber768")
kyber_public = kem.generate_keypair()
kyber_private = kem.export_secret_key()
kyber_public = kem.generate_keypair
kyber_private = kem.export_secret_key
```
3. Store the Kyber public key in the prekey bundle, private key in the database
**Phase 3: X3DH extension**
Modify `perform_x3dh_sender` at `x3dh_manager.py:208-281`:
Modify `perform_x3dh_sender` at `frontend/src/crypto/x3dh.ts`:
1. After computing the classical `key_material` (line 252-255), check if the bundle includes a Kyber key
1. After computing the classical `key_material` , check if the bundle includes a Kyber key
2. If yes: encapsulate against the Kyber public key to get `(pq_shared_secret, pq_ciphertext)`
3. Concatenate: `combined_material = key_material + pq_shared_secret`
4. Change the HKDF info parameter: `info=b'PQXDH'` (to distinguish from classical sessions)
5. Include `pq_ciphertext` in the `X3DHResult` so it can be sent to the recipient
Modify `perform_x3dh_receiver` at `x3dh_manager.py:283-350`:
Modify `perform_x3dh_receiver` at `frontend/src/crypto/x3dh.ts`:
1. After computing the classical key material, check if a PQ ciphertext was provided
2. If yes: decapsulate to recover `pq_shared_secret`
@ -668,7 +667,7 @@ Modify `perform_x3dh_receiver` at `x3dh_manager.py:283-350`:
**Phase 4: Backwards compatibility**
If the peer does not have a Kyber public key in their prekey bundle (they have not upgraded), fall back to classical-only X3DH. The code path at `x3dh_manager.py:246-255` already handles the optional one-time prekey with an if/else. Use the same pattern for Kyber.
If the peer does not have a Kyber public key in their prekey bundle (they have not upgraded), fall back to classical-only X3DH. The code path at `frontend/src/crypto/x3dh.ts` already handles the optional one-time prekey with an if/else. Use the same pattern for Kyber.
**Key sizes to be aware of:**
- X25519 public key: 32 bytes
@ -702,11 +701,11 @@ This challenge implements true cryptographic deletion: the ciphertext remains in
**Where to start reading:**
- `backend/app/core/encryption/double_ratchet.py:96-109` is `_kdf_ck` which derives message keys. Each message gets a unique key.
- `backend/app/core/encryption/double_ratchet.py:215-244` is `_store_skipped_message_keys` with eviction logic at line 232-234. This is the pattern you will extend for TTL-based eviction.
- `backend/app/core/encryption/double_ratchet.py:246-258` is `_evict_oldest_skipped_keys` which deletes keys.
- `frontend/src/crypto/key-store.ts:353-358` stores ratchet state in IndexedDB
- `frontend/src/crypto/primitives.ts:294-298` has `generateRandomBytes` using `crypto.getRandomValues`
- `frontend/src/crypto/double-ratchet.ts` is `_kdf_ck` which derives message keys. Each message gets a unique key.
- `frontend/src/crypto/double-ratchet.ts` is `_store_skipped_message_keys` with eviction logic -234. This is the pattern you will extend for TTL-based eviction.
- `frontend/src/crypto/double-ratchet.ts` is `_evict_oldest_skipped_keys` which deletes keys.
- `frontend/src/crypto/key-store.ts` stores ratchet state in IndexedDB
- `frontend/src/crypto/frontend/src/crypto/primitives.ts` has `generateRandomBytes` using `crypto.getRandomValues`
**Implementation approach:**
@ -726,7 +725,7 @@ This challenge implements true cryptographic deletion: the ciphertext remains in
4. **Key destruction:** When the timer expires:
- Frontend: overwrite the key material with random bytes before deleting from IndexedDB. In JavaScript: `crypto.getRandomValues(keyBuffer)` then delete.
- Backend: if you store message keys server-side (for the deprecated server-side encryption path), overwrite with `os.urandom(32)` then delete.
- Backend: just delete the SurrealDB row. The server holds no keys, so there's nothing to wipe — but the recipient's locally-cached plaintext (and message key, if it's still in the skipped-key cache) needs to be overwritten on the client.
5. **UI changes:**
- Show a countdown timer on disappearing messages
@ -738,7 +737,7 @@ This challenge implements true cryptographic deletion: the ciphertext remains in
- What happens if the recipient never reads the message? The sender's copy should still eventually disappear. Implement a "maximum lifetime" that triggers regardless of read status.
- What about screenshots? You cannot prevent them. This is a social problem, not a technical one. Signal shows a notification when a screenshot is taken (on mobile), but this is best-effort.
- JavaScript garbage collection does not guarantee memory is wiped. When you overwrite a `Uint8Array`, the old values may still exist in memory until the GC reclaims the page. True secure deletion in JavaScript is not possible. Document this limitation.
- The skipped message key cache (`double_ratchet.py:47-49`) stores keys for out-of-order messages. These must also respect the TTL.
- The skipped message key cache (`frontend/src/crypto/double-ratchet.ts`) stores keys for out-of-order messages. These must also respect the TTL.
**How to test:**
- Enable disappearing messages (30 second timer) in a conversation
@ -759,9 +758,9 @@ This challenge implements true cryptographic deletion: the ciphertext remains in
**What to build:** Implement deniable authentication so that neither Alice nor Bob can prove to a third party that the other sent a specific message. This is a privacy property of the Signal Protocol that most implementations skip or get wrong.
**Prerequisites:** Solid understanding of the X3DH implementation at `x3dh_manager.py:56-353`
**Prerequisites:** Solid understanding of the X3DH implementation at `frontend/src/crypto/x3dh.ts`
**Why it matters:** In the current implementation, if Alice has Bob's signed messages (ciphertext + authentication tags), she could potentially show them to a third party to prove Bob said something. This is because the associated data (`x3dh_manager.py:272`) binds both identity keys to the session: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes`.
**Why it matters:** In the current implementation, if Alice has Bob's signed messages (ciphertext + authentication tags), she could potentially show them to a third party to prove Bob said something. This is because the associated data (`frontend/src/crypto/x3dh.ts`) binds both identity keys to the session: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes`.
Deniable authentication means that Alice could have forged the messages herself (because she also has the shared secret), so they are not valid cryptographic proof of Bob's authorship. This matters for journalists, activists, whistleblowers, and anyone who might be legally compelled to prove message authorship.
@ -773,9 +772,9 @@ In a deniable protocol, the authentication MAC can be computed by EITHER party,
**Where to start reading:**
- `x3dh_manager.py:272` is the associated data computation: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes`
- `double_ratchet.py:323-362` is `encrypt_message` where associated data is used in AES-GCM
- `double_ratchet.py:364-416` is `decrypt_message` which verifies the authentication tag
- `frontend/src/crypto/x3dh.ts` is the associated data computation: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes`
- `frontend/src/crypto/double-ratchet.ts` is `encrypt_message` where associated data is used in AES-GCM
- `frontend/src/crypto/double-ratchet.ts` is `decrypt_message` which verifies the authentication tag
**Implementation approach:**
@ -783,7 +782,7 @@ In a deniable protocol, the authentication MAC can be computed by EITHER party,
2. Alternatively, implement a "triple DH" variant where you add a third DH: `DH5 = IK_A x IK_B`. This creates a shared secret that only requires Alice and Bob's identity keys. Since both parties can compute it, a transcript of messages authenticated with this secret is not proof against either party.
3. The associated data modification at `x3dh_manager.py:272` and the corresponding line at `x3dh_manager.py:341` must change in tandem (sender and receiver must produce the same associated data).
3. The associated data modification at `frontend/src/crypto/x3dh.ts` and the corresponding line at `frontend/src/crypto/x3dh.ts` must change in tandem (sender and receiver must produce the same associated data).
**How to verify deniability:**
@ -831,10 +830,10 @@ WhatsApp solved this with Google Drive/iCloud backups encrypted with a user pass
**Where to start reading:**
- `frontend/src/crypto/key-store.ts:85-99` saves identity keys to IndexedDB. This is the data that needs backing up.
- `frontend/src/crypto/key-store.ts:353-358` saves ratchet states
- `frontend/src/crypto/key-store.ts:392-412` `clearAllKeys()` deletes everything on logout
- `frontend/src/crypto/message-store.ts:61-67` saves decrypted messages
- `frontend/src/crypto/key-store.ts` saves identity keys to IndexedDB. This is the data that needs backing up.
- `frontend/src/crypto/key-store.ts` saves ratchet states
- `frontend/src/crypto/key-store.ts` `clearAllKeys` deletes everything on logout
- `frontend/src/crypto/message-store.ts` saves decrypted messages
**Implementation phases:**
@ -935,7 +934,7 @@ Build a corporate messaging system with compliance features (backup and recovery
### Optimize Ratchet State Storage
The current implementation serializes the entire ratchet state to PostgreSQL on every single message. Look at `message_service.py:208-267` (`_save_ratchet_state_to_db`). Every call to `send_encrypted_message` or `decrypt_received_message` triggers a full ratchet state write with `await session.commit()` at line 258.
The current implementation serializes the entire ratchet state to PostgreSQL on every single message. Look at `backend/app/services/message_service.py` (`_save_ratchet_state_to_db`). Every call to `send_encrypted_message` or `decrypt_received_message` triggers a full ratchet state write with `await session.commit` .
This is a bottleneck. For a conversation with rapid back-and-forth messaging, you are writing to PostgreSQL twice per message (once for encrypt, once for decrypt).
@ -945,7 +944,7 @@ This is a bottleneck. For a conversation with rapid back-and-forth messaging, yo
1. On first ratchet state load, cache it in Redis with a key like `ratchet:{user_id}:{peer_user_id}`
2. After each ratchet advance, write to Redis only (sub-millisecond)
3. Periodically (every 10 messages or every 30 seconds) flush to PostgreSQL (durable storage)
4. On WebSocket disconnect (`websocket_manager.py:97-133`), flush all cached states to PostgreSQL
4. On WebSocket disconnect (`websocket_manager.py`), flush all cached states to PostgreSQL
5. On startup, load from PostgreSQL (Redis is not durable across restarts)
**Measurement:** Benchmark messages per second before and after with a simple load test. Target: greater than 100 messages per second per conversation.
@ -957,9 +956,9 @@ This is a bottleneck. For a conversation with rapid back-and-forth messaging, yo
1. Add timing instrumentation around `_save_ratchet_state_to_db`:
```python
import time
start = time.perf_counter()
await session.commit()
elapsed_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter
await session.commit
elapsed_ms = (time.perf_counter - start) * 1000
logger.info("Ratchet state save: %.2fms", elapsed_ms)
```
@ -967,7 +966,7 @@ This is a bottleneck. For a conversation with rapid back-and-forth messaging, yo
- Average time per `_save_ratchet_state_to_db` call
- Average time per `_load_ratchet_state_from_db` call
- Total messages per second (end-to-end)
- PostgreSQL connection pool utilization (`DB_POOL_SIZE` at `config.py:118`)
- PostgreSQL connection pool utilization (`DB_POOL_SIZE` at `config.py`)
3. Implement Redis caching, repeat the same load test, compare.
@ -990,13 +989,13 @@ Use `locust` or `k6` to load test the WebSocket layer. Determine how many concur
1. Maximum concurrent WebSocket connections before memory exhaustion
2. Message throughput: messages per second at 100, 500, 1000, 5000 concurrent connections
3. Latency: p50, p95, p99 message delivery latency at various connection counts
4. Identify the bottleneck: Is it CPU (from encryption operations)? Memory (from connection pool at `websocket_manager.py:39`)? I/O (from SurrealDB writes)? Python GIL contention?
4. Identify the bottleneck: Is it CPU (from encryption operations)? Memory (from connection pool at `websocket_manager.py`)? I/O (from SurrealDB writes)? Python GIL contention?
**Target:** Document the breaking point and propose a horizontal scaling strategy.
**Scaling considerations:**
The current architecture has a single `ConnectionManager` instance at `websocket_manager.py:296` holding all WebSocket connections in memory (`self.active_connections: dict[UUID, list[WebSocket]]`). This does not scale horizontally because connections on Server A are invisible to Server B.
The current architecture has a single `ConnectionManager` instance at `websocket_manager.py` holding all WebSocket connections in memory (`self.active_connections: dict[UUID, list[WebSocket]]`). This does not scale horizontally because connections on Server A are invisible to Server B.
To scale to multiple server instances:
1. Use Redis Pub/Sub for cross-instance message routing. When Server A needs to send a message to a user connected to Server B, it publishes to a Redis channel. Server B subscribes and forwards to the local WebSocket.
@ -1007,10 +1006,13 @@ To scale to multiple server instances:
**Load test script outline (k6):**
```javascript
import ws from 'k6/ws';
export default function () {
const url = 'ws://localhost:8000/ws?user_id=...';
const res = ws.connect(url, {}, function (socket) {
socket.on('open', () => {
export default function {
// The server expects a session cookie; load tests should sign in first
// and pass the cookie via params.headers.Cookie.
const url = 'ws://localhost:8000/ws';
const params = { headers: { Cookie: 'chat_session=<session-token>' } };
const res = ws.connect(url, params, function (socket) {
socket.on('open', => {
socket.send(JSON.stringify({
type: 'encrypted_message',
recipient_id: '...',
@ -1021,7 +1023,7 @@ export default function () {
}));
});
socket.on('message', (msg) => {});
socket.setTimeout(() => socket.close(), 30000);
socket.setTimeout( => socket.close, 30000);
});
}
```
@ -1038,7 +1040,7 @@ Run with: `k6 run --vus 100 --duration 60s load_test.js`
Build a system where identity key changes are logged to a verifiable append-only log, similar to Certificate Transparency for TLS certificates. This prevents the server from silently swapping identity keys to perform a man-in-the-middle attack.
Currently, the server at `prekey_service.py:293-361` serves prekey bundles. A compromised server could serve a fake identity key (its own), perform X3DH with both parties, and relay decrypted messages. The users would not know.
Currently, the server at `backend/app/services/prekey_service.py` serves prekey bundles. A compromised server could serve a fake identity key (its own), perform X3DH with both parties, and relay decrypted messages. The users would not know.
A key transparency log makes this detectable: every identity key is logged, and clients can audit the log to verify their key has not been replaced.
@ -1062,13 +1064,13 @@ Perform a manual security audit of the X3DH and Double Ratchet implementations.
**Audit checklist:**
1. **HKDF parameter usage** (`x3dh_manager.py:258-264`): Is the salt correct? Is the info string distinct between different uses? Are the output lengths appropriate?
1. **HKDF parameter usage** (`frontend/src/crypto/x3dh.ts`): Is the salt correct? Is the info string distinct between different uses? Are the output lengths appropriate?
2. **Nonce generation** (`double_ratchet.py:122`): Is `os.urandom(AES_GCM_NONCE_SIZE)` called for every encryption? Is there any risk of nonce reuse?
2. **Nonce generation** (`frontend/src/crypto/double-ratchet.ts`): Is `os.urandom(AES_GCM_NONCE_SIZE)` called for every encryption? Is there any risk of nonce reuse?
3. **WebCrypto API usage** (`primitives.ts:224-259`): Is the `iv` parameter always fresh? Are `additionalData` bindings correct? Are key usages properly restricted?
3. **WebCrypto API usage** (`frontend/src/crypto/primitives.ts`): Is the `iv` parameter always fresh? Are `additionalData` bindings correct? Are key usages properly restricted?
4. **Timing side channels**: Look at `primitives.ts:388-397` (`constantTimeEqual`). Is this actually constant time in JavaScript? (Spoiler: the `if (a.length !== b.length) return false` on line 389 is an early return that leaks length information. Is this a problem in context?)
4. **Timing side channels**: Look at `frontend/src/crypto/primitives.ts` (`constantTimeEqual`). Is this actually constant time in JavaScript? (Spoiler: the `if (a.length !== b.length) return false` is an early return that leaks length information. Is this a problem in context?)
5. **Random number generation**: Verify that all randomness comes from `os.urandom` (Python) or `crypto.getRandomValues` (JavaScript). Search for any use of `random` or `Math.random`.
@ -1082,7 +1084,7 @@ Write a report documenting your findings.
Build a system where the server does not know who sent a message, only who it is for. Signal implemented this as "Sealed Sender."
Currently, the WebSocket endpoint at `websocket.py:25-84` receives messages from authenticated users. The server knows `sender_id` (from the WebSocket connection) and `recipient_id` (from the message payload). It stores both in SurrealDB (`message_service.py:292-302`).
Currently, the WebSocket endpoint at `websocket.py` receives messages from authenticated users. The server knows `sender_id` (from the WebSocket connection) and `recipient_id` (from the message payload). It stores both in SurrealDB (`backend/app/services/message_service.py`).
With Sealed Sender, the sender identity is encrypted inside the E2E encrypted payload. The server only sees the recipient (needed for routing). The sender is revealed only after the recipient decrypts.
@ -1133,17 +1135,17 @@ Use Hypothesis (Python) to write property-based tests for the cryptographic impl
**Properties to verify:**
1. **X3DH symmetry:** For any two valid identity key pairs and any valid prekey bundle, `perform_x3dh_sender` and `perform_x3dh_receiver` produce the same `shared_key`. Reference: `x3dh_manager.py:208-281` and `x3dh_manager.py:283-350`.
1. **X3DH symmetry:** For any two valid identity key pairs and any valid prekey bundle, `perform_x3dh_sender` and `perform_x3dh_receiver` produce the same `shared_key`. Reference: `frontend/src/crypto/x3dh.ts` and `frontend/src/crypto/x3dh.ts`.
2. **Double Ratchet round-trip:** For any sequence of plaintext messages in any order, encrypting with `encrypt_message` and decrypting with `decrypt_message` always returns the original plaintext. Reference: `double_ratchet.py:323-362` and `double_ratchet.py:364-416`.
2. **Double Ratchet round-trip:** For any sequence of plaintext messages in any order, encrypting with `encrypt_message` and decrypting with `decrypt_message` always returns the original plaintext. Reference: `frontend/src/crypto/double-ratchet.ts` and `frontend/src/crypto/double-ratchet.ts`.
3. **AES-GCM round-trip:** For any plaintext and any valid key, `aesGcmEncrypt` followed by `aesGcmDecrypt` returns the original plaintext. Reference: `primitives.ts:224-292`.
3. **AES-GCM round-trip:** For any plaintext and any valid key, `aesGcmEncrypt` followed by `aesGcmDecrypt` returns the original plaintext. Reference: `frontend/src/crypto/primitives.ts`.
4. **No nonce reuse:** For N encryptions with the same key, all N nonces are distinct. (This should hold with overwhelming probability for random 12-byte nonces, but verify it empirically for N=10000.)
5. **Forward secrecy:** After advancing the ratchet, old message keys cannot be derived from the new state. Generate a ratchet state, encrypt a message, advance the ratchet 100 steps, then verify that the message key from step 0 cannot be recomputed from the state at step 100.
6. **Out-of-order decryption:** For any permutation of N messages, decrypting them in that permuted order produces the same set of plaintexts as decrypting in the original order. This tests the skipped message key logic at `double_ratchet.py:215-244`.
6. **Out-of-order decryption:** For any permutation of N messages, decrypting them in that permuted order produces the same set of plaintexts as decrypting in the original order. This tests the skipped message key logic at `frontend/src/crypto/double-ratchet.ts`.
**Hypothesis strategy example:**
@ -1162,7 +1164,7 @@ def test_out_of_order_delivery(message_count, delivery_order):
# Assert all plaintexts match the originals
```
**Where to put the tests:** Create a `tests/` directory in the backend and structure tests as `tests/test_x3dh.py`, `tests/test_double_ratchet.py`, `tests/test_crypto_properties.py`.
**Where to put the tests:** The crypto property tests belong in the frontend Vitest suite next to `x3dh.test.ts` and `double-ratchet.test.ts`. Add a new file like `frontend/src/crypto/properties.test.ts` for randomized round-trip and tamper checks. Reserve `backend/tests/` for backend integration tests (auth, prekey storage, session enforcement).
---