diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md index d416b275..fb7315f8 100644 --- a/PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md @@ -20,19 +20,19 @@ The Signal Protocol is now the gold standard for encrypted messaging. Signal (th ### Cryptographic Protocols -- **X3DH key exchange for asynchronous session establishment.** The fundamental problem with Diffie-Hellman key exchange is that both parties need to be online at the same time. If Alice wants to send a message to Bob, but Bob is offline, classic DH cannot work. X3DH (Extended Triple Diffie-Hellman) solves this by having Bob pre-publish a bundle of public keys to the server: a long-term identity key, a medium-term signed prekey, and a set of single-use one-time prekeys. When Alice wants to start a conversation, she downloads Bob's bundle and performs three (or four, if a one-time prekey is available) DH computations to derive a shared secret. Bob can complete his half of the exchange when he comes back online. This project implements X3DH in `backend/app/core/encryption/x3dh_manager.py` on the server side and `frontend/src/crypto/x3dh.ts` on the client side, using X25519 for Diffie-Hellman and Ed25519 for prekey signatures. +- **X3DH key exchange for asynchronous session establishment.** The fundamental problem with Diffie-Hellman key exchange is that both parties need to be online at the same time. If Alice wants to send a message to Bob, but Bob is offline, classic DH cannot work. X3DH (Extended Triple Diffie-Hellman) solves this by having Bob pre-publish a bundle of public keys to the server: a long-term identity key, a medium-term signed prekey, and a set of single-use one-time prekeys. When Alice wants to start a conversation, she downloads Bob's bundle and performs three (or four, if a one-time prekey is available) DH computations to derive a shared secret. Bob can complete his half of the exchange when he comes back online. **All X3DH math runs in the browser** in `frontend/src/crypto/x3dh.ts` (`initiateX3DH` and `receiveX3DH`); the backend only stores and serves the public bundle in `backend/app/services/prekey_service.py`. X25519 is used for Diffie-Hellman, Ed25519 for prekey signatures. -- **Double Ratchet algorithm for forward secrecy and post-compromise security.** Once X3DH establishes an initial shared secret, the Double Ratchet takes over for ongoing message encryption. It combines two ratcheting mechanisms: a symmetric-key ratchet (using HMAC-based key derivation) that advances with every message, and a Diffie-Hellman ratchet that advances every time the conversation "turn" changes. The symmetric ratchet provides forward secrecy: if an attacker steals your current keys, they cannot decrypt past messages because each message key is derived from and then discarded after the previous chain key. The DH ratchet provides post-compromise security (also called "future secrecy" or "self-healing"): even if an attacker compromises your current state completely, once you send a new message with a fresh DH key pair, the attacker loses access because they do not know the new private key. The implementation lives in `backend/app/core/encryption/double_ratchet.py` and `frontend/src/crypto/double-ratchet.ts`. +- **Double Ratchet algorithm for forward secrecy and post-compromise security.** Once X3DH establishes an initial shared secret, the Double Ratchet takes over for ongoing message encryption. It combines two ratcheting mechanisms: a symmetric-key ratchet (HMAC-based) that advances with every message, and a Diffie-Hellman ratchet that advances every time the conversation "turn" changes. The symmetric ratchet provides forward secrecy: if an attacker steals your current keys, they cannot decrypt past messages because each message key is derived from and then discarded after the previous chain key. The DH ratchet provides post-compromise security (also called "future secrecy" or "self-healing"): even if an attacker compromises your current state completely, once you send a new message with a fresh DH key pair, the attacker loses access because they do not know the new private key. **The Double Ratchet runs entirely in the browser** in `frontend/src/crypto/double-ratchet.ts` (`encryptMessage` / `decryptMessage`); the backend only relays the resulting ciphertext. -- **AES-256-GCM authenticated encryption.** Each message is encrypted using AES-256-GCM (Galois/Counter Mode), which provides both confidentiality and integrity. GCM is an AEAD (Authenticated Encryption with Associated Data) cipher, meaning it produces a ciphertext and an authentication tag. If anyone tampers with the ciphertext, the tag verification fails and decryption is rejected. The "associated data" in this project is the concatenation of both users' identity public keys, binding each encrypted message to the specific conversation participants. The implementation uses 12-byte random nonces and 32-byte keys derived from the Double Ratchet chain. You will see this in the `_encrypt_with_message_key` and `_decrypt_with_message_key` methods of the `DoubleRatchet` class, and in the `aesGcmEncrypt`/`aesGcmDecrypt` functions in `frontend/src/crypto/primitives.ts`. +- **AES-256-GCM authenticated encryption.** Each message is encrypted using AES-256-GCM (Galois/Counter Mode), which provides both confidentiality and integrity. GCM is an AEAD (Authenticated Encryption with Associated Data) cipher, meaning it produces a ciphertext and an authentication tag. If anyone tampers with the ciphertext, the tag verification fails and decryption is rejected. The "associated data" in this project is the serialized message header (DH public key, message number, previous chain length), binding each ciphertext to the ratchet state that produced it. The implementation uses 12-byte random nonces and 32-byte keys derived from the Double Ratchet chain. You'll see it in `aesGcmEncrypt` / `aesGcmDecrypt` in `frontend/src/crypto/primitives.ts`, called from inside `encryptMessage` / `decryptMessage`. -- **HKDF-SHA256 key derivation.** Raw Diffie-Hellman outputs are not suitable for direct use as encryption keys. They have non-uniform distribution and may contain structural biases. HKDF (HMAC-based Key Derivation Function) extracts entropy from DH outputs and expands it into keys of the required length. This project uses HKDF-SHA256 throughout: in X3DH to derive the initial shared secret from concatenated DH outputs, and in the Double Ratchet to derive new root keys and chain keys from each DH ratchet step. The X3DH derivation uses a salt of all-zero bytes and an info string of `b'X3DH'`, prepending a block of `0xFF` bytes as specified by the Signal Protocol. You will see this in both the Python `cryptography` library's HKDF and the browser's WebCrypto `deriveBits` API with HKDF. +- **HKDF-SHA256 key derivation.** Raw Diffie-Hellman outputs are not suitable for direct use as encryption keys. They have non-uniform distribution and may contain structural biases. HKDF (HMAC-based Key Derivation Function) extracts entropy from DH outputs and expands it into keys of the required length. This project uses HKDF-SHA256 throughout: in X3DH to derive the initial shared secret from concatenated DH outputs (with the spec-mandated `0xFF * 32` F prefix and `b'X3DH'` info), and in the Double Ratchet's KDF_RK (root-key derivation) which uses the previous root key as HKDF salt and the new DH output as input keying material. The chain-key step (KDF_CK) uses HMAC-SHA256 with single-byte tags `0x01` (message key) and `0x02` (next chain key) per the Signal spec. -- **Ed25519 digital signatures and X25519 Diffie-Hellman.** The project uses two related but distinct elliptic curve operations on Curve25519. X25519 is used for Diffie-Hellman key exchange: two parties each generate a key pair, exchange public keys, and compute a shared secret. Ed25519 is used for digital signatures: the identity key owner signs their signed prekey so that other users can verify it was genuinely published by that identity, not substituted by a malicious server. The distinction matters because the mathematical operations are different (X25519 operates on the Montgomery form of the curve, Ed25519 on the Edwards form), but they share the same underlying curve. This project maintains separate X25519 and Ed25519 key pairs for each user identity, as you can see in the `generate_identity_keypair_x25519` and `generate_identity_keypair_ed25519` methods. +- **Ed25519 digital signatures and X25519 Diffie-Hellman.** The project uses two related but distinct elliptic curve operations on Curve25519. X25519 is used for Diffie-Hellman key exchange: two parties each generate a key pair, exchange public keys, and compute a shared secret. Ed25519 is used for digital signatures: the identity key owner signs their signed prekey so that other users can verify it was genuinely published by that identity, not substituted by a malicious server. The mathematical operations are different (X25519 operates on the Montgomery form of the curve, Ed25519 on the Edwards form), but they share the same underlying curve. Each user generates separate X25519 and Ed25519 identity key pairs in `generateIdentityKeyPair` (`frontend/src/crypto/x3dh.ts`); only the public halves ever leave the device. ### Authentication and Identity -- **WebAuthn/FIDO2 passkey registration and authentication.** WebAuthn is a W3C standard that allows users to authenticate using public key cryptography instead of passwords. During registration, the browser creates a new key pair on a hardware or platform authenticator (a YubiKey, a fingerprint reader, a phone's secure enclave). The public key is sent to the server. The private key never leaves the authenticator. During login, the server sends a random challenge, the authenticator signs it with the private key, and the server verifies the signature against the stored public key. No shared secret ever crosses the network. The implementation uses the `py_webauthn` library on the server side (`backend/app/core/passkey/passkey_manager.py`) and the browser's native `navigator.credentials` API on the client. Registration options include resident key requirement (enabling usernameless login) and user verification preference. +- **WebAuthn/FIDO2 passkey registration and authentication.** WebAuthn is a W3C standard that allows users to authenticate using public key cryptography instead of passwords. During registration, the browser creates a new key pair on a hardware or platform authenticator (a YubiKey, a fingerprint reader, a phone's secure enclave). The public key is sent to the server. The private key never leaves the authenticator. During login, the server sends a random challenge, the authenticator signs it with the private key, and the server verifies the signature against the stored public key. No shared secret ever crosses the network. The implementation uses the `py_webauthn` library on the server (`backend/app/core/passkey/passkey_manager.py`) and the browser's native `navigator.credentials` API on the client. Registration options require resident keys (enabling usernameless login) and **require** user verification (PIN/biometric), since this is an E2EE chat app and unattended-device access shouldn't unlock the conversation. The WebAuthn `user.id` is a 64-byte random `webauthn_user_handle` per spec, stored on the User row and never exposed to clients. - **Public key cryptography for identity verification.** Each user in the system has two types of public keys: WebAuthn credentials (for authentication with the server) and Signal Protocol identity keys (for end-to-end encryption with other users). The WebAuthn credential proves to the server that you are who you claim to be. The Signal Protocol identity key lets other users verify your encryption keys. These are separate key pairs serving separate purposes. The identity key is an Ed25519 key pair that signs the user's medium-term signed prekeys, letting recipients verify that the prekey bundle they downloaded was genuinely published by that user and not substituted by a compromised server. @@ -40,21 +40,21 @@ The Signal Protocol is now the gold standard for encrypted messaging. Signal (th ### Fullstack Architecture -- **FastAPI async backend with SQLModel ORM.** The backend is built with FastAPI, using its native async support with `asyncio` and `async/await` throughout. Database models use SQLModel, which merges SQLAlchemy's ORM capabilities with Pydantic's data validation. The app factory pattern in `factory.py` creates the FastAPI instance with a lifespan context manager that handles startup (initializing PostgreSQL, connecting to Redis, connecting to SurrealDB) and shutdown (disconnecting cleanly). API routes are organized into four routers: `auth` (WebAuthn registration and login), `encryption` (prekey bundle management), `rooms` (chat room CRUD), and `websocket` (real-time message transport). The backend uses `ORJSONResponse` as the default response class for faster JSON serialization. +- **FastAPI async backend with SQLModel ORM.** The backend is built with FastAPI, using its native async support with `asyncio` and `async/await` throughout. Database models use SQLModel, which merges SQLAlchemy's ORM capabilities with Pydantic's data validation. The app factory pattern in `factory.py` creates the FastAPI instance with a lifespan context manager that handles startup (initializing PostgreSQL, connecting to Redis, connecting to SurrealDB) and shutdown (disconnecting cleanly). API routes are organized into four routers: `auth` (WebAuthn registration / login / `/auth/me` / `/auth/logout`), `encryption` (public prekey bundle upload and lookup), `rooms` (chat room CRUD with membership-gated access), and `websocket` (real-time message transport, authenticated via the same session cookie). All non-auth endpoints depend on `current_user` from `app/core/dependencies.py`, which resolves the session cookie against Redis. Rate limits are applied via `slowapi`. The backend uses `ORJSONResponse` as the default response class for faster JSON serialization. - **SolidJS reactive frontend with TypeScript.** The frontend uses SolidJS, a reactive UI framework with fine-grained reactivity (no virtual DOM). State management uses Nanostores with the `@nanostores/solid` binding and `@nanostores/persistent` for localStorage persistence. Routing uses `@solidjs/router`. Server state management uses `@tanstack/solid-query` for data fetching, caching, and synchronization. The component structure is organized into domain folders: `Auth` (AuthCard, AuthForm, PasskeyButton), `Chat` (ChatInput, ChatHeader, ConversationList, MessageBubble, MessageList, EncryptionBadge, OnlineStatus, TypingIndicator, UserSearch), `Layout` (AppShell, Header, Sidebar, ProtectedRoute), and `UI` (Avatar, Badge, Button, Dropdown, Input, Modal, Spinner, Toast, Tooltip). Styling uses Tailwind CSS v4 via the Vite plugin. - **WebSocket real-time messaging.** Messages are delivered in real time over WebSocket connections. The backend's `websocket_manager.py` maintains a connection pool and handles broadcasting. The frontend's `websocket/websocket-manager.ts` manages connection lifecycle, reconnection, and heartbeats. Message types include encrypted messages, typing indicators, presence updates, read receipts, and error notifications. The WebSocket protocol carries ciphertext between clients; the server routes messages without decrypting them. Heartbeat interval is configurable (default 30 seconds), and each user can have up to 5 concurrent connections (for multiple devices or tabs). -- **Multi-database architecture: PostgreSQL, SurrealDB, Redis.** The project uses three databases, each chosen for a specific access pattern. PostgreSQL (via `asyncpg` and SQLModel) stores authentication data, user profiles, WebAuthn credentials, and encryption key metadata (identity keys, signed prekeys, one-time prekeys, ratchet state). This data is relational, requires ACID transactions, and benefits from SQL's querying power. SurrealDB stores real-time chat data: messages, room membership, and presence information. Its live query feature enables real-time push notifications without polling. Redis stores ephemeral state: WebAuthn challenge tokens (with TTL-based expiration), rate limiting counters, and session caches. Each database has its own manager class (`Base.py` for PostgreSQL, `surreal_manager.py` for SurrealDB, `redis_manager.py` for Redis) handling connection lifecycle and operations. +- **Multi-database architecture: PostgreSQL, SurrealDB, Redis.** The project uses three databases, each chosen for a specific access pattern. PostgreSQL (via `asyncpg` and SQLModel) stores authentication data, user profiles, WebAuthn credentials, and **public** prekey material (identity public keys, signed prekey publics + signatures, unused one-time prekey publics). No private key ever lives server-side. SurrealDB stores real-time chat data: messages (ciphertext only), room membership, and presence. Its live query feature enables real-time push notifications without polling. Redis stores ephemeral state: WebAuthn registration context (challenge + user handle + display name), authentication challenges (keyed by challenge bytes for replay-safety), session tokens (`session: -> user_id` with 24h TTL), and `slowapi` rate-limit counters. Each database has its own manager class (`Base.py` for PostgreSQL, `surreal_manager.py` for SurrealDB, `redis_manager.py` for Redis). - **Docker containerization with Nginx reverse proxy.** The entire application runs in Docker containers orchestrated by Docker Compose. The development compose file (`dev.compose.yml`) defines six services: PostgreSQL 16 Alpine, SurrealDB (latest), Redis 8 Alpine, the FastAPI backend, the Vite dev server for the frontend, and Nginx Alpine as a reverse proxy. Each database has health checks to ensure the backend only starts after its dependencies are ready. The Nginx configuration routes HTTP traffic and proxies WebSocket connections. There are separate Dockerfiles for development (with hot-reload via volume mounts) and production (with optimized builds) in `conf/docker/dev/` and `conf/docker/prod/`. ### Security Engineering -- **Zero-knowledge server design.** The server in this application is designed so that it cannot read messages even if it wanted to. Private keys are generated on client devices and stored in the browser's IndexedDB. The server stores only public keys and encrypted message blobs. When Alice sends a message to Bob, the client-side crypto layer encrypts the plaintext using the Double Ratchet, sends the ciphertext over the WebSocket, and the server forwards it to Bob. At no point does the server possess any decryption key. The associated data binding (concatenation of both users' identity public keys) ensures that even if the server tried to relay a message to the wrong recipient, decryption would fail because the associated data would not match. This is a meaningful defense against a compromised server attempting message misdirection. +- **Zero-knowledge server design.** The server in this application is designed so that it cannot read messages even if it wanted to. Private keys are generated on client devices (`generateIdentityKeyPair` in `frontend/src/crypto/x3dh.ts`) and stored in the browser's IndexedDB (`frontend/src/crypto/key-store.ts`). The server stores only public keys (`backend/app/services/prekey_service.py:store_client_keys` writes only the public halves; the schema columns for private keys were dropped) and ciphertext blobs in SurrealDB. When Alice sends a message to Bob, `crypto-service.ts:encrypt` runs the Double Ratchet locally, hands the ciphertext to the WebSocket, and the server's `message_service.store_encrypted_message` does a pure pass-through write — there is no `decrypt` code path on the backend. The header serialization that becomes the AES-GCM associated data binds each ciphertext to the ratchet state that produced it, so a compromised server cannot redirect ciphertext between sessions without breaking authentication. -- **Key rotation and lifecycle management.** Cryptographic keys in this system have defined lifecycles. One-time prekeys are single-use: once consumed during an X3DH exchange, they are deleted from the server. The system maintains a pool of 100 one-time prekeys and replenishes them when the count drops. Signed prekeys rotate every 48 hours, with a 7-day retention period for old keys to handle in-flight messages. Identity keys are long-term but can be rotated (configurable, default 90 days). Ratchet state is per-conversation and advances with every message. Skipped message keys (for out-of-order delivery) are cached up to a configurable limit (default 2000) with the oldest keys evicted first. These lifecycle parameters are centralized in `config.py` as named constants. +- **Key rotation and lifecycle management.** Cryptographic keys in this system have defined lifecycles, all driven by the client. One-time prekeys are single-use: when a sender fetches Bob's bundle, the chosen OPK is marked `is_used=True` server-side (so it can't be handed out twice), but the private OPK lives only in Bob's IndexedDB. The client maintains a pool of 100 OPKs and replenishes (and re-uploads the new public OPKs) when the unused count drops below half (`crypto-service.ts:replenishOneTimePreKeys`). Signed prekeys rotate every 48 hours; the client generates a new SPK locally and re-runs `uploadKeys` to publish the new public + signature. Identity keys are long-term and intentionally don't rotate — they're the trust anchor that signs the SPK. Ratchet state is per-conversation, advances with every message, and is serialized into the IndexedDB ratchet store after every encrypt/decrypt. Skipped message keys (for out-of-order delivery) live in the in-memory ratchet state, capped at `MAX_SKIP_MESSAGE_KEYS`. - **Threat modeling for messaging applications.** Building this project exposes you to the specific threat model that the Signal Protocol is designed to address. The primary threats include: a compromised server attempting to read or modify messages (mitigated by E2E encryption and associated data binding), an attacker who steals a user's current session keys (mitigated by forward secrecy from the symmetric ratchet), an attacker who fully compromises a device and then loses access (mitigated by post-compromise security from the DH ratchet), a man-in-the-middle attempting to substitute public keys (mitigated by identity key signing and optional safety number verification), and an attacker who intercepts one-time prekeys (mitigated by the three base DH computations that provide security even without the one-time prekey). Understanding which threats each mechanism addresses is the core of cryptographic protocol design. @@ -158,34 +158,30 @@ encrypted-p2p-chat/ | | | +-- websocket.py # WebSocket connection and message routing | | | | | |-- core/ -| | | |-- encryption/ -| | | | |-- x3dh_manager.py # X3DH protocol: key generation, exchange, verification -| | | | +-- double_ratchet.py # Double Ratchet: symmetric + DH ratchet, encrypt/decrypt | | | |-- passkey/ | | | | +-- passkey_manager.py # WebAuthn credential registration and verification +| | | |-- dependencies.py # current_user, issue_session, revoke_session | | | |-- websocket_manager.py # Connection pool, broadcasting, heartbeat management | | | |-- surreal_manager.py # SurrealDB client: connect, query, live subscriptions -| | | |-- redis_manager.py # Redis client: challenges, rate limits, ephemeral cache +| | | |-- redis_manager.py # Redis client: sessions, challenges, ephemeral cache | | | |-- enums.py # Shared enumerations (message types, status codes) | | | |-- exceptions.py # Domain-specific exception classes | | | +-- exception_handlers.py # FastAPI exception handler registration | | | | | |-- models/ # SQLModel database models (PostgreSQL) | | | |-- Base.py # SQLAlchemy engine, session factory, init_db -| | | |-- User.py # User account with username and display name -| | | |-- Credential.py # WebAuthn credential storage (public key, counter) -| | | |-- IdentityKey.py # Long-term X25519 + Ed25519 identity key pairs -| | | |-- SignedPrekey.py # Medium-term signed prekeys (rotate every 48h) -| | | |-- OneTimePrekey.py # Single-use prekeys for X3DH (pool of 100) -| | | |-- RatchetState.py # Per-conversation Double Ratchet state blob -| | | +-- SkippedMessageKey.py # Cached keys for out-of-order message delivery +| | | |-- User.py # User account + 64-byte WebAuthn user_handle +| | | |-- Credential.py # WebAuthn credential storage (public key, counter, BE/BS flags) +| | | |-- IdentityKey.py # Long-term X25519 + Ed25519 identity public keys (no privates) +| | | |-- SignedPrekey.py # Client-uploaded signed prekey publics + signatures +| | | +-- OneTimePrekey.py # Single-use prekey publics (consumed via is_used flag) | | | | | |-- services/ # Business logic layer -| | | |-- auth_service.py # Registration and authentication orchestration -| | | |-- prekey_service.py # Key bundle management and rotation scheduling -| | | |-- message_service.py # Encryption/decryption coordination +| | | |-- auth_service.py # Registration / authentication / search (no key gen) +| | | |-- prekey_service.py # Public-only prekey bundle storage + lookup +| | | |-- message_service.py # Pass-through ciphertext writer (server never decrypts) | | | |-- presence_service.py # Online/offline status tracking via SurrealDB -| | | +-- websocket_service.py # WebSocket message routing and type dispatch +| | | +-- websocket_service.py # WebSocket message routing + membership check + per-user rate cap | | | | | |-- schemas/ # Pydantic request/response models | | | |-- auth.py # Registration/login options and verification schemas @@ -199,11 +195,10 @@ encrypted-p2p-chat/ | | +-- main.py # Uvicorn entry point | | | |-- tests/ # pytest test suite -| | |-- conftest.py # Shared fixtures (async engine, test client) -| | |-- test_auth_service.py # Authentication flow tests -| | |-- test_encryption.py # Double Ratchet encrypt/decrypt round-trip tests -| | |-- test_message_service.py # Message service integration tests -| | +-- test_x3dh.py # X3DH key exchange protocol tests +| | |-- conftest.py # Shared fixtures (in-memory async SQLite, test users) +| | |-- test_auth_service.py # User CRUD and lookup +| | |-- test_prekey_service.py # Bundle/upload regression tests (incl. broken-filter regression) +| | +-- test_session_auth.py # Protected endpoints return 401 without session cookie | | | |-- alembic/ # Database migration scripts | | +-- env.py # Alembic environment configuration @@ -267,8 +262,7 @@ encrypted-p2p-chat/ | | | +-- room.service.ts # Room and message API calls | | | | | |-- stores/ # Nanostores state management -| | | |-- auth.store.ts # Authentication state (user, token) -| | | |-- session.store.ts # Session lifecycle management +| | | |-- auth.store.ts # Current user + persistent userId hint | | | |-- rooms.store.ts # Room list and active room state | | | |-- messages.store.ts # Message history per conversation | | | |-- presence.store.ts # User online status tracking @@ -408,7 +402,7 @@ Check that ports 5432, 8001, 6379, 8000, 5173, and 80 are not already in use. No WebAuthn requires either `localhost` or a valid HTTPS origin. The default configuration uses `RP_ID=localhost` and `RP_ORIGIN=http://localhost` in `.env.example`. Make sure your `.env` file has these values set correctly. If you are accessing the frontend at `http://localhost:5173`, then `RP_ORIGIN` must match that exact origin (scheme + host + port). If you are testing from a LAN IP address or a non-localhost hostname, you will need a TLS certificate because WebAuthn will refuse to create credentials on non-secure origins other than localhost. **"No identity key" error when starting a chat:** -After WebAuthn registration, the client must generate and upload encryption keys (identity key pair, signed prekey, one-time prekeys) to the server. This happens in `crypto-service.ts`. If this step fails silently, the user will have an account but no encryption keys, and other users will not be able to initiate encrypted sessions with them. Check the browser console for errors during the key generation step. Common causes include IndexedDB access being blocked (private browsing mode in some browsers restricts IndexedDB), WebCrypto API not being available (requires a secure context), or network errors during the key upload POST request to the `/api/encryption/keys` endpoint. +After WebAuthn registration, the client must generate and upload **public** encryption keys (identity key pair publics, signed prekey + signature, one-time prekey publics) to the server. This happens in `crypto-service.ts:initialize`, which then calls `uploadPublicKeys` → `POST /encryption/upload-keys/{user_id}`. If this step fails silently, the user will have an account but no published prekey bundle, and other users will get a 400 "User has not uploaded keys yet" when trying to initiate an X3DH session. Check the browser console for errors during the key generation step. Common causes include IndexedDB access being blocked (private browsing mode in some browsers restricts IndexedDB), WebCrypto API not being available (requires a secure context), or the session cookie not being sent (verify the request has `credentials: 'include'`, which the shared `api-client.ts` already sets). **Database migration errors:** If you see schema mismatch errors on startup, run migrations first. For Docker: `just migrate head`. For local development: `just migrate-local head`. If you need to start fresh, `just dev-down` followed by removing the Docker volumes will give you a clean database. diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md index 78647dda..4bfb5db5 100644 --- a/PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md @@ -92,37 +92,37 @@ impossible for the server to read the data in the first place. ### How It Works (in this project) ``` -Alice's Device Server Bob's Device -+----------------+ +----------------+ +----------------+ -| Plaintext | | | | | -| "Hello Bob" | | Encrypted | | Plaintext | -| | | ---> | blob only | ---> | "Hello Bob" | -| v | | No keys | | ^ | -| AES-256-GCM | | No access | | AES-256-GCM | -| encrypt | | No decrypt | | decrypt | -+----------------+ +----------------+ +----------------+ +Alice's Device Server Bob's Device ++----------------+ +----------------+ +----------------+ +| Plaintext | | | | | +| "Hello Bob" | | Encrypted | | Plaintext | +| | | ---> | blob only | ---> | "Hello Bob" | +| v | | No keys | | ^ | +| AES-256-GCM | | No access | | AES-256-GCM | +| encrypt | | No decrypt | | decrypt | ++----------------+ +----------------+ +----------------+ -Keys derived from Server stores Keys derived from -Double Ratchet on ciphertext, nonce, Double Ratchet on -Alice's device and header verbatim Bob's device +Keys derived from Server stores Keys derived from +Double Ratchet on ciphertext, nonce, Double Ratchet on +Alice's device and header verbatim Bob's device ``` -On the server side, `message_service.py:269-314` implements the -`store_encrypted_message` method. Look at the docstring on line 280: +On the server side, `backend/app/services/message_service.py` implements the +`store_encrypted_message` method. Look at the docstring : `"Stores client-encrypted message in SurrealDB (pass-through, no server encryption)"`. The function receives `ciphertext`, `nonce`, and `header` as strings -from the client, and stores them directly in SurrealDB at lines 292-302 +from the client, and stores them directly in SurrealDB without any decryption or re-encryption step. The server is a passthrough. It writes what it receives and reads what it wrote. At no point does it import any cryptographic key or call any decryption function on these message parameters. -On the client side, `crypto-service.ts:217-249` implements the +On the client side, `frontend/src/crypto/crypto-service.ts` implements the `encrypt` method. The client calls `encryptMessage` from -`double-ratchet.ts` at line 229, receives ciphertext and nonce, then +`double-ratchet.ts` , receives ciphertext and nonce, then sends these as base64-encoded strings to the server. The plaintext never leaves the client's process. The actual symmetric encryption -happens in `primitives.ts:224-259` using the WebCrypto API's AES-GCM +happens in `frontend/src/crypto/primitives.ts` using the WebCrypto API's AES-GCM implementation, which runs in the browser's native cryptographic module rather than in JavaScript. This means the plaintext is never even accessible to JavaScript debugging tools during the encryption @@ -166,21 +166,21 @@ solving a different problem: Signal Protocol = X3DH (initial handshake) + Double Ratchet (ongoing encryption) X3DH: "How do Alice and Bob agree on a shared secret - when Bob might be offline?" + when Bob might be offline?" - Solves the ASYNCHRONOUS key agreement problem. - Bob uploads prekey bundles to the server ahead of time. - Alice can start a conversation using Bob's prekeys - without Bob being online. + Solves the ASYNCHRONOUS key agreement problem. + Bob uploads prekey bundles to the server ahead of time. + Alice can start a conversation using Bob's prekeys + without Bob being online. Double Ratchet: "Once they share a secret, how do they - encrypt each message with a UNIQUE key - that can never be recovered?" + encrypt each message with a UNIQUE key + that can never be recovered?" - Solves the FORWARD SECRECY and POST-COMPROMISE - SECURITY problem. Every message gets its own - ephemeral encryption key. Compromising one key - does not reveal past or future messages. + Solves the FORWARD SECRECY and POST-COMPROMISE + SECURITY problem. Every message gets its own + ephemeral encryption key. Compromising one key + does not reveal past or future messages. ``` X3DH runs once at the start of a conversation. The Double Ratchet runs @@ -227,9 +227,9 @@ cryptographic identity. It is generated once and kept for the lifetime of the account. This project generates two identity keypairs per user: - An X25519 keypair for Diffie-Hellman operations - (ref: `x3dh_manager.py:60-86`) + (ref: `frontend/src/crypto/x3dh.ts`) - An Ed25519 keypair for digital signatures - (ref: `x3dh_manager.py:88-114`) + (ref: `frontend/src/crypto/x3dh.ts`) The X25519 keypair participates directly in the DH calculations. The Ed25519 keypair signs the signed prekey to prove it belongs to the same @@ -244,10 +244,10 @@ their identity. The signed prekey is an X25519 keypair that rotates periodically. In this project, rotation happens every 48 hours as configured at -`config.py:76` (`SIGNED_PREKEY_ROTATION_HOURS=48`). +`config.py` (`SIGNED_PREKEY_ROTATION_HOURS=48`). -When a new SPK is generated (`x3dh_manager.py:116-152`), the public -key is signed using the Ed25519 identity key at line 141: +When a new SPK is generated (`frontend/src/crypto/x3dh.ts`), the public +key is signed using the Ed25519 identity key : `signature = identity_private.sign(spk_public_bytes)`. This signature proves that the SPK was created by the holder of the identity key. When Alice downloads Bob's prekey bundle, she verifies this signature before @@ -260,24 +260,23 @@ computed with them become unrecoverable). Longer rotation means fewer key management operations and less complexity. The 48-hour window used here is consistent with Signal's recommendation. -Old signed prekeys are retained for 7 days after deactivation -(`config.py:77`, `SIGNED_PREKEY_RETENTION_DAYS=7`) to allow -messages-in-flight that were encrypted against the old SPK to still be -decrypted. After that retention period, -`prekey_service.py:428-465` (`cleanup_old_signed_prekeys`) deletes them -permanently. +Old signed prekeys are kept around (with `is_active = False`) so +messages-in-flight that were encrypted against the previous SPK can +still complete their initial X3DH on the receiver. There is no +automated reaper in this codebase; pruning very old inactive SPKs is a +suggested extension in `04-CHALLENGES.md`. **One-Time Prekey (OPK) -- Single use, consumed and deleted** One-time prekeys are X25519 keypairs that are used exactly once and then discarded. They are generated in batches -(ref: `x3dh_manager.py:154-174` for generation, -`prekey_service.py:363-407` for batch replenishment) and uploaded to the +(ref: `frontend/src/crypto/x3dh.ts` for generation, +`backend/app/services/prekey_service.py` for batch replenishment) and uploaded to the server. When Alice initiates a conversation with Bob, the server gives Alice one of Bob's unused OPKs and marks it as consumed -(`prekey_service.py:331-338`). This OPK participates in the fourth DH +(`backend/app/services/prekey_service.py`). This OPK participates in the fourth DH operation (DH4) of the X3DH handshake. Because the OPK is used only once and then deleted, it provides an additional layer of forward secrecy specifically for the initial message of a conversation. @@ -287,7 +286,7 @@ other users initiating conversations), X3DH falls back to three DH operations instead of four. The protocol still works, but the initial message has slightly weaker forward secrecy because the fourth DH operation is skipped. The system generates 100 OPKs initially -(`config.py:75`, `DEFAULT_ONE_TIME_PREKEY_COUNT=100`) and replenishes +(`config.py`, `DEFAULT_ONE_TIME_PREKEY_COUNT=100`) and replenishes them when the supply drops below half. **Ephemeral Key (EK) -- Generated per session, never stored** @@ -297,11 +296,11 @@ sender) at the moment she initiates a conversation. It is used in DH2, DH3, and DH4 of the X3DH handshake. It is never stored on disk; it exists only in memory for the duration of the key exchange computation. -In the code, it is generated at `x3dh_manager.py:227-228`: +In the code, it is generated at `frontend/src/crypto/x3dh.ts`: ```python -alice_ek_private = X25519PrivateKey.generate() -alice_ek_public = alice_ek_private.public_key() +alice_ek_private = X25519PrivateKey.generate +alice_ek_public = alice_ek_private.public_key ``` After the shared secret is computed, Alice sends the ephemeral public @@ -319,34 +318,34 @@ The four secrets are concatenated and fed into HKDF to produce the final shared key. ``` -Alice (sender) has: Bob (receiver) has: - IK_A (identity private key) IK_B (identity public key) - EK_A (ephemeral, just generated) SPK_B (signed prekey public) - OPK_B (one-time prekey public) +Alice (sender) has: Bob (receiver) has: + IK_A (identity private key) IK_B (identity public key) + EK_A (ephemeral, just generated) SPK_B (signed prekey public) + OPK_B (one-time prekey public) DH Operations (each produces 32 bytes): - DH1 = X25519(IK_A_private, SPK_B_public) - Alice's identity x Bob's signed prekey + DH1 = X25519(IK_A_private, SPK_B_public) + Alice's identity x Bob's signed prekey - DH2 = X25519(EK_A_private, IK_B_public) - Alice's ephemeral x Bob's identity + DH2 = X25519(EK_A_private, IK_B_public) + Alice's ephemeral x Bob's identity - DH3 = X25519(EK_A_private, SPK_B_public) - Alice's ephemeral x Bob's signed prekey + DH3 = X25519(EK_A_private, SPK_B_public) + Alice's ephemeral x Bob's signed prekey - DH4 = X25519(EK_A_private, OPK_B_public) [optional] - Alice's ephemeral x Bob's one-time prekey + DH4 = X25519(EK_A_private, OPK_B_public) [optional] + Alice's ephemeral x Bob's one-time prekey Key Material Derivation: - input = 0xFF * 32 || DH1 || DH2 || DH3 || DH4 - salt = 0x00 * 32 - info = "X3DH" - SK = HKDF-SHA256(salt, input, info, length=32) + input = 0xFF * 32 || DH1 || DH2 || DH3 || DH4 + salt = 0x00 * 32 + info = "X3DH" + SK = HKDF-SHA256(salt, input, info, length=32) ``` -In the codebase, the sender side is at `x3dh_manager.py:241-264`: +In the codebase, the sender side is at `frontend/src/crypto/x3dh.ts`: - Line 241: `dh1 = alice_ik_private.exchange(bob_spk_public)` -- DH1 - Line 242: `dh2 = alice_ek_private.exchange(bob_ik_public)` -- DH2 @@ -355,15 +354,15 @@ In the codebase, the sender side is at `x3dh_manager.py:241-264`: - Line 252: `key_material = dh1 + dh2 + dh3 + dh4` -- concatenation - Lines 257-264: HKDF derivation with `0xFF * 32` prefix and `b'X3DH'` info string -The receiver side at `x3dh_manager.py:308-333` performs the same +The receiver side at `frontend/src/crypto/x3dh.ts` performs the same operations but with the roles reversed. DH1 becomes -`bob_spk_private.exchange(alice_ik_public)` (line 308), because Bob has +`bob_spk_private.exchange(alice_ik_public)` , because Bob has the SPK private key and Alice's IK public key. The property of Diffie-Hellman guarantees that `X25519(a_priv, B_pub)` produces the same result as `X25519(b_priv, A_pub)`, so both sides compute identical shared secrets. -The `0xFF * 32` prefix prepended at line 257 (`f = b'\xff' * X25519_KEY_SIZE`) +The `0xFF * 32` prefix prepended (`f = b'\xff' * X25519_KEY_SIZE`) is a fixed padding specified by the X3DH standard. It ensures the HKDF input is at least 32 bytes long even in edge cases and provides domain separation from other uses of the same keys. @@ -422,20 +421,20 @@ Before performing any DH operations, Alice must verify that Bob's signed prekey actually belongs to Bob. A malicious server could substitute its own SPK and intercept communications. -The verification happens at `x3dh_manager.py:217-220` inside +The verification happens at `frontend/src/crypto/x3dh.ts` inside `perform_x3dh_sender`: ```python if not self.verify_signed_prekey(bob_bundle.signed_prekey, - bob_bundle.signed_prekey_signature, - bob_identity_public_ed25519): - raise ValueError("Invalid signed prekey signature") + bob_bundle.signed_prekey_signature, + bob_identity_public_ed25519): + raise ValueError("Invalid signed prekey signature") ``` -The `verify_signed_prekey` method at `x3dh_manager.py:176-206` uses +The `verify_signed_prekey` method at `frontend/src/crypto/x3dh.ts` uses Ed25519 signature verification. It decodes the SPK public key bytes, the signature bytes, and the Ed25519 identity public key bytes, then calls -`identity_public.verify(signature_bytes, spk_public_bytes)` at line 196. +`identity_public.verify(signature_bytes, spk_public_bytes)` . Ed25519 verification either succeeds or raises `InvalidSignature`. If verification fails, the entire X3DH handshake is aborted. @@ -465,15 +464,15 @@ The Double Ratchet provides two critical security properties that go beyond what a static shared key could provide: 1. **Forward secrecy** -- Compromising a current key does not expose - past messages. Even if an attacker steals the current encryption key, - they cannot derive previous keys and therefore cannot decrypt earlier - messages. + past messages. Even if an attacker steals the current encryption key, + they cannot derive previous keys and therefore cannot decrypt earlier + messages. 2. **Post-compromise security (break-in recovery)** -- After a key - compromise, future messages become secure again once a new DH ratchet - step occurs. If an attacker temporarily gains access to key material, - they lose access to the conversation as soon as the keys advance - through a new Diffie-Hellman exchange. + compromise, future messages become secure again once a new DH ratchet + step occurs. If an attacker temporarily gains access to key material, + they lose access to the conversation as soon as the keys advance + through a new Diffie-Hellman exchange. The name "Double Ratchet" refers to the fact that it combines two ratcheting mechanisms: a **DH ratchet** (Diffie-Hellman ratchet) that @@ -487,18 +486,18 @@ the sending chain, and the receiving chain. ``` Root Chain (KDF_RK) - | - |-- [DH ratchet step] --> new root key + new chain key - | - +-- Sending Chain (KDF_CK) - | |-- advance --> Message Key 1 --> Encrypt msg 1 - | |-- advance --> Message Key 2 --> Encrypt msg 2 - | +-- advance --> Message Key 3 --> Encrypt msg 3 - | - +-- Receiving Chain (KDF_CK) - |-- advance --> Message Key 1 --> Decrypt msg 1 - |-- advance --> Message Key 2 --> Decrypt msg 2 - +-- advance --> Message Key 3 --> Decrypt msg 3 + | + |-- [DH ratchet step] --> new root key + new chain key + | + +-- Sending Chain (KDF_CK) + | |-- advance --> Message Key 1 --> Encrypt msg 1 + | |-- advance --> Message Key 2 --> Encrypt msg 2 + | +-- advance --> Message Key 3 --> Encrypt msg 3 + | + +-- Receiving Chain (KDF_CK) + |-- advance --> Message Key 1 --> Decrypt msg 1 + |-- advance --> Message Key 2 --> Decrypt msg 2 + +-- advance --> Message Key 3 --> Decrypt msg 3 ``` Each chain is a sequence of key derivation operations. The root chain @@ -515,20 +514,20 @@ The root chain advances during a DH ratchet step. It takes the current root key and a fresh DH output (from a new DH key exchange) and produces a new root key and a new chain key. -Reference: `double_ratchet.py:79-94` +Reference: `frontend/src/crypto/double-ratchet.ts` ```python def _kdf_rk(self, root_key: bytes, dh_output: bytes) -> tuple[bytes, bytes]: - hkdf = HKDF( - algorithm = hashes.SHA256(), - length = HKDF_OUTPUT_SIZE * 2, # 64 bytes total - salt = root_key, # current root key as salt - info = b'', - ) - output = hkdf.derive(dh_output) - new_root_key = output[: HKDF_OUTPUT_SIZE] # first 32 bytes - new_chain_key = output[HKDF_OUTPUT_SIZE :] # last 32 bytes - return new_root_key, new_chain_key + hkdf = HKDF( + algorithm = hashes.SHA256, + length = HKDF_OUTPUT_SIZE * 2, # 64 bytes total + salt = root_key, # current root key as salt + info = b'', + ) + output = hkdf.derive(dh_output) + new_root_key = output[: HKDF_OUTPUT_SIZE] # first 32 bytes + new_chain_key = output[HKDF_OUTPUT_SIZE :] # last 32 bytes + return new_root_key, new_chain_key ``` HKDF-SHA256 is used with the current root key as the salt and the DH @@ -543,19 +542,19 @@ and the message chains. The symmetric chains advance with every message. Each step takes the current chain key and produces the next chain key and a message key. -Reference: `double_ratchet.py:96-109` +Reference: `frontend/src/crypto/double-ratchet.ts` ```python def _kdf_ck(self, chain_key: bytes) -> tuple[bytes, bytes]: - h_chain = hmac.HMAC(chain_key, hashes.SHA256()) - h_chain.update(b'\x01') - next_chain_key = h_chain.finalize() + h_chain = hmac.HMAC(chain_key, hashes.SHA256) + h_chain.update(b'\x01') + next_chain_key = h_chain.finalize - h_message = hmac.HMAC(chain_key, hashes.SHA256()) - h_message.update(b'\x02') - message_key = h_message.finalize() + h_message = hmac.HMAC(chain_key, hashes.SHA256) + h_message.update(b'\x02') + message_key = h_message.finalize - return next_chain_key, message_key + return next_chain_key, message_key ``` Two separate HMAC-SHA256 computations are performed using the same chain @@ -563,8 +562,8 @@ key but with different constants: ``` chain_key --+-- HMAC(chain_key, 0x01) --> next_chain_key (kept for future) - | - +-- HMAC(chain_key, 0x02) --> message_key (used once, discarded) + | + +-- HMAC(chain_key, 0x02) --> message_key (used once, discarded) ``` The use of different constants (0x01 and 0x02) is essential. If the same @@ -598,18 +597,18 @@ KDF_RK. Message flow and DH ratchet steps: Alice sends msgs 1,2,3 using DH keypair A1: - A1 --> msg1(mk1), msg2(mk2), msg3(mk3) - [symmetric ratchet advances 3 times, same DH key] + A1 --> msg1(mk1), msg2(mk2), msg3(mk3) + [symmetric ratchet advances 3 times, same DH key] Bob receives, generates new DH keypair B1, sends reply: - DH ratchet: root_key' = KDF_RK(root_key, DH(B1_priv, A1_pub)) - B1 --> msg4(mk1'), msg5(mk2') - [new chain, new keys, A1 compromise no longer helps] + DH ratchet: root_key' = KDF_RK(root_key, DH(B1_priv, A1_pub)) + B1 --> msg4(mk1'), msg5(mk2') + [new chain, new keys, A1 compromise no longer helps] Alice receives, generates new DH keypair A2, sends reply: - DH ratchet: root_key'' = KDF_RK(root_key', DH(A2_priv, B1_pub)) - A2 --> msg6(mk1''), msg7(mk2'') - [new chain again, B1 compromise no longer helps] + DH ratchet: root_key'' = KDF_RK(root_key', DH(A2_priv, B1_pub)) + A2 --> msg6(mk1''), msg7(mk2'') + [new chain again, B1 compromise no longer helps] ``` Each DH ratchet step introduces fresh random entropy (from the newly @@ -618,20 +617,20 @@ even if an attacker had compromised all previous key material, the new DH exchange produces a shared secret they cannot predict, and all subsequent keys are secure again. -The implementation spans `double_ratchet.py:155-213`: +The implementation spans `frontend/src/crypto/double-ratchet.ts`: -- `_dh_ratchet_send` (lines 155-176): Called when the sender needs to - advance the ratchet. Generates a new DH keypair at line 159, performs - DH with the peer's public key at line 169, and derives new root and - sending chain keys at lines 171-174. +- `_dh_ratchet_send` : Called when the sender needs to + advance the ratchet. Generates a new DH keypair , performs + DH with the peer's public key , and derives new root and + sending chain keys . -- `_dh_ratchet_receive` (lines 178-213): Called when a received message - contains a new DH public key. Updates the peer public key at line 189, - performs DH with the existing private key to derive a new receiving - chain key at lines 195-198, then generates a new private key at line - 200 and performs another DH to derive a new sending chain key at lines - 208-211. This double DH step on the receiver side ensures both - receiving and sending chains are updated. +- `_dh_ratchet_receive` : Called when a received message + contains a new DH public key. Updates the peer public key , + performs DH with the existing private key to derive a new receiving + chain key , then generates a new private key at line + 200 and performs another DH to derive a new sending chain key at lines + 208-211. This double DH step on the receiver side ensures both + receiving and sending chains are updated. ### Out-of-Order Message Handling @@ -644,17 +643,17 @@ Internet messages can arrive out of order. If Alice sends messages 1, 2, 4. Derive mk3 and decrypt message 3 The skipped message key mechanism handles this. Reference: -`double_ratchet.py:215-277`. +`frontend/src/crypto/double-ratchet.ts`. -`_store_skipped_message_keys` (lines 215-244) is called when the +`_store_skipped_message_keys` is called when the received message number is greater than the expected message number. It iterates through the gap, deriving and caching each skipped message key: ```python chain_key = state.receiving_chain_key for msg_num in range(state.receiving_message_number, until_message_number): - chain_key, message_key = self._kdf_ck(chain_key) - state.skipped_message_keys[(dh_public_key, msg_num)] = message_key + chain_key, message_key = self._kdf_ck(chain_key) + state.skipped_message_keys[(dh_public_key, msg_num)] = message_key state.receiving_chain_key = chain_key ``` @@ -663,24 +662,24 @@ message_number)`. This tuple key is necessary because message numbers reset with each DH ratchet step: message 0 under DH key A1 is different from message 0 under DH key A2. -`_try_skipped_message_key` (lines 260-277) checks whether a received +`_try_skipped_message_key` checks whether a received message matches a previously cached skipped key. If it does, the cached key is used for decryption and then removed from the cache (it is -consumed by `dict.pop()` at line 270). +consumed by `dict.pop` ). Security limits prevent abuse. An attacker who sends messages with enormous message numbers could force the ratchet to derive and store millions of keys, exhausting memory. Two limits are enforced: -- `MAX_SKIP_MESSAGE_KEYS = 1000` (`config.py:73`): No more than 1000 - message keys can be skipped in a single gap. If a message arrives - claiming to be message number 5000 when we expect message 0, the - decryption is rejected at lines 226-230. +- `MAX_SKIP_MESSAGE_KEYS = 1000` (`config.py`): No more than 1000 + message keys can be skipped in a single gap. If a message arrives + claiming to be message number 5000 when we expect message 0, the + decryption is rejected . -- `MAX_CACHED_MESSAGE_KEYS = 2000` (`config.py:74`): The total number - of cached skipped keys across all ratchet epochs. If the cache is - full, the oldest keys are evicted at lines 232-234 via - `_evict_oldest_skipped_keys`. +- `MAX_CACHED_MESSAGE_KEYS = 2000` (`config.py`): The total number + of cached skipped keys across all ratchet epochs. If the cache is + full, the oldest keys are evicted via + `_evict_oldest_skipped_keys`. ### Forward Secrecy Proof @@ -691,25 +690,25 @@ Assume at message N, an attacker steals the current chain_key_N. ``` What the attacker CAN compute (forward direction): - chain_key_N -----> HMAC(chain_key_N, 0x01) = chain_key_N+1 - chain_key_N+1 ---> HMAC(chain_key_N+1, 0x01) = chain_key_N+2 - ... and so on for all future chain keys + chain_key_N -----> HMAC(chain_key_N, 0x01) = chain_key_N+1 + chain_key_N+1 ---> HMAC(chain_key_N+1, 0x01) = chain_key_N+2 + ... and so on for all future chain keys What the attacker CANNOT compute (backward direction): - chain_key_N <-/-- chain_key_N-1 + chain_key_N <-/-- chain_key_N-1 - Why? Because HMAC is a one-way function. + Why? Because HMAC is a one-way function. - chain_key_N = HMAC(chain_key_N-1, 0x01) + chain_key_N = HMAC(chain_key_N-1, 0x01) - Given chain_key_N, you cannot solve for chain_key_N-1. - This would require inverting HMAC-SHA256, which is - computationally infeasible (preimage resistance). + Given chain_key_N, you cannot solve for chain_key_N-1. + This would require inverting HMAC-SHA256, which is + computationally infeasible (preimage resistance). Therefore: - message_key_N-1 = HMAC(chain_key_N-1, 0x02) <-- UNREACHABLE - message_key_N-2 = HMAC(chain_key_N-2, 0x02) <-- UNREACHABLE - message_key_1 = HMAC(chain_key_1, 0x02) <-- UNREACHABLE + message_key_N-1 = HMAC(chain_key_N-1, 0x02) <-- UNREACHABLE + message_key_N-2 = HMAC(chain_key_N-2, 0x02) <-- UNREACHABLE + message_key_1 = HMAC(chain_key_1, 0x02) <-- UNREACHABLE ``` The attacker can decrypt messages N+1, N+2, N+3, and so on (until the @@ -752,46 +751,46 @@ The encryption flow is: 1. The Double Ratchet derives a message key (32 bytes) via KDF_CK 2. A random 12-byte nonce is generated using `os.urandom` (backend) or - `crypto.getRandomValues` (frontend) + `crypto.getRandomValues` (frontend) 3. AES-256-GCM encrypts the plaintext using the message key and nonce 4. Associated data (sender and recipient identifiers) is authenticated - but not encrypted + but not encrypted 5. The output is ciphertext + a 16-byte authentication tag (GCM appends - the tag to the ciphertext) + the tag to the ciphertext) -Backend implementation at `double_ratchet.py:111-130`: +Backend implementation at `frontend/src/crypto/double-ratchet.ts`: ```python def _encrypt_with_message_key(self, message_key, plaintext, associated_data): - aesgcm = AESGCM(message_key) - nonce = os.urandom(AES_GCM_NONCE_SIZE) # 12 bytes from config.py:69 - ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data) - return nonce, ciphertext + aesgcm = AESGCM(message_key) + nonce = os.urandom(AES_GCM_NONCE_SIZE) # 12 bytes from config.py + ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data) + return nonce, ciphertext ``` -Backend decryption at `double_ratchet.py:132-153` catches `InvalidTag` -exceptions (line 151), which indicate that the ciphertext was tampered +Backend decryption at `frontend/src/crypto/double-ratchet.ts` catches `InvalidTag` +exceptions , which indicate that the ciphertext was tampered with, the wrong key was used, or the associated data does not match. The error is re-raised as `ValueError("Message tampered or corrupted")` -at line 153. +. -Frontend implementation at `primitives.ts:224-259` uses the WebCrypto +Frontend implementation at `frontend/src/crypto/primitives.ts` uses the WebCrypto API: ```typescript const nonce = generateRandomBytes(AES_GCM_NONCE_SIZE) const ciphertext = await subtle.encrypt( - { - name: "AES-GCM", - iv: nonce.buffer, - additionalData: associatedData?.buffer, - }, - cryptoKey, - plaintext.buffer + { + name: "AES-GCM", + iv: nonce.buffer, + additionalData: associatedData?.buffer, + }, + cryptoKey, + plaintext.buffer ) ``` -Frontend decryption at `primitives.ts:261-292` mirrors this with +Frontend decryption at `frontend/src/crypto/primitives.ts` mirrors this with `subtle.decrypt`. The WebCrypto API throws a `DOMException` if authentication fails, which is functionally equivalent to the Python `InvalidTag` exception. @@ -817,12 +816,12 @@ security reason to prefer CBC over GCM for new implementations. ### Parameters -As defined in `config.py:68-70`: +As defined in `config.py`: ``` -AES_GCM_KEY_SIZE = 32 (256 bits) line 68 -AES_GCM_NONCE_SIZE = 12 (96 bits) line 69 -HKDF_OUTPUT_SIZE = 32 (256 bits) line 70 +AES_GCM_KEY_SIZE = 32 (256 bits) +AES_GCM_NONCE_SIZE = 12 (96 bits) +HKDF_OUTPUT_SIZE = 32 (256 bits) ``` The 12-byte (96-bit) nonce is the recommended size for GCM. Longer @@ -913,44 +912,44 @@ and you cannot forge a signature without the private key. ``` Step 1: Client requests registration options - Client ----> POST /auth/register/begin ----> Server + Client ----> POST /auth/register/begin ----> Server - Server: - - Generates 32-byte random challenge (secrets.token_bytes) - - Stores challenge in Redis with 10-minute TTL - - Returns WebAuthn PublicKeyCredentialCreationOptions + Server: + - Generates 32-byte random challenge (secrets.token_bytes) + - Stores challenge in Redis with 10-minute TTL + - Returns WebAuthn PublicKeyCredentialCreationOptions Step 2: Browser creates credential - Browser ----> navigator.credentials.create(options) ----> Authenticator + Browser ----> navigator.credentials.create(options) ----> Authenticator - Authenticator: - - Prompts user for biometric/PIN verification - - Generates new ECDSA or EdDSA keypair - - Stores private key internally (NEVER exported) - - Returns attestation object (signed credential public key) + Authenticator: + - Prompts user for biometric/PIN verification + - Generates new ECDSA or EdDSA keypair + - Stores private key internally (NEVER exported) + - Returns attestation object (signed credential public key) Step 3: Client sends attestation for verification - Client ----> POST /auth/register/complete ----> Server + Client ----> POST /auth/register/complete ----> Server - Server: - - Verifies attestation signature - - Verifies challenge matches stored value - - Extracts credential public key and credential ID - - Stores PUBLIC KEY + credential ID in PostgreSQL - - Deletes challenge from Redis - - Private key STAYS on authenticator -- server never sees it + Server: + - Verifies attestation signature + - Verifies challenge matches stored value + - Extracts credential public key and credential ID + - Stores PUBLIC KEY + credential ID in PostgreSQL + - Deletes challenge from Redis + - Private key STAYS on authenticator -- server never sees it ``` -Implementation reference: `passkey_manager.py:55-94` -(`generate_registration_options`). At line 65, a 32-byte challenge is +Implementation reference: `backend/app/core/passkey/passkey_manager.py` +(`generate_registration_options`). a 32-byte challenge is generated: `challenge = secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)`. At lines 74-87, the WebAuthn options are constructed with RP configuration, user information, and authenticator requirements. The -authenticator selection at lines 82-85 specifies +authenticator selection specifies `ResidentKeyRequirement.REQUIRED`, which forces creation of a discoverable credential (passkey). -Registration verification at `passkey_manager.py:96-130` +Registration verification at `backend/app/core/passkey/passkey_manager.py` (`verify_registration`) calls `verify_registration_response` at lines 105-110, which validates the attestation object, checks the challenge, verifies the RP ID, and confirms the origin. @@ -959,40 +958,40 @@ verifies the RP ID, and confirms the origin. ``` Step 1: Client requests authentication options - Client ----> POST /auth/authenticate/begin ----> Server + Client ----> POST /auth/authenticate/begin ----> Server - Server: - - Generates new 32-byte challenge - - Stores challenge in Redis with 10-minute TTL - - Returns WebAuthn PublicKeyCredentialRequestOptions + Server: + - Generates new 32-byte challenge + - Stores challenge in Redis with 10-minute TTL + - Returns WebAuthn PublicKeyCredentialRequestOptions Step 2: Browser signs challenge - Browser ----> navigator.credentials.get(options) ----> Authenticator + Browser ----> navigator.credentials.get(options) ----> Authenticator - Authenticator: - - Prompts user for biometric/PIN - - Signs challenge with stored private key - - Increments signature counter - - Returns assertion (signed challenge + counter) + Authenticator: + - Prompts user for biometric/PIN + - Signs challenge with stored private key + - Increments signature counter + - Returns assertion (signed challenge + counter) Step 3: Client sends assertion for verification - Client ----> POST /auth/authenticate/complete ----> Server + Client ----> POST /auth/authenticate/complete ----> Server - Server: - - Retrieves stored public key from PostgreSQL - - Verifies signature using stored public key - - Verifies challenge matches stored value - - Checks signature counter INCREASED (clone detection!) - - Updates stored counter value - - Returns authenticated session + Server: + - Retrieves stored public key from PostgreSQL + - Verifies signature using stored public key + - Verifies challenge matches stored value + - Checks signature counter INCREASED (clone detection!) + - Updates stored counter value + - Returns authenticated session ``` -Implementation reference: `passkey_manager.py:132-160` -(`generate_authentication_options`). At line 139, a fresh challenge is +Implementation reference: `backend/app/core/passkey/passkey_manager.py` +(`generate_authentication_options`). a fresh challenge is generated. At lines 148-153, WebAuthn authentication options are constructed. -Authentication verification at `passkey_manager.py:162-207` +Authentication verification at `backend/app/core/passkey/passkey_manager.py` (`verify_authentication`). At lines 173-180, the assertion is verified against the expected challenge, RP ID, origin, and stored credential public key. The critical clone detection check follows. @@ -1009,26 +1008,26 @@ If the server receives an assertion with a counter value that has not increased (or has decreased), it indicates one of two things: 1. The authenticator hardware was cloned (its key material was - extracted and loaded onto a second device) + extracted and loaded onto a second device) 2. A replay attack is being attempted Both scenarios are security incidents that warrant blocking authentication and alerting the user. -Reference: `passkey_manager.py:184-193`: +Reference: `backend/app/core/passkey/passkey_manager.py`: ```python if (credential_current_sign_count != 0 and new_sign_count != 0 - and new_sign_count <= credential_current_sign_count): - logger.error( - "Signature counter did not increase: current=%s, new=%s. " - "Possible cloned authenticator detected!", - credential_current_sign_count, - new_sign_count - ) - raise ValueError( - "Signature counter anomaly detected - potential cloned authenticator" - ) + and new_sign_count <= credential_current_sign_count): + logger.error( + "Signature counter did not increase: current=%s, new=%s. " + "Possible cloned authenticator detected!", + credential_current_sign_count, + new_sign_count + ) + raise ValueError( + "Signature counter anomaly detected - potential cloned authenticator" + ) ``` The conditions `credential_current_sign_count != 0` and @@ -1043,17 +1042,17 @@ support signature counters. ## Constant-Time Comparison -A detail worth calling out: `primitives.ts:388-397` implements a +A detail worth calling out: `frontend/src/crypto/primitives.ts` implements a constant-time byte array comparison: ```typescript export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean { - if (a.length !== b.length) return false - let result = 0 - for (let i = 0; i < a.length; i++) { - result |= a[i] ^ b[i] - } - return result === 0 + if (a.length !== b.length) return false + let result = 0 + for (let i = 0; i < a.length; i++) { + result |= a[i] ^ b[i] + } + return result === 0 } ``` @@ -1080,72 +1079,72 @@ the full security architecture of the application: ``` +-----------------------------------------------------------+ -| AUTHENTICATION LAYER | -| | -| WebAuthn/Passkeys | -| | | -| +--> User identity established | -| +--> No password to steal, phish, or brute force | -| +--> Clone detection via signature counter | -| | | -| v | -| +----------------------------------------------------+ | -| | KEY AGREEMENT LAYER | | -| | | | -| | X3DH Key Exchange | | -| | +--> Asynchronous (works when peer is offline) | | -| | +--> 4 DH operations for mutual authentication | | -| | +--> Produces initial shared secret (32 bytes) | | -| | | | | -| | v | | -| | Double Ratchet Initialization | | -| | +--> Shared secret becomes root key | | -| | +--> Sending and receiving chains created | | -| +----------------------------------------------------+ | -| | | -| v | -| +----------------------------------------------------+ | -| | MESSAGE ENCRYPTION LAYER | | -| | | | -| | Double Ratchet (ongoing) | | -| | +--> KDF_CK derives per-message keys | | -| | +--> DH ratchet provides post-compromise security| | -| | +--> Skipped key cache handles out-of-order msgs | | -| | | | | -| | v | | -| | AES-256-GCM | | -| | +--> Encrypts plaintext with message key | | -| | +--> Random 12-byte nonce per message | | -| | +--> Authentication tag detects tampering | | -| +----------------------------------------------------+ | -| | | -| v | -| +----------------------------------------------------+ | -| | TRANSPORT LAYER | | -| | | | -| | WebSocket (real-time delivery) | | -| | +--> Carries encrypted blobs between clients | | -| | +--> Server sees only ciphertext + metadata | | -| | | | -| | SurrealDB (persistence) | | -| | +--> Stores encrypted messages at rest | | -| | +--> No decryption capability on server | | -| +----------------------------------------------------+ | +| AUTHENTICATION LAYER | +| | +| WebAuthn/Passkeys | +| | | +| +--> User identity established | +| +--> No password to steal, phish, or brute force | +| +--> Clone detection via signature counter | +| | | +| v | +| +----------------------------------------------------+ | +| | KEY AGREEMENT LAYER | | +| | | | +| | X3DH Key Exchange | | +| | +--> Asynchronous (works when peer is offline) | | +| | +--> 4 DH operations for mutual authentication | | +| | +--> Produces initial shared secret (32 bytes) | | +| | | | | +| | v | | +| | Double Ratchet Initialization | | +| | +--> Shared secret becomes root key | | +| | +--> Sending and receiving chains created | | +| +----------------------------------------------------+ | +| | | +| v | +| +----------------------------------------------------+ | +| | MESSAGE ENCRYPTION LAYER | | +| | | | +| | Double Ratchet (ongoing) | | +| | +--> KDF_CK derives per-message keys | | +| | +--> DH ratchet provides post-compromise security| | +| | +--> Skipped key cache handles out-of-order msgs | | +| | | | | +| | v | | +| | AES-256-GCM | | +| | +--> Encrypts plaintext with message key | | +| | +--> Random 12-byte nonce per message | | +| | +--> Authentication tag detects tampering | | +| +----------------------------------------------------+ | +| | | +| v | +| +----------------------------------------------------+ | +| | TRANSPORT LAYER | | +| | | | +| | WebSocket (real-time delivery) | | +| | +--> Carries encrypted blobs between clients | | +| | +--> Server sees only ciphertext + metadata | | +| | | | +| | SurrealDB (persistence) | | +| | +--> Stores encrypted messages at rest | | +| | +--> No decryption capability on server | | +| +----------------------------------------------------+ | +-----------------------------------------------------------+ ``` The layers interact in a strict top-down sequence for new conversations: 1. The user authenticates with WebAuthn (proving their identity without - a password) + a password) 2. X3DH establishes a shared secret with the peer (even if the peer is - offline) + offline) 3. The shared secret initializes the Double Ratchet - (`double_ratchet.py:279-302`) + (`frontend/src/crypto/double-ratchet.ts`) 4. Each message is encrypted with a unique AES-256-GCM key derived from - the ratchet (`double_ratchet.py:323-362`) + the ratchet (`frontend/src/crypto/double-ratchet.ts`) 5. The encrypted message is transmitted via WebSocket and stored in - SurrealDB (`message_service.py:269-314`) + SurrealDB (`backend/app/services/message_service.py`) For ongoing conversations, only steps 4 and 5 repeat. The X3DH handshake happens once per conversation. The Double Ratchet then runs @@ -1224,11 +1223,11 @@ well-tested implementations. 112 bits for use through 2030+ (NIST SP 800-57 Part 1). **CWE-330: Use of Insufficiently Random Values** -- Randomness comes -from two sources: `os.urandom()` on the backend (which reads from the +from two sources: `os.urandom` on the backend (which reads from the operating system's CSPRNG -- `/dev/urandom` on Linux) and -`crypto.getRandomValues()` on the frontend (which uses the browser's +`crypto.getRandomValues` on the frontend (which uses the browser's CSPRNG). Both are cryptographically secure. Nonce generation at -`double_ratchet.py:122` and `primitives.ts:294-297` use these +`frontend/src/crypto/double-ratchet.ts` and `frontend/src/crypto/primitives.ts` use these exclusively. **CWE-311: Missing Encryption of Sensitive Data** -- All message content @@ -1268,20 +1267,20 @@ cryptography textbook. server never generates or holds encryption keys for message content. Key generation happens in two places: -1. On the backend, `x3dh_manager.py:60-86` generates X25519 identity - keypairs using `X25519PrivateKey.generate()`, which calls into - OpenSSL's random number generator. These keys are for the X3DH - protocol, and the private keys are stored in the database for the - server-side key exchange path. +1. On the backend, `frontend/src/crypto/x3dh.ts` generates X25519 identity + keypairs using `X25519PrivateKey.generate`, which calls into + OpenSSL's random number generator. These keys are for the X3DH + protocol, and the private keys are stored in the database for the + server-side key exchange path. -2. On the frontend, `primitives.ts:15-24` generates X25519 keypairs - using the WebCrypto API (`subtle.generateKey`), which uses the - browser's hardware-backed CSPRNG. In the client-side encryption - model, these keys never leave the browser. +2. On the frontend, `frontend/src/crypto/primitives.ts` generates X25519 keypairs + using the WebCrypto API (`subtle.generateKey`), which uses the + browser's hardware-backed CSPRNG. In the client-side encryption + model, these keys never leave the browser. -The `store_encrypted_message` function (`message_service.py:269-314`) +The `store_encrypted_message` function (`backend/app/services/message_service.py`) receives pre-encrypted ciphertext from the client and stores it directly -in SurrealDB at lines 292-302 without any server-side decryption. The +in SurrealDB without any server-side decryption. The server's role is explicitly that of a blind relay. Even if the entire server infrastructure were compromised, the attacker would obtain only encrypted blobs with no corresponding decryption keys. @@ -1302,7 +1301,7 @@ Ratchet combination used in this project. Each message gets a unique AES key through the ratchet mechanism, providing forward secrecy across billions of daily messages. WhatsApp's implementation stores prekey bundles on their servers (just as this project does via -`prekey_service.py:293-361`), allowing asynchronous session +`backend/app/services/prekey_service.py`), allowing asynchronous session establishment. The X3DH handshake is performed when a user initiates a new conversation, and the Double Ratchet runs continuously thereafter. @@ -1315,12 +1314,12 @@ same situation occurred with FBI requests in the United States and government demands in India and the UK. **Connection to this project.** The X3DH implementation at -`x3dh_manager.py:208-281` and the Double Ratchet at -`double_ratchet.py:279-416` implement the same cryptographic operations +`frontend/src/crypto/x3dh.ts` and the Double Ratchet at +`frontend/src/crypto/double-ratchet.ts` implement the same cryptographic operations described in the Signal Protocol specification. The same four DH -operations (lines 241-252), the same HKDF derivation (lines 257-264), -the same KDF chain operations (lines 79-109 of double_ratchet.py), and -the same skipped message key mechanism (lines 215-277). The protocol +operations , the same HKDF derivation , +the same KDF chain operations (lines 79-109 of frontend/src/crypto/double-ratchet.ts), and +the same skipped message key mechanism . The protocol specifications are public, the formal security proofs are published, and the implementation follows them directly. @@ -1351,7 +1350,7 @@ authentication, even when the passwords are used to derive encryption keys. In this project's WebAuthn implementation -(`passkey_manager.py:55-94`), there is no password. The user +(`backend/app/core/passkey/passkey_manager.py`), there is no password. The user authenticates with a biometric or PIN on their authenticator device. The authenticator holds an ECDSA or EdDSA private key that is hardware-bound and never exported. If the server database is fully compromised, the @@ -1375,27 +1374,27 @@ answer these questions. If you cannot answer one confidently, re-read the relevant section. 1. **Why does X3DH need four separate DH operations instead of just - one?** What specific security property does each operation provide? - What attack becomes possible if any single operation is removed? + one?** What specific security property does each operation provide? + What attack becomes possible if any single operation is removed? 2. **If an attacker compromises a Double Ratchet chain key at message N, - which messages can they decrypt?** Which messages remain protected? - When does the attacker lose access? Trace through the KDF_CK function - to prove your answer. + which messages can they decrypt?** Which messages remain protected? + When does the attacker lose access? Trace through the KDF_CK function + to prove your answer. 3. **Why is WebAuthn resistant to phishing attacks, while traditional - passwords are not?** What property of the WebAuthn protocol prevents - a fake login page from capturing usable credentials? Hint: think - about what the authenticator checks before signing the challenge. + passwords are not?** What property of the WebAuthn protocol prevents + a fake login page from capturing usable credentials? Hint: think + about what the authenticator checks before signing the challenge. 4. **What happens if Alice sends Bob messages 1, 2, 3, 4, 5 and Bob - receives them in order 1, 3, 5, 2, 4?** Walk through the skipped - message key mechanism for each received message. How many keys are - cached after processing message 3? After processing message 5? + receives them in order 1, 3, 5, 2, 4?** Walk through the skipped + message key mechanism for each received message. How many keys are + cached after processing message 3? After processing message 5? 5. **Why does the Double Ratchet use two different HMAC constants (0x01 - and 0x02) in KDF_CK?** What would go wrong if both the chain key and - the message key were derived with the same constant? + and 0x02) in KDF_CK?** What would go wrong if both the chain key and + the message key were derived with the same constant? ## Further Reading @@ -1410,58 +1409,58 @@ relevant section. ### Academic Analysis - "A Formal Security Analysis of the Signal Messaging Protocol" -- - Cohn-Gordon, Cremers, Dowling, Garratt, Stebila. IEEE EuroS&P 2017. - Formal proof that the Signal Protocol meets its claimed security - properties (authenticated key exchange, forward secrecy, post-compromise - security) under the Gap-DH assumption. + Cohn-Gordon, Cremers, Dowling, Garratt, Stebila. IEEE EuroS&P 2017. + Formal proof that the Signal Protocol meets its claimed security + properties (authenticated key exchange, forward secrecy, post-compromise + security) under the Gap-DH assumption. - "The Signal Protocol: A Cryptographic Analysis" -- Cohn-Gordon et al. - (2017). Extended version with proofs covering the X3DH and Double - Ratchet components individually and composed. + (2017). Extended version with proofs covering the X3DH and Double + Ratchet components individually and composed. - "On Ends-to-Ends Encryption: Asynchronous Group Messaging with Strong - Security Guarantees" -- Cohn-Gordon et al. IEEE S&P 2018. Extends the - analysis to group messaging scenarios. + Security Guarantees" -- Cohn-Gordon et al. IEEE S&P 2018. Extends the + analysis to group messaging scenarios. ### FIDO and WebAuthn - FIDO2 technical overview: https://fidoalliance.org/fido2/ - FIDO Alliance whitepaper on passkeys: - https://fidoalliance.org/passkeys/ + https://fidoalliance.org/passkeys/ - NIST SP 800-63B Digital Identity Guidelines (authenticator types and - assurance levels) + assurance levels) ### Cryptographic Primitives - Daniel J. Bernstein, "Curve25519: new Diffie-Hellman speed records" - (2006) -- The paper introducing the X25519 function used for DH in - this project. + (2006) -- The paper introducing the X25519 function used for DH in + this project. - Daniel J. Bernstein, Niels Duif, Tanja Lange, Peter Schwabe, Bo-Yin - Yang, "High-speed high-security signatures" (2012) -- The paper - introducing Ed25519 used for signing prekeys. + Yang, "High-speed high-security signatures" (2012) -- The paper + introducing Ed25519 used for signing prekeys. - NIST SP 800-38D, "Recommendation for Block Cipher Modes of Operation: - Galois/Counter Mode (GCM) and GMAC" -- The specification for the - AES-GCM mode used for message encryption. + Galois/Counter Mode (GCM) and GMAC" -- The specification for the + AES-GCM mode used for message encryption. - Hugo Krawczyk, "Cryptographic Extraction and Key Derivation: The HKDF - Scheme" (2010) -- The paper behind HKDF, the key derivation function - used throughout the Double Ratchet. + Scheme" (2010) -- The paper behind HKDF, the key derivation function + used throughout the Double Ratchet. ### Historical Context - Nikita Borisov, Ian Goldberg, Eric Brewer, "Off-the-Record - Communication, or, Why Not To Use PGP" (2004) -- The OTR protocol - that introduced the concept of deniable, forward-secret messaging and - directly inspired the Signal Protocol's design. + Communication, or, Why Not To Use PGP" (2004) -- The OTR protocol + that introduced the concept of deniable, forward-secret messaging and + directly inspired the Signal Protocol's design. - Whitfield Diffie and Martin Hellman, "New Directions in Cryptography" - (1976) -- The original paper introducing public key cryptography and - the Diffie-Hellman key exchange that underlies X3DH. + (1976) -- The original paper introducing public key cryptography and + the Diffie-Hellman key exchange that underlies X3DH. - Phil Zimmermann and PGP: The "Crypto Wars" of the 1990s, where the - US government attempted to restrict the export of strong cryptography. - Zimmermann published PGP's source code in a printed book to circumvent - export controls under First Amendment protection. This history is why - cryptographic software can be freely distributed today. + US government attempted to restrict the export of strong cryptography. + Zimmermann published PGP's source code in a printed book to circumvent + export controls under First Amendment protection. This history is why + cryptographic software can be freely distributed today. diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md index d87f8d60..6c56d9a3 100644 --- a/PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md @@ -9,57 +9,57 @@ Everything that follows is grounded in the actual source code. File references u ## High-Level Architecture ``` - HTTPS (TLS) - ┌────────────────────────────────────────────────────────────┐ - │ Client Browser │ - │ ┌──────────────────────────────────────────────────────┐ │ - │ │ SolidJS 1.9 + TypeScript 5.9 │ │ - │ │ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │ │ - │ │ │ WebCrypto │ │ IndexedDB │ │ nanostores │ │ │ - │ │ │ X25519 │ │ Private │ │ Reactive │ │ │ - │ │ │ Ed25519 │ │ Keys │ │ State │ │ │ - │ │ │ AES-256-GCM │ │ Ratchet │ │ │ │ │ - │ │ │ HKDF-SHA256 │ │ States │ │ │ │ │ - │ │ └──────────────┘ └──────────────┘ └────────────┘ │ │ - │ └──────────────────────────┬───────────────────────────┘ │ - │ │ │ - └──────────────────────────────┼──────────────────────────────┘ - │ - HTTPS + WSS │ - │ - ┌──────────────────────────────┼──────────────────────────────┐ - │ Nginx Reverse Proxy │ │ - │ ┌───────────────────────────┴──────────────────────────┐ │ - │ │ /api/* ──────► HTTP ──────► FastAPI :8000 │ │ - │ │ /ws ──────► WS ──────► FastAPI :8000 │ │ - │ │ /* ──────► Static files (SolidJS build) │ │ - │ └──────────────────────────────────────────────────────┘ │ - └──────────────────────────────┬──────────────────────────────┘ - │ - ┌──────────────────────────────┼──────────────────────────────┐ - │ FastAPI Backend │ │ - │ (Python 3.13, ORJSONResponse, async/await throughout) │ - │ ┌──────────────────────────────────────────────────────┐ │ - │ │ Routers: auth, rooms, encryption, websocket │ │ - │ │ Services: auth, prekey, message, presence, websocket│ │ - │ │ Core: x3dh, double_ratchet, passkey, surreal, redis │ │ - │ └────────┬──────────────┬──────────────┬───────────────┘ │ - │ │ │ │ │ - └───────────┼──────────────┼──────────────┼───────────────────┘ - │ │ │ - asyncpg │ WebSocket │ redis. │ - (TCP 5432) │ (WS 8000) │ asyncio │ - │ │ (TCP 6379) │ - ┌───────────┴──┐ ┌───────┴──────┐ ┌───┴────────────┐ - │ PostgreSQL │ │ SurrealDB │ │ Redis 8 │ - │ 16-alpine │ │ (latest) │ │ (alpine) │ - │ │ │ │ │ │ - │ Auth data │ │ Messages │ │ Challenges │ - │ Credentials │ │ Presence │ │ Rate limits │ - │ X3DH keys │ │ Rooms │ │ │ - │ Ratchet │ │ Live │ │ TTL-based │ - │ states │ │ queries │ │ auto-expiry │ - └──────────────┘ └──────────────┘ └────────────────┘ + HTTPS (TLS) + ┌────────────────────────────────────────────────────────────┐ + │ Client Browser │ + │ ┌──────────────────────────────────────────────────────┐ │ + │ │ SolidJS 1.9 + TypeScript 5.9 │ │ + │ │ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │ │ + │ │ │ WebCrypto │ │ IndexedDB │ │ nanostores │ │ │ + │ │ │ X25519 │ │ Private │ │ Reactive │ │ │ + │ │ │ Ed25519 │ │ Keys │ │ State │ │ │ + │ │ │ AES-256-GCM │ │ Ratchet │ │ │ │ │ + │ │ │ HKDF-SHA256 │ │ States │ │ │ │ │ + │ │ └──────────────┘ └──────────────┘ └────────────┘ │ │ + │ └──────────────────────────┬───────────────────────────┘ │ + │ │ │ + └──────────────────────────────┼──────────────────────────────┘ + │ + HTTPS + WSS │ + │ + ┌──────────────────────────────┼──────────────────────────────┐ + │ Nginx Reverse Proxy │ │ + │ ┌───────────────────────────┴──────────────────────────┐ │ + │ │ /api/* ──────► HTTP ──────► FastAPI :8000 │ │ + │ │ /ws ──────► WS ──────► FastAPI :8000 │ │ + │ │ /* ──────► Static files (SolidJS build) │ │ + │ └──────────────────────────────────────────────────────┘ │ + └──────────────────────────────┬──────────────────────────────┘ + │ + ┌──────────────────────────────┼──────────────────────────────┐ + │ FastAPI Backend │ │ + │ (Python 3.13, ORJSONResponse, async/await throughout) │ + │ ┌──────────────────────────────────────────────────────┐ │ + │ │ Routers: auth, rooms, encryption, websocket │ │ + │ │ Services: auth, prekey, message, presence, websocket│ │ + │ │ Core: x3dh, double_ratchet, passkey, surreal, redis │ │ + │ └────────┬──────────────┬──────────────┬───────────────┘ │ + │ │ │ │ │ + └───────────┼──────────────┼──────────────┼───────────────────┘ + │ │ │ + asyncpg │ WebSocket │ redis. │ + (TCP 5432) │ (WS 8000) │ asyncio │ + │ │ (TCP 6379) │ + ┌───────────┴──┐ ┌───────┴──────┐ ┌───┴────────────┐ + │ PostgreSQL │ │ SurrealDB │ │ Redis 8 │ + │ 16-alpine │ │ (latest) │ │ (alpine) │ + │ │ │ │ │ │ + │ Auth data │ │ Messages │ │ Challenges │ + │ Credentials │ │ Presence │ │ Rate limits │ + │ X3DH keys │ │ Rooms │ │ │ + │ Ratchet │ │ Live │ │ TTL-based │ + │ states │ │ queries │ │ auto-expiry │ + └──────────────┘ └──────────────┘ └────────────────┘ ``` A few things worth noting about this diagram: @@ -80,38 +80,38 @@ A few things worth noting about this diagram: **Key File:** `factory.py` -The application uses the factory pattern. `create_app()` at lines 63-115 builds the FastAPI instance: +The application uses the factory pattern. `create_app` builds the FastAPI instance: ``` -create_app() (factory.py:63-115) - ├── FastAPI( - │ title = settings.APP_NAME, - │ default_response_class = ORJSONResponse, - │ lifespan = lifespan, - │ ) - ├── CORSMiddleware (factory.py:78-85) - ├── GZipMiddleware, minimum_size=1000 (factory.py:87) - ├── register_exception_handlers (factory.py:89) - ├── Root endpoint "/" (factory.py:91-101) - ├── Health endpoint "/health" (factory.py:103-108) - └── Routers: - ├── auth_router → /auth/* - ├── rooms_router → /rooms/* - ├── encryption_router → /encryption/* - └── websocket_router → /ws +create_app (factory.py) + ├── FastAPI( + │ title = settings.APP_NAME, + │ default_response_class = ORJSONResponse, + │ lifespan = lifespan, + │ ) + ├── CORSMiddleware (factory.py) + ├── GZipMiddleware, minimum_size=1000 (factory.py) + ├── register_exception_handlers (factory.py) + ├── Root endpoint "/" (factory.py) + ├── Health endpoint "/health" (factory.py) + └── Routers: + ├── auth_router → /auth/* + ├── rooms_router → /rooms/* + ├── encryption_router → /encryption/* + └── websocket_router → /ws ``` -The lifespan manager at lines 39-61 runs database connections on startup and disconnections on shutdown: +The lifespan manager runs database connections on startup and disconnections on shutdown: ``` -lifespan(app) (factory.py:39-61) - Startup: - 1. await init_db() → PostgreSQL tables via SQLModel - 2. await redis_manager.connect() → Redis connection pool - 3. await surreal_db.connect() → SurrealDB WebSocket connection - Shutdown: - 1. await redis_manager.disconnect() - 2. await surreal_db.disconnect() +lifespan(app) (factory.py) + Startup: + 1. await init_db → PostgreSQL tables via SQLModel + 2. await redis_manager.connect → Redis connection pool + 3. await surreal_db.connect → SurrealDB WebSocket connection + Shutdown: + 1. await redis_manager.disconnect + 2. await surreal_db.disconnect ``` The startup order matters. PostgreSQL is initialized first because the auth tables must exist before any request processing. Redis and SurrealDB follow because they depend on having a running application context. If any of these fail, the application does not start. @@ -120,10 +120,10 @@ The startup order matters. PostgreSQL is initialized first because the auth tabl | Route Prefix | Purpose | Key Endpoints | |---|---|---| -| `/auth` | WebAuthn registration + login | `register/begin`, `register/complete`, `authenticate/begin`, `authenticate/complete`, `users/search` | -| `/encryption` | X3DH key management | `prekey-bundle/{id}`, `upload-keys/{id}`, `initialize-keys/{id}`, `rotate-signed-prekey/{id}`, `opk-count/{id}` | -| `/rooms` | Chat room CRUD | Room creation, listing, participant management | -| `/ws` | WebSocket endpoint | Real-time messaging, typing, presence, receipts | +| `/auth` | WebAuthn registration + login + session | `register/begin`, `register/complete`, `authenticate/begin`, `authenticate/complete`, `me`, `logout`, `users/search` | +| `/encryption` | Public prekey bundle storage and lookup | `prekey-bundle/{id}`, `upload-keys/{id}` | +| `/rooms` | Chat room CRUD (membership-gated) | List, create, get, delete, get messages | +| `/ws` | WebSocket endpoint (session-cookie authenticated) | Real-time messaging, typing, presence, receipts | --- @@ -133,53 +133,53 @@ The startup order matters. PostgreSQL is initialized first because the auth tabl **Key Files:** `core/websocket_manager.py`, `api/websocket.py`, `services/websocket_service.py` -The `ConnectionManager` class (`websocket_manager.py:31-296`) is the center of the real-time system. It maintains three dictionaries: +The `ConnectionManager` class (`websocket_manager.py`) is the center of the real-time system. It maintains three dictionaries: ```python -self.active_connections: dict[UUID, list[WebSocket]] = {} # line 39 -self.live_query_ids: dict[UUID, str] = {} # line 40 -self.heartbeat_tasks: dict[UUID, asyncio.Task] = {} # line 41 +self.active_connections: dict[UUID, list[WebSocket]] = {} +self.live_query_ids: dict[UUID, str] = {} +self.heartbeat_tasks: dict[UUID, asyncio.Task] = {} ``` -The first maps user IDs to lists of WebSocket connections. Each user can have up to 5 simultaneous connections (`WS_MAX_CONNECTIONS_PER_USER = 5`, `config.py:141`). This supports multi-device usage without exhausting server resources. When a sixth connection attempt arrives, the manager sends an error and closes the socket (`websocket_manager.py:52-64`). +The first maps user IDs to lists of WebSocket connections. Each user can have up to 5 simultaneous connections (`WS_MAX_CONNECTIONS_PER_USER = 5`, `config.py`). This supports multi-device usage without exhausting server resources. When a sixth connection attempt arrives, the manager sends an error and closes the socket (`websocket_manager.py`). **Connection lifecycle:** ``` -connect(websocket, user_id) (websocket_manager.py:43-95) - 1. Accept WebSocket - 2. Check connection limit (max 5 per user) - 3. Add to active_connections pool - 4. Set user online via presence_service - 5. Start heartbeat loop (30s interval) - 6. Subscribe to SurrealDB live query for user's messages - 7. Return True +connect(websocket, user_id) (websocket_manager.py) + 1. Accept WebSocket + 2. Check connection limit (max 5 per user) + 3. Add to active_connections pool + 4. Set user online via presence_service + 5. Start heartbeat loop (30s interval) + 6. Subscribe to SurrealDB live query for user's messages + 7. Return True -disconnect(websocket, user_id) (websocket_manager.py:97-133) - 1. Remove WebSocket from user's connection list - 2. If last connection: - a. Set user offline via presence_service - b. Kill SurrealDB live query - c. Cancel heartbeat task +disconnect(websocket, user_id) (websocket_manager.py) + 1. Remove WebSocket from user's connection list + 2. If last connection: + a. Set user offline via presence_service + b. Kill SurrealDB live query + c. Cancel heartbeat task ``` -**Heartbeat:** The `_heartbeat_loop` method (`websocket_manager.py:177-201`) sends a ping every `WS_HEARTBEAT_INTERVAL` seconds (30s, from `config.py:140`). If the send fails, the connection is marked dead and disconnected. This catches silently dropped connections that TCP keepalive alone would miss. +**Heartbeat:** The `_heartbeat_loop` method (`websocket_manager.py`) sends a ping every `WS_HEARTBEAT_INTERVAL` seconds (30s, from `config.py`). If the send fails, the connection is marked dead and disconnected. This catches silently dropped connections that TCP keepalive alone would miss. -**Live query subscription:** When a user connects, `_subscribe_to_messages` (`websocket_manager.py:203-223`) registers a SurrealDB live query that watches for new messages where the user is the recipient. SurrealDB pushes new messages to the server in real-time through this subscription. The callback wraps updates and dispatches them through `_handle_live_message` (`websocket_manager.py:225-251`), which forwards the encrypted payload to all of the user's active WebSocket connections. +**Live query subscription:** When a user connects, `_subscribe_to_messages` (`websocket_manager.py`) registers a SurrealDB live query that watches for new messages where the user is the recipient. SurrealDB pushes new messages to the server in real-time through this subscription. The callback wraps updates and dispatches them through `_handle_live_message` (`websocket_manager.py`), which forwards the encrypted payload to all of the user's active WebSocket connections. -**Message routing:** The WebSocket endpoint (`websocket.py:25-84`) is thin. It accepts connections, delegates to `connection_manager.connect()`, and then sits in a loop reading JSON messages. Each message gets routed by `websocket_service.route_message()`, which dispatches based on the `type` field: +**Message routing:** The WebSocket endpoint (`websocket.py`) is thin. It accepts connections, delegates to `connection_manager.connect`, and then sits in a loop reading JSON messages. Each message gets routed by `websocket_service.route_message`, which dispatches based on the `type` field: ``` -route_message(websocket, user_id, message) (websocket_service.py:40-85) - ├── "encrypted_message" → handle_encrypted_message() - ├── "typing" → handle_typing_indicator() - ├── "presence" → handle_presence_update() - ├── "receipt" → handle_read_receipt() - ├── "heartbeat" → handle_heartbeat() - └── unknown → send error response +route_message(websocket, user_id, message) (backend/app/services/websocket_service.py) + ├── "encrypted_message" → handle_encrypted_message + ├── "typing" → handle_typing_indicator + ├── "presence" → handle_presence_update + ├── "receipt" → handle_read_receipt + ├── "heartbeat" → handle_heartbeat + └── unknown → send error response ``` -**Dead connection cleanup:** When `send_message()` (`websocket_manager.py:135-153`) fails to send to a connection, it collects the dead connection and calls `disconnect()` on it. This prevents stale connections from accumulating. +**Dead connection cleanup:** When `send_message` (`websocket_manager.py`) fails to send to a connection, it collects the dead connection and calls `disconnect` on it. This prevents stale connections from accumulating. --- @@ -187,7 +187,7 @@ route_message(websocket, user_id, message) (websocket_service.py:40-85) **Purpose:** Implements the Signal Protocol cryptographic primitives for end-to-end encryption. -**Key Files:** `core/encryption/x3dh_manager.py`, `core/encryption/double_ratchet.py`, `frontend/src/crypto/primitives.ts` +**Key Files:** `frontend/src/crypto/frontend/src/crypto/x3dh.ts`, `frontend/src/crypto/frontend/src/crypto/double-ratchet.ts`, `frontend/src/crypto/primitives.ts` The encryption system has two layers: @@ -195,13 +195,13 @@ The encryption system has two layers: 2. **Double Ratchet:** Uses the X3DH shared secret to derive per-message encryption keys with forward secrecy. Every message gets a unique key, and compromising one key does not reveal past or future messages. -**X3DHManager** (`x3dh_manager.py:56-353`) handles: +**X3DHManager** (`frontend/src/crypto/x3dh.ts`) handles: - Key generation for identity keys (X25519 for DH, Ed25519 for signing) -- Signed prekey generation with Ed25519 signatures (`x3dh_manager.py:116-152`) -- One-time prekey generation (`x3dh_manager.py:154-174`) -- Signed prekey verification (`x3dh_manager.py:176-206`) -- Sender-side X3DH exchange (`x3dh_manager.py:208-281`) -- Receiver-side X3DH exchange (`x3dh_manager.py:283-350`) +- Signed prekey generation with Ed25519 signatures (`frontend/src/crypto/x3dh.ts`) +- One-time prekey generation (`frontend/src/crypto/x3dh.ts`) +- Signed prekey verification (`frontend/src/crypto/x3dh.ts`) +- Sender-side X3DH exchange (`frontend/src/crypto/x3dh.ts`) +- Receiver-side X3DH exchange (`frontend/src/crypto/x3dh.ts`) The X3DH exchange on the sender side (`perform_x3dh_sender`, lines 208-281) works like this: @@ -209,84 +209,80 @@ The X3DH exchange on the sender side (`perform_x3dh_sender`, lines 208-281) work Alice wants to message Bob (who might be offline): 1. Alice fetches Bob's prekey bundle from server: - - Bob's identity key (IK_B) - - Bob's signed prekey (SPK_B) + signature - - Bob's one-time prekey (OPK_B), if available + - Bob's identity key (IK_B) + - Bob's signed prekey (SPK_B) + signature + - Bob's one-time prekey (OPK_B), if available 2. Alice verifies SPK_B signature using Bob's Ed25519 identity key 3. Alice generates ephemeral keypair (EK_A) 4. Four DH operations: - DH1 = X25519(IK_A_private, SPK_B) # Alice identity x Bob signed prekey - DH2 = X25519(EK_A_private, IK_B) # Alice ephemeral x Bob identity - DH3 = X25519(EK_A_private, SPK_B) # Alice ephemeral x Bob signed prekey - DH4 = X25519(EK_A_private, OPK_B) # Alice ephemeral x Bob one-time prekey - # (only if OPK available) + DH1 = X25519(IK_A_private, SPK_B) # Alice identity x Bob signed prekey + DH2 = X25519(EK_A_private, IK_B) # Alice ephemeral x Bob identity + DH3 = X25519(EK_A_private, SPK_B) # Alice ephemeral x Bob signed prekey + DH4 = X25519(EK_A_private, OPK_B) # Alice ephemeral x Bob one-time prekey + # (only if OPK available) 5. Concatenate: key_material = DH1 || DH2 || DH3 [|| DH4] 6. Derive shared key via HKDF-SHA256: - shared_key = HKDF( - salt = 0x00 * 32, - ikm = 0xFF * 32 || key_material, - info = "X3DH", - length = 32 - ) + shared_key = HKDF( + salt = 0x00 * 32, + ikm = 0xFF * 32 || key_material, + info = "X3DH", + length = 32 + ) 7. Return: shared_key, associated_data (IK_A_pub || IK_B_pub), EK_A_pub ``` The `0xFF * 32` prefix before the key material is a domain separator. This matches the Signal specification and prevents potential cross-protocol attacks. -**DoubleRatchet** (`double_ratchet.py:64-419`) handles: -- Sender initialization from X3DH shared secret (`double_ratchet.py:279-302`) -- Receiver initialization (`double_ratchet.py:304-321`) -- Message encryption with chain key advancement (`double_ratchet.py:323-362`) -- Message decryption with out-of-order support (`double_ratchet.py:364-416`) -- DH ratchet steps for forward secrecy (`double_ratchet.py:155-213`) -- Skipped message key storage for out-of-order delivery (`double_ratchet.py:215-258`) +**DoubleRatchet** (`frontend/src/crypto/double-ratchet.ts`) handles: +- Sender initialization from X3DH shared secret (`frontend/src/crypto/double-ratchet.ts`) +- Receiver initialization (`frontend/src/crypto/double-ratchet.ts`) +- Message encryption with chain key advancement (`frontend/src/crypto/double-ratchet.ts`) +- Message decryption with out-of-order support (`frontend/src/crypto/double-ratchet.ts`) +- DH ratchet steps for forward secrecy (`frontend/src/crypto/double-ratchet.ts`) +- Skipped message key storage for out-of-order delivery (`frontend/src/crypto/double-ratchet.ts`) The ratchet uses two KDF chains: ``` Root Key Chain: - _kdf_rk(root_key, dh_output) → (new_root_key, new_chain_key) - Uses HKDF-SHA256 with root_key as salt (double_ratchet.py:79-94) + _kdf_rk(root_key, dh_output) → (new_root_key, new_chain_key) + Uses HKDF-SHA256 with root_key as salt (frontend/src/crypto/double-ratchet.ts) Symmetric Key Chain: - _kdf_ck(chain_key) → (next_chain_key, message_key) - Uses HMAC-SHA256 with 0x01 for chain key, 0x02 for message key - (double_ratchet.py:96-109) + _kdf_ck(chain_key) → (next_chain_key, message_key) + Uses HMAC-SHA256 with 0x01 for chain key, 0x02 for message key + (frontend/src/crypto/double-ratchet.ts) ``` -Each message is encrypted with AES-256-GCM using a 12-byte random nonce (`double_ratchet.py:111-130`). The message key is derived from the sending chain and used exactly once. After encryption, the sending chain advances, and the old message key is discarded. +Each message is encrypted with AES-256-GCM using a 12-byte random nonce (`frontend/src/crypto/double-ratchet.ts`). The message key is derived from the sending chain and used exactly once. After encryption, the sending chain advances, and the old message key is discarded. **Security limits from `config.py`:** -- `MAX_SKIP_MESSAGE_KEYS = 1000` (line 73): Maximum messages that can arrive out of order before the protocol rejects them -- `MAX_CACHED_MESSAGE_KEYS = 2000` (line 74): Maximum stored skipped keys before eviction -- `AES_GCM_NONCE_SIZE = 12` (line 69): 96-bit nonces for AES-GCM -- `HKDF_OUTPUT_SIZE = 32` (line 70): 256-bit derived keys +- `MAX_SKIP_MESSAGE_KEYS = 1000` : Maximum messages that can arrive out of order before the protocol rejects them +- `MAX_CACHED_MESSAGE_KEYS = 2000` : Maximum stored skipped keys before eviction +- `AES_GCM_NONCE_SIZE = 12` : 96-bit nonces for AES-GCM +- `HKDF_OUTPUT_SIZE = 32` : 256-bit derived keys -**Client-side crypto:** The frontend `primitives.ts` (lines 1-397) mirrors the backend crypto using the WebCrypto API. It provides: -- `generateX25519KeyPair()` (line 15) -- `x25519DeriveSharedSecret()` (line 26) -- `generateEd25519KeyPair()` (line 89) -- `ed25519Sign()` / `ed25519Verify()` (lines 99, 113) -- `hkdfDerive()` / `hkdfDeriveKey()` (lines 166, 194) -- `aesGcmEncrypt()` / `aesGcmDecrypt()` (lines 224, 261) -- `hmacSha256()` / `hmacSha256Verify()` (lines 310, 329) -- `constantTimeEqual()` (line 388): Constant-time comparison to prevent timing side channels +**Client-side crypto:** The frontend `primitives.ts` mirrors the backend crypto using the WebCrypto API. It provides: +- `generateX25519KeyPair` +- `x25519DeriveSharedSecret` +- `generateEd25519KeyPair` +- `ed25519Sign` / `ed25519Verify` (lines 99, 113) +- `hkdfDerive` / `hkdfDeriveKey` (lines 166, 194) +- `aesGcmEncrypt` / `aesGcmDecrypt` (lines 224, 261) +- `hmacSha256` / `hmacSha256Verify` (lines 310, 329) +- `constantTimeEqual` : Constant-time comparison to prevent timing side channels The client also has `crypto-service.ts`, `double-ratchet.ts`, `x3dh.ts`, `key-store.ts`, and `message-store.ts` which orchestrate these primitives into the full protocol flows. -**Server-side vs client-side encryption paths:** +**One encryption path: client-side, server is opaque.** -The `MessageService` (`message_service.py`) has two paths: -- `store_encrypted_message()` (lines 269-314): Client-side passthrough. The server receives ciphertext, nonce, and header as strings and stores them as-is in SurrealDB. No decryption or re-encryption on the server. -- `send_encrypted_message()` (lines 316-402): Server-side encryption. Marked `[DEPRECATED]` in the docstring (line 325). This path loads the ratchet state from PostgreSQL, encrypts on the server, and stores in SurrealDB. It exists for backwards compatibility during migration to full client-side encryption. - -The client-side path is the intended production path. In this path, the server literally cannot read messages because it never has the keys. +`MessageService` (`backend/app/services/message_service.py`) exposes a single method, `store_encrypted_message`, which receives `ciphertext`, `nonce`, and `header` as opaque strings and writes them straight into SurrealDB. The server has no decryption helper and no ratchet state — it literally cannot read messages because it never has the keys. --- @@ -296,66 +292,66 @@ The client-side path is the intended production path. In this path, the server l **Key Files:** `core/passkey/passkey_manager.py`, `api/auth.py`, `services/auth_service.py`, `core/redis_manager.py` -The `PasskeyManager` (`passkey_manager.py:43-210`) wraps the `py_webauthn` library. It is configured with the Relying Party (RP) settings from `config.py`: +The `PasskeyManager` (`backend/app/core/passkey/passkey_manager.py`) wraps the `py_webauthn` library. It is configured with the Relying Party (RP) settings from `config.py`: ```python -self.rp_id = settings.RP_ID # e.g., "localhost" or "chat.example.com" -self.rp_name = settings.RP_NAME # "Encrypted P2P Chat" +self.rp_id = settings.RP_ID # e.g., "localhost" or "chat.example.com" +self.rp_name = settings.RP_NAME # "Encrypted P2P Chat" self.rp_origin = settings.RP_ORIGIN # "https://chat.example.com" ``` **Registration flow:** ``` -Client Server Redis - │ │ │ - │ POST /auth/register/begin│ │ - │ { username, display_name }│ │ - │ ─────────────────────────►│ │ - │ │ │ - │ PasskeyManager.generate_registration_options() - │ (passkey_manager.py:55-94) │ - │ │ │ - │ │ SET webauthn:reg_challenge:{username} - │ │ challenge_bytes, TTL=600s│ - │ │ ──────────────────────────► - │ │ │ - │ ◄─── registration options (publicKey config) ────────│ - │ │ │ - │ Browser WebAuthn API │ │ - │ navigator.credentials │ │ - │ .create(options) │ │ - │ User touches │ │ - │ authenticator │ │ - │ │ │ - │ POST /auth/register/complete │ - │ { credential, username } │ │ - │ ─────────────────────────►│ │ - │ │ │ - │ │ GET+DEL webauthn:reg_challenge:{username} - │ │ ──────────────────────────► - │ │ ◄── challenge_bytes ──────│ - │ │ │ - │ PasskeyManager.verify_registration() │ - │ (passkey_manager.py:96-130) │ - │ │ │ - │ Create User in PostgreSQL │ - │ Store Credential in PostgreSQL │ - │ Initialize X3DH keys │ - │ (prekey_service.py:152-219) │ - │ │ │ - │ ◄─── UserResponse (id, username, etc.) ──────────────│ +Client Server Redis + │ │ │ + │ POST /auth/register/begin│ │ + │ { username, display_name }│ │ + │ ─────────────────────────►│ │ + │ │ │ + │ PasskeyManager.generate_registration_options + │ (backend/app/core/passkey/passkey_manager.py) │ + │ │ │ + │ │ SET webauthn:reg_challenge:{username} + │ │ challenge_bytes, TTL=600s│ + │ │ ──────────────────────────► + │ │ │ + │ ◄─── registration options (publicKey config) ────────│ + │ │ │ + │ Browser WebAuthn API │ │ + │ navigator.credentials │ │ + │ .create(options) │ │ + │ User touches │ │ + │ authenticator │ │ + │ │ │ + │ POST /auth/register/complete │ + │ { credential, username } │ │ + │ ─────────────────────────►│ │ + │ │ │ + │ │ GET+DEL webauthn:reg_challenge:{username} + │ │ ──────────────────────────► + │ │ ◄── challenge_bytes ──────│ + │ │ │ + │ PasskeyManager.verify_registration │ + │ (backend/app/core/passkey/passkey_manager.py) │ + │ │ │ + │ Create User in PostgreSQL │ + │ Store Credential in PostgreSQL │ + │ Initialize X3DH keys │ + │ (backend/app/services/prekey_service.py) │ + │ │ │ + │ ◄─── UserResponse (id, username, etc.) ──────────────│ ``` **Authentication flow:** -The authentication flow is similar but uses `generate_authentication_options` and `verify_authentication`. A critical detail is the signature counter check (`passkey_manager.py:184-193`): if the new counter is not greater than the stored counter, it raises a `ValueError` indicating a potentially cloned authenticator. This is the WebAuthn clone detection mechanism. +The authentication flow is similar but uses `generate_authentication_options` and `verify_authentication`. A critical detail is the signature counter check (`backend/app/core/passkey/passkey_manager.py`): if the new counter is not greater than the stored counter, it raises a `ValueError` indicating a potentially cloned authenticator. This is the WebAuthn clone detection mechanism. -**Challenge storage:** Redis stores challenges with a 600-second TTL (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600`, `config.py:84`). The `get_registration_challenge` and `get_authentication_challenge` methods use Redis pipelines to atomically GET and DELETE the challenge (`redis_manager.py:86-95`). This ensures each challenge is used exactly once. +**Challenge storage:** Redis stores challenges with a 600-second TTL (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600`, `config.py`). The `get_registration_challenge` and `get_authentication_challenge` methods use Redis pipelines to atomically GET and DELETE the challenge (`redis_manager.py`). This ensures each challenge is used exactly once. -The challenge itself is 32 random bytes (`WEBAUTHN_CHALLENGE_BYTES = 32`, `config.py:85`), stored as hex in Redis. +The challenge itself is 32 random bytes (`WEBAUTHN_CHALLENGE_BYTES = 32`, `config.py`), stored as hex in Redis. -**Auth endpoints** (`auth.py:31-103`): +**Auth endpoints** (`auth.py`): | Endpoint | Method | Status | Description | |---|---|---|---| @@ -373,23 +369,23 @@ The challenge itself is 32 random bytes (`WEBAUTHN_CHALLENGE_BYTES = 32`, `confi **PostgreSQL via SQLModel/SQLAlchemy async:** Handles all relational data with ACID guarantees. -The engine is configured in `models/Base.py:38-44`: +The engine is configured in `models/Base.py`: ```python engine = create_async_engine( - str(settings.DATABASE_URL), # postgresql+asyncpg://... - echo = settings.DEBUG, # SQL logging in development - pool_size = settings.DB_POOL_SIZE, # 20 (config.py:118) - max_overflow = settings.DB_MAX_OVERFLOW, # 40 (config.py:119) - pool_pre_ping = True, # Detect stale connections + str(settings.DATABASE_URL), # postgresql+asyncpg://... + echo = settings.DEBUG, # SQL logging in development + pool_size = settings.DB_POOL_SIZE, # 20 (config.py) + max_overflow = settings.DB_MAX_OVERFLOW, # 40 (config.py) + pool_pre_ping = True, # Detect stale connections ) ``` -The session factory (`Base.py:47-51`) creates `AsyncSession` instances with `expire_on_commit=False` so that objects remain usable after commit without requiring a refresh. +The session factory (`Base.py`) creates `AsyncSession` instances with `expire_on_commit=False` so that objects remain usable after commit without requiring a refresh. -**SurrealDB via AsyncSurreal:** Handles real-time messaging data. The `SurrealDBManager` (`surreal_manager.py:28-428`) connects over WebSocket to SurrealDB and provides methods for message CRUD, room management, presence tracking, and live query subscriptions. The live query feature is the primary reason SurrealDB was chosen: it pushes new records to subscribers in real-time, eliminating the need for polling. +**SurrealDB via AsyncSurreal:** Handles real-time messaging data. The `SurrealDBManager` (`surreal_manager.py`) connects over WebSocket to SurrealDB and provides methods for message CRUD, room management, presence tracking, and live query subscriptions. The live query feature is the primary reason SurrealDB was chosen: it pushes new records to subscribers in real-time, eliminating the need for polling. -**Redis via redis.asyncio:** Handles ephemeral data. The `RedisManager` (`redis_manager.py:19-174`) uses a connection pool of 50 connections (`redis_manager.py:39`) and stores challenges as hex-encoded bytes with TTL-based expiry. +**Redis via redis.asyncio:** Handles ephemeral data. The `RedisManager` (`redis_manager.py`) uses a connection pool of 50 connections (`redis_manager.py`) and stores challenges as hex-encoded bytes with TTL-based expiry. --- @@ -400,124 +396,124 @@ The session factory (`Base.py:47-51`) creates `AsyncSession` instances with `exp This is the most important flow in the system. Here is what happens step by step when Alice sends a message to Bob, assuming they already have an established Double Ratchet session: ``` -Alice's Browser Server Bob's Browser - │ │ │ - ┌────┴────────────────┐ │ │ - │ 1. User types msg │ │ │ - │ in ChatInput.tsx │ │ │ - └────┬────────────────┘ │ │ - │ │ │ - ┌────┴────────────────┐ │ │ - │ 2. crypto-service.ts│ │ │ - │ encrypts: │ │ │ - │ a. Advance send │ │ │ - │ chain via │ │ │ - │ HMAC-SHA256 │ │ │ - │ b. Derive unique │ │ │ - │ message key │ │ │ - │ c. AES-256-GCM │ │ │ - │ encrypt with │ │ │ - │ random nonce │ │ │ - │ d. Build header: │ │ │ - │ {dh_pub_key, │ │ │ - │ msg_number, │ │ │ - │ prev_chain} │ │ │ - └────┬────────────────┘ │ │ - │ │ │ - │ 3. WebSocket send: │ │ - │ { │ │ - │ type: "encrypted_message", │ - │ recipient_id: bob_uuid, │ │ - │ room_id: room_uuid, │ │ - │ ciphertext: "base64...",│ │ - │ nonce: "base64...", │ │ - │ header: "{...json...}", │ │ - │ temp_id: "client_123" │ │ - │ } │ │ - │ ───────────────────────────► │ - │ │ │ - │ ┌────────────┴───────────────┐ │ - │ │ 4. websocket.py:46-54 │ │ - │ │ receives JSON, parses │ │ - │ │ routes via │ │ - │ │ websocket_service │ │ - │ └────────────┬───────────────┘ │ - │ │ │ - │ ┌────────────┴───────────────┐ │ - │ │ 5. websocket_service.py: │ │ - │ │ handle_encrypted_message│ │ - │ │ (lines 87-179) │ │ - │ │ Extracts fields, │ │ - │ │ opens DB session │ │ - │ └────────────┬───────────────┘ │ - │ │ │ - │ ┌────────────┴───────────────┐ │ - │ │ 6. message_service.py: │ │ - │ │ store_encrypted_message │ │ - │ │ (lines 269-314) │ │ - │ │ Looks up sender user │ │ - │ │ for username. │ │ - │ │ Stores ciphertext, │ │ - │ │ nonce, header AS-IS in │ │ - │ │ SurrealDB. │ │ - │ │ NO DECRYPTION. │ │ - │ └────────────┬───────────────┘ │ - │ │ │ - │ ┌────────────┴───────────────┐ │ - │ │ 7. SurrealDB live query │ │ - │ │ fires for Bob because │ │ - │ │ recipient_id matches │ │ - │ └────────────┬───────────────┘ │ - │ │ │ - │ ┌────────────┴───────────────┐ │ - │ │ 8. websocket_manager.py: │ │ - │ │ _handle_live_message │ │ - │ │ (lines 225-251) │ │ - │ │ Wraps as EncryptedMsgWS │ │ - │ └────────────┬───────────────┘ │ - │ │ │ - │ ┌────────────┴───────────────┐ │ - │ │ 9. send_message(bob_uuid) │ │ - │ │ (lines 135-153) │ │ - │ │ Sends to ALL of Bob's │ │ - │ │ active WebSockets │ │ - │ └────────────┬───────────────┘ │ - │ │ ─────────────────────────────► - │ │ │ - │ │ ┌────────────────┴──┐ - │ │ │ 10. Bob's crypto- │ - │ │ │ service.ts │ - │ │ │ decrypts: │ - │ │ │ a. Check for │ - │ │ │ skipped │ - │ │ │ msg keys │ - │ │ │ b. If new DH │ - │ │ │ pub key, │ - │ │ │ DH ratchet │ - │ │ │ step │ - │ │ │ c. Advance │ - │ │ │ recv chain │ - │ │ │ d. Derive msg │ - │ │ │ key │ - │ │ │ e. AES-256-GCM│ - │ │ │ decrypt │ - │ │ └────────┬─────────┘ - │ │ │ - │ │ ┌────────┴─────────┐ - │ │ │ 11. Plaintext │ - │ │ │ rendered in │ - │ │ │ MessageList │ - │ │ └──────────────────┘ - │ │ │ - │ ◄── confirmation ─────────│ │ - │ { type: "message_sent", │ │ - │ temp_id: "client_123", │ │ - │ status: "sent" } │ │ +Alice's Browser Server Bob's Browser + │ │ │ + ┌────┴────────────────┐ │ │ + │ 1. User types msg │ │ │ + │ in ChatInput.tsx │ │ │ + └────┬────────────────┘ │ │ + │ │ │ + ┌────┴────────────────┐ │ │ + │ 2. crypto-service.ts│ │ │ + │ encrypts: │ │ │ + │ a. Advance send │ │ │ + │ chain via │ │ │ + │ HMAC-SHA256 │ │ │ + │ b. Derive unique │ │ │ + │ message key │ │ │ + │ c. AES-256-GCM │ │ │ + │ encrypt with │ │ │ + │ random nonce │ │ │ + │ d. Build header: │ │ │ + │ {dh_pub_key, │ │ │ + │ msg_number, │ │ │ + │ prev_chain} │ │ │ + └────┬────────────────┘ │ │ + │ │ │ + │ 3. WebSocket send: │ │ + │ { │ │ + │ type: "encrypted_message", │ + │ recipient_id: bob_uuid, │ │ + │ room_id: room_uuid, │ │ + │ ciphertext: "base64...",│ │ + │ nonce: "base64...", │ │ + │ header: "{...json...}", │ │ + │ temp_id: "client_123" │ │ + │ } │ │ + │ ───────────────────────────► │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 4. websocket.py │ │ + │ │ receives JSON, parses │ │ + │ │ routes via │ │ + │ │ websocket_service │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 5. websocket_service.py: │ │ + │ │ handle_encrypted_message│ │ + │ │ │ │ + │ │ Extracts fields, │ │ + │ │ opens DB session │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 6. message_service.py: │ │ + │ │ store_encrypted_message │ │ + │ │ │ │ + │ │ Looks up sender user │ │ + │ │ for username. │ │ + │ │ Stores ciphertext, │ │ + │ │ nonce, header AS-IS in │ │ + │ │ SurrealDB. │ │ + │ │ NO DECRYPTION. │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 7. SurrealDB live query │ │ + │ │ fires for Bob because │ │ + │ │ recipient_id matches │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 8. websocket_manager.py: │ │ + │ │ _handle_live_message │ │ + │ │ │ │ + │ │ Wraps as EncryptedMsgWS │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 9. send_message(bob_uuid) │ │ + │ │ │ │ + │ │ Sends to ALL of Bob's │ │ + │ │ active WebSockets │ │ + │ └────────────┬───────────────┘ │ + │ │ ─────────────────────────────► + │ │ │ + │ │ ┌────────────────┴──┐ + │ │ │ 10. Bob's crypto- │ + │ │ │ service.ts │ + │ │ │ decrypts: │ + │ │ │ a. Check for │ + │ │ │ skipped │ + │ │ │ msg keys │ + │ │ │ b. If new DH │ + │ │ │ pub key, │ + │ │ │ DH ratchet │ + │ │ │ step │ + │ │ │ c. Advance │ + │ │ │ recv chain │ + │ │ │ d. Derive msg │ + │ │ │ key │ + │ │ │ e. AES-256-GCM│ + │ │ │ decrypt │ + │ │ └────────┬─────────┘ + │ │ │ + │ │ ┌────────┴─────────┐ + │ │ │ 11. Plaintext │ + │ │ │ rendered in │ + │ │ │ MessageList │ + │ │ └──────────────────┘ + │ │ │ + │ ◄── confirmation ─────────│ │ + │ { type: "message_sent", │ │ + │ temp_id: "client_123", │ │ + │ status: "sent" } │ │ ``` -The server never sees the plaintext. It acts purely as a relay that stores and forwards ciphertext blobs. The `store_encrypted_message` method (`message_service.py:269-314`) literally just wraps the received fields into a dict and calls `surreal_db.create_message()`. +The server never sees the plaintext. It acts purely as a relay that stores and forwards ciphertext blobs. The `store_encrypted_message` method (`backend/app/services/message_service.py`) literally just wraps the received fields into a dict and calls `surreal_db.create_message`. -The confirmation message sent back to Alice (`websocket_service.py:148-159`) includes the `temp_id` so the client can match it to the optimistically rendered message in the UI. +The confirmation message sent back to Alice (`backend/app/services/websocket_service.py`) includes the `temp_id` so the client can match it to the optimistically rendered message in the UI. --- @@ -527,45 +523,45 @@ This flow happens once per user. ``` Step 1: POST /auth/register/begin - auth_service.py:331-374 - → PasskeyManager generates challenge - → Challenge stored in Redis with 600s TTL - → Registration options returned to client + backend/app/services/auth_service.py + → PasskeyManager generates challenge + → Challenge stored in Redis with 600s TTL + → Registration options returned to client Step 2: Browser WebAuthn API (navigator.credentials.create) - → User interacts with authenticator (Touch ID, YubiKey, etc.) - → Browser creates credential bound to RP origin + → User interacts with authenticator (Touch ID, YubiKey, etc.) + → Browser creates credential bound to RP origin Step 3: POST /auth/register/complete - auth_service.py:376-437 - → Challenge retrieved from Redis (GET+DEL atomic) - → PasskeyManager verifies credential against challenge - → User record created in PostgreSQL - → Credential record created in PostgreSQL - → prekey_service.initialize_user_keys() called - (prekey_service.py:152-219) + backend/app/services/auth_service.py + → Challenge retrieved from Redis (GET+DEL atomic) + → PasskeyManager verifies credential against challenge + → User record created in PostgreSQL + → Credential record created in PostgreSQL + → prekey_service.initialize_user_keys called + (backend/app/services/prekey_service.py) -Step 4: Server-side key initialization (prekey_service.py:152-219) - → Generate X25519 identity keypair (IK) - → Generate Ed25519 signing keypair - → Store both public+private in PostgreSQL identity_keys table - → Generate signed prekey (SPK) signed with Ed25519 IK - → Generate 100 one-time prekeys (OPKs) - → Store all in PostgreSQL +Step 4: Server-side key initialization (backend/app/services/prekey_service.py) + → Generate X25519 identity keypair (IK) + → Generate Ed25519 signing keypair + → Store both public+private in PostgreSQL identity_keys table + → Generate signed prekey (SPK) signed with Ed25519 IK + → Generate 100 one-time prekeys (OPKs) + → Store all in PostgreSQL Step 5: (Client-side, post-registration) - Client generates its own X3DH keys using WebCrypto: - → X25519 identity keypair - → Ed25519 signing keypair - → Signed prekey with signature - → 100 one-time prekeys + Client generates its own X3DH keys using WebCrypto: + → X25519 identity keypair + → Ed25519 signing keypair + → Signed prekey with signature + → 100 one-time prekeys Step 6: POST /encryption/upload-keys/{user_id} - encryption.py:72-95 - → prekey_service.store_client_keys() - (prekey_service.py:45-150) - → Only PUBLIC keys stored on server - → Private keys remain in browser IndexedDB + backend/app/api/encryption.py + → prekey_service.store_client_keys + (backend/app/services/prekey_service.py) + → Only PUBLIC keys stored on server + → Private keys remain in browser IndexedDB ``` Note there is a dual path here. Step 4 is the server-side key generation (used as a fallback and for backwards compatibility). Step 5-6 is the client-side key generation path (the preferred production path). In the client-side path, the server never sees private keys. @@ -578,45 +574,45 @@ When Alice wants to message Bob for the first time: ``` Step 1: Alice's client requests Bob's prekey bundle - GET /encryption/prekey-bundle/{bob_id} - encryption.py:32-51 + GET /encryption/prekey-bundle/{bob_id} + backend/app/api/encryption.py Step 2: prekey_service.get_prekey_bundle(session, bob_id) - prekey_service.py:293-361 - → Fetch Bob's identity key (IK) - → Fetch Bob's active signed prekey (SPK) - → If no active SPK, auto-rotate (prekey_service.py:321) - → Fetch one unused one-time prekey (OPK) - → Mark OPK as used (single-use guarantee, line 332) - → If unused OPK count < 20, auto-replenish 100 more - (encryption.py:47-49) - → Return PreKeyBundle{IK, IK_ed25519, SPK, SPK_sig, OPK} + backend/app/services/prekey_service.py + → Fetch Bob's identity key (IK) + → Fetch Bob's active signed prekey (SPK) + → If no active SPK, auto-rotate (backend/app/services/prekey_service.py) + → Fetch one unused one-time prekey (OPK) + → Mark OPK as used (single-use guarantee,) + → If unused OPK count < 20, auto-replenish 100 more + (backend/app/api/encryption.py) + → Return PreKeyBundle{IK, IK_ed25519, SPK, SPK_sig, OPK} Step 3: Alice performs X3DH sender-side locally - Using the prekey bundle and her own identity key: - a. Verify SPK signature with Bob's Ed25519 IK - b. Generate ephemeral keypair (EK) - c. Compute DH1..DH4 - d. Derive shared_key via HKDF + Using the prekey bundle and her own identity key: + a. Verify SPK signature with Bob's Ed25519 IK + b. Generate ephemeral keypair (EK) + c. Compute DH1..DH4 + d. Derive shared_key via HKDF Step 4: Initialize Double Ratchet with shared_key - Alice's client calls double_ratchet.initialize_sender() - with the shared key and Bob's SPK as the initial peer key + Alice's client calls double_ratchet.initialize_sender + with the shared key and Bob's SPK as the initial peer key Step 5: First message includes X3DH header - The header contains Alice's ephemeral public key and - identity key reference so Bob can derive the same - shared secret when he comes online + The header contains Alice's ephemeral public key and + identity key reference so Bob can derive the same + shared secret when he comes online Step 6: Bob receives the message - Bob's client uses the X3DH header + his own private keys - to perform the receiver-side X3DH exchange - (x3dh_manager.py:283-350) - Both parties now share the same root key for the - Double Ratchet + Bob's client uses the X3DH header + his own private keys + to perform the receiver-side X3DH exchange + (frontend/src/crypto/x3dh.ts) + Both parties now share the same root key for the + Double Ratchet ``` -The OPK single-use guarantee is enforced at the database level. When `get_prekey_bundle` fetches an OPK, it immediately marks `is_used = True` and commits (`prekey_service.py:332-345`). If someone else requests the same OPK concurrently, they will get a different one or none. +The OPK single-use guarantee is enforced at the database level. When `get_prekey_bundle` fetches an OPK, it immediately marks `is_used = True` and commits (`backend/app/services/prekey_service.py`). If someone else requests the same OPK concurrently, they will get a different one or none. --- @@ -624,9 +620,9 @@ The OPK single-use guarantee is enforced at the database level. When `get_prekey ### Application Factory Pattern -**Where:** `factory.py:63-115` +**Where:** `factory.py` -**Why:** The `create_app()` function builds and returns a fully configured FastAPI instance. This separates app creation from app execution (`main.py` just calls `create_app()`). The practical benefit is testability: you can call `create_app()` with different configurations in tests without starting a server. The lifespan manager (`factory.py:39-61`) ensures all databases are connected before the first request and properly disconnected on shutdown. +**Why:** The `create_app` function builds and returns a fully configured FastAPI instance. This separates app creation from app execution (`main.py` just calls `create_app`). The practical benefit is testability: you can call `create_app` with different configurations in tests without starting a server. The lifespan manager (`factory.py`) ensures all databases are connected before the first request and properly disconnected on shutdown. ### Service Layer Pattern @@ -635,42 +631,40 @@ The OPK single-use guarantee is enforced at the database level. When `get_prekey **Why:** Business logic lives in service classes, not in API endpoint functions. The API layer (`api/`) is thin: it handles request validation and response formatting, then delegates to services. Services are stateless singletons instantiated at module level: ```python -message_service = MessageService() # message_service.py:469 -prekey_service = PrekeyService() # prekey_service.py:468 -auth_service = AuthService() # auth_service.py:601 -websocket_service = WebSocketService() # websocket_service.py:324 +message_service = MessageService # backend/app/services/message_service.py +prekey_service = PrekeyService # backend/app/services/prekey_service.py +auth_service = AuthService # backend/app/services/auth_service.py +websocket_service = WebSocketService # backend/app/services/websocket_service.py ``` This means any endpoint can import and call any service without worrying about instantiation or dependency injection. The tradeoff is that services cannot be easily swapped at runtime, but for this application that is not needed. ### Connection Pool Pattern -**Where:** `websocket_manager.py:31-42` +**Where:** `websocket_manager.py` -**Why:** The `ConnectionManager` maps user IDs to lists of WebSocket connections. This supports multiple simultaneous devices per user (up to 5, enforced at `websocket_manager.py:52`). When a message needs to be delivered, `send_message()` iterates over all connections for that user. Dead connections are detected during send attempts and cleaned up immediately. +**Why:** The `ConnectionManager` maps user IDs to lists of WebSocket connections. This supports multiple simultaneous devices per user (up to 5, enforced at `websocket_manager.py`). When a message needs to be delivered, `send_message` iterates over all connections for that user. Dead connections are detected during send attempts and cleaned up immediately. ### Observer Pattern (Live Queries) -**Where:** `websocket_manager.py:203-224`, `surreal_manager.py:341-359` +**Where:** `websocket_manager.py`, `surreal_manager.py` **Why:** SurrealDB live queries implement a push-based notification system. When a new message is created in SurrealDB with `recipient_id = bob`, SurrealDB pushes that record to the server through the live query callback. The server then forwards it to Bob's WebSocket connections. This eliminates polling entirely. The alternative would be the server polling SurrealDB for new messages, which would add latency proportional to the polling interval and waste resources when no messages are pending. -The subscription is per-user, not per-room. `live_messages_for_user()` (`surreal_manager.py:341-359`) watches `WHERE recipient_id = '{user_id}'`, so the server receives all messages destined for that user regardless of room. +The subscription is per-user, not per-room. `live_messages_for_user` (`surreal_manager.py`) watches `WHERE recipient_id = '{user_id}'`, so the server receives all messages destined for that user regardless of room. ### Singleton Pattern **Where:** Module-level instances at the bottom of each manager file: ```python -x3dh_manager = X3DHManager() # x3dh_manager.py:353 -double_ratchet = DoubleRatchet() # double_ratchet.py:419 -passkey_manager = PasskeyManager() # passkey_manager.py:210 -connection_manager = ConnectionManager() # websocket_manager.py:296 -surreal_db = SurrealDBManager() # surreal_manager.py:428 -redis_manager = RedisManager() # redis_manager.py:174 +passkey_manager = PasskeyManager # backend/app/core/passkey/passkey_manager.py +connection_manager = ConnectionManager # backend/app/core/websocket_manager.py +surreal_db = SurrealDBManager # backend/app/core/surreal_manager.py +redis_manager = RedisManager # backend/app/core/redis_manager.py ``` -**Why:** These managers hold no per-request state. `X3DHManager` and `DoubleRatchet` are pure functions wrapped in a class (they take state as arguments and return results). `PasskeyManager` holds only the RP configuration. The connection managers (`surreal_db`, `redis_manager`, `connection_manager`) hold shared connection pools. A single instance per process is the correct model. +**Why:** These managers either hold no per-request state (`PasskeyManager` holds only the RP configuration) or manage shared connection pools (`surreal_db`, `redis_manager`, `connection_manager`). A single instance per process is the correct model. The X3DH and Double Ratchet implementations live in the browser as plain TypeScript modules — there is no server-side counterpart that needs a singleton. --- @@ -678,60 +672,61 @@ redis_manager = RedisManager() # redis_manager.py:174 ``` ┌─────────────────────────────────────────────────────────────────┐ -│ API Layer (api/) │ -│ ┌───────────┐ ┌──────────────┐ ┌─────────┐ ┌──────────────┐ │ -│ │ auth.py │ │ encryption.py│ │ rooms.py│ │ websocket.py │ │ -│ │ lines │ │ lines 1-127 │ │ │ │ lines 1-85 │ │ -│ │ 1-104 │ │ │ │ │ │ │ │ -│ └───────────┘ └──────────────┘ └─────────┘ └──────────────┘ │ -│ HTTP/WS endpoints. Request validation. Response formatting. │ -│ Thin wrappers that delegate to services. │ +│ API Layer (api/) │ +│ ┌───────────┐ ┌──────────────┐ ┌─────────┐ ┌──────────────┐ │ +│ │ auth.py │ │ encryption.py│ │ rooms.py│ │ websocket.py │ │ +│ │ lines │ │ lines 1-127 │ │ │ │ lines 1-85 │ │ +│ │ 1-104 │ │ │ │ │ │ │ │ +│ └───────────┘ └──────────────┘ └─────────┘ └──────────────┘ │ +│ HTTP/WS endpoints. Request validation. Response formatting. │ +│ Thin wrappers that delegate to services. │ ├─────────────────────────────────────────────────────────────────┤ -│ Service Layer (services/) │ -│ ┌──────────────┐ ┌───────────────┐ ┌─────────────────────┐ │ -│ │ auth_service │ │ prekey_service│ │ message_service │ │ -│ │ lines 1-601 │ │ lines 1-468 │ │ lines 1-469 │ │ -│ └──────────────┘ └───────────────┘ └─────────────────────┘ │ -│ ┌──────────────────┐ ┌──────────────────┐ │ -│ │ presence_service │ │ websocket_service│ │ -│ └──────────────────┘ └──────────────────┘ │ -│ Business logic. Orchestration. Error handling. │ -│ Stateless singletons. Import Core + Models. │ +│ Service Layer (services/) │ +│ ┌──────────────┐ ┌───────────────┐ ┌─────────────────────┐ │ +│ │ auth_service │ │ prekey_service│ │ message_service │ │ +│ │ lines 1-601 │ │ lines 1-468 │ │ lines 1-469 │ │ +│ └──────────────┘ └───────────────┘ └─────────────────────┘ │ +│ ┌──────────────────┐ ┌──────────────────┐ │ +│ │ presence_service │ │ websocket_service│ │ +│ └──────────────────┘ └──────────────────┘ │ +│ Business logic. Orchestration. Error handling. │ +│ Stateless singletons. Import Core + Models. │ ├─────────────────────────────────────────────────────────────────┤ -│ Core Layer (core/) │ -│ ┌──────────────┐ ┌───────────────┐ ┌─────────────────────┐ │ -│ │ x3dh_manager │ │ double_ratchet│ │ passkey_manager │ │ -│ │ lines 1-353 │ │ lines 1-419 │ │ lines 1-210 │ │ -│ └──────────────┘ └───────────────┘ └─────────────────────┘ │ -│ ┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ -│ │ websocket_manager│ │ surreal_mgr │ │ redis_manager │ │ -│ │ lines 1-296 │ │ lines 1-428 │ │ lines 1-174 │ │ -│ └──────────────────┘ └──────────────┘ └──────────────────┘ │ -│ ┌──────────────────┐ ┌──────────────────────────────────┐ │ -│ │ exceptions.py │ │ exception_handlers.py │ │ -│ └──────────────────┘ └──────────────────────────────────┘ │ -│ Protocol implementations. Database clients. WebSocket pool. │ -│ No imports from API or Services. │ +│ Core Layer (core/) │ +│ ┌─────────────────────┐ ┌─────────────────┐ │ +│ │ passkey_manager │ │ dependencies │ │ +│ │ (WebAuthn ceremony) │ │ (current_user) │ │ +│ └─────────────────────┘ └─────────────────┘ │ +│ ┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ +│ │ websocket_manager│ │ surreal_mgr │ │ redis_manager │ │ +│ │ (connection pool)│ │ (live query) │ │ (sessions+chal.) │ │ +│ └──────────────────┘ └──────────────┘ └──────────────────┘ │ +│ ┌──────────────────┐ ┌──────────────────────────────────┐ │ +│ │ exceptions.py │ │ exception_handlers.py │ │ +│ └──────────────────┘ └──────────────────────────────────┘ │ +│ Database clients. WebSocket pool. Auth dependencies. │ +│ No server-side cryptography lives here — that's the client. │ +│ No imports from API or Services. │ ├─────────────────────────────────────────────────────────────────┤ -│ Model Layer (models/) │ -│ ┌──────┐ ┌────────────┐ ┌─────────────┐ ┌──────────────┐ │ -│ │ User │ │ Credential │ │ IdentityKey │ │ SignedPrekey │ │ -│ └──────┘ └────────────┘ └─────────────┘ └──────────────┘ │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ -│ │ OneTimePrekey│ │ RatchetState │ │ SkippedMessageKey │ │ -│ └──────────────┘ └──────────────┘ └──────────────────────┘ │ -│ SQLModel ORM classes. Data structures. Validation. │ -│ Import only config constants and Base. │ +│ Model Layer (models/) │ +│ ┌──────┐ ┌────────────┐ ┌─────────────┐ ┌──────────────┐ │ +│ │ User │ │ Credential │ │ IdentityKey │ │ SignedPrekey │ │ +│ └──────┘ └────────────┘ └─────────────┘ └──────────────┘ │ +│ ┌──────────────┐ │ +│ │ OneTimePrekey│ Public-only prekey material — no privates. │ +│ └──────────────┘ │ +│ SQLModel ORM classes. Data structures. Validation. │ +│ Import only config constants and Base. │ ├─────────────────────────────────────────────────────────────────┤ -│ Schema Layer (schemas/) │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ -│ │ auth.py │ │ common.py│ │ rooms.py │ │ websocket.py │ │ -│ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │ -│ ┌──────────────┐ │ -│ │ surreal.py │ │ -│ └──────────────┘ │ -│ Pydantic request/response models. API contracts. │ -│ Serialization. No business logic. │ +│ Schema Layer (schemas/) │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ +│ │ auth.py │ │ common.py│ │ rooms.py │ │ websocket.py │ │ +│ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │ +│ ┌──────────────┐ │ +│ │ surreal.py │ │ +│ └──────────────┘ │ +│ Pydantic request/response models. API contracts. │ +│ Serialization. No business logic. │ └─────────────────────────────────────────────────────────────────┘ ``` @@ -745,169 +740,125 @@ The only exception is `websocket_service.py`, which imports `connection_manager` ### PostgreSQL Schema (Auth + Keys) -All PostgreSQL models inherit from `BaseDBModel` (`models/Base.py:20-34`), which provides `created_at` and `updated_at` timestamp fields with timezone awareness. +All PostgreSQL models inherit from `BaseDBModel` (`models/Base.py`), which provides `created_at` and `updated_at` timestamp fields with timezone awareness. -**users** (`models/User.py:24-68`) +**users** (`models/User.py`) ``` ┌────────────────────────────────────────────────────────────┐ -│ users │ +│ users │ ├──────────────────────┬──────────────────┬──────────────────┤ -│ Column │ Type │ Constraints │ +│ Column │ Type │ Constraints │ ├──────────────────────┼──────────────────┼──────────────────┤ -│ id │ UUID │ PK, default uuid4│ -│ username │ VARCHAR(50) │ UNIQUE, INDEX │ -│ display_name │ VARCHAR(100) │ NOT NULL │ -│ is_active │ BOOLEAN │ default True │ -│ is_verified │ BOOLEAN │ default False │ -│ identity_key │ VARCHAR(500) │ nullable │ -│ signed_prekey │ VARCHAR(500) │ nullable │ -│ signed_prekey_sig │ VARCHAR(500) │ nullable │ -│ one_time_prekeys │ TEXT │ nullable │ -│ created_at │ TIMESTAMP(tz) │ NOT NULL │ -│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +│ id │ UUID │ PK, default uuid4│ +│ username │ VARCHAR(50) │ UNIQUE, INDEX │ +│ display_name │ VARCHAR(100) │ NOT NULL │ +│ is_active │ BOOLEAN │ default True │ +│ is_verified │ BOOLEAN │ default False │ +│ identity_key │ VARCHAR(500) │ nullable │ +│ signed_prekey │ VARCHAR(500) │ nullable │ +│ signed_prekey_sig │ VARCHAR(500) │ nullable │ +│ one_time_prekeys │ TEXT │ nullable │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ ├──────────────────────┴──────────────────┴──────────────────┤ -│ Relationships: credentials (1:many → Credential) │ +│ Relationships: credentials (1:many → Credential) │ └────────────────────────────────────────────────────────────┘ ``` -**credentials** (`models/Credential.py:27-78`) +**credentials** (`models/Credential.py`) ``` ┌────────────────────────────────────────────────────────────┐ -│ credentials │ +│ credentials │ ├──────────────────────┬──────────────────┬──────────────────┤ -│ Column │ Type │ Constraints │ +│ Column │ Type │ Constraints │ ├──────────────────────┼──────────────────┼──────────────────┤ -│ id │ INTEGER │ PK, autoincrement│ -│ credential_id │ VARCHAR(512) │ UNIQUE, INDEX │ -│ public_key │ VARCHAR(1024) │ NOT NULL │ -│ sign_count │ INTEGER │ default 0 │ -│ aaguid │ VARCHAR(64) │ nullable │ -│ backup_eligible │ BOOLEAN │ default False │ -│ backup_state │ BOOLEAN │ default False │ -│ attestation_type │ VARCHAR(50) │ nullable │ -│ transports │ VARCHAR(200) │ nullable │ -│ user_id │ UUID │ FK → users.id │ -│ device_name │ VARCHAR(100) │ nullable │ -│ last_used_at │ TIMESTAMP(tz) │ nullable │ -│ created_at │ TIMESTAMP(tz) │ NOT NULL │ -│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +│ id │ INTEGER │ PK, autoincrement│ +│ credential_id │ VARCHAR(512) │ UNIQUE, INDEX │ +│ public_key │ VARCHAR(1024) │ NOT NULL │ +│ sign_count │ INTEGER │ default 0 │ +│ aaguid │ VARCHAR(64) │ nullable │ +│ backup_eligible │ BOOLEAN │ default False │ +│ backup_state │ BOOLEAN │ default False │ +│ attestation_type │ VARCHAR(50) │ nullable │ +│ transports │ VARCHAR(200) │ nullable │ +│ user_id │ UUID │ FK → users.id │ +│ device_name │ VARCHAR(100) │ nullable │ +│ last_used_at │ TIMESTAMP(tz) │ nullable │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ ├──────────────────────┴──────────────────┴──────────────────┤ -│ Relationships: user (many:1 → User) │ +│ Relationships: user (many:1 → User) │ └────────────────────────────────────────────────────────────┘ ``` -**identity_keys** (`models/IdentityKey.py:18-48`) +**identity_keys** (`backend/app/models/IdentityKey.py`) ``` ┌────────────────────────────────────────────────────────────┐ -│ identity_keys │ +│ identity_keys │ ├──────────────────────┬──────────────────┬──────────────────┤ -│ Column │ Type │ Constraints │ +│ Column │ Type │ Constraints │ ├──────────────────────┼──────────────────┼──────────────────┤ -│ id │ INTEGER │ PK, autoincrement│ -│ user_id │ UUID │ FK → users.id │ -│ │ │ UNIQUE, INDEX │ -│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ -│ private_key │ VARCHAR(64) │ NOT NULL (X25519)│ -│ public_key_ed25519 │ VARCHAR(64) │ NOT NULL │ -│ private_key_ed25519 │ VARCHAR(64) │ NOT NULL │ -│ created_at │ TIMESTAMP(tz) │ NOT NULL │ -│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +│ id │ INTEGER │ PK, autoincrement│ +│ user_id │ UUID │ FK → users.id │ +│ │ │ UNIQUE, INDEX │ +│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ +│ public_key_ed25519 │ VARCHAR(64) │ NOT NULL │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ └────────────────────────────────────────────────────────────┘ ``` -Note: When client-side key generation is used (`store_client_keys`, `prekey_service.py:45-150`), the `private_key` and `private_key_ed25519` fields are stored as empty strings. Only the public keys are actually stored on the server. The private key fields remain in the schema for backward compatibility with the server-side key generation path. +Public-only. The server never stores identity private keys — those live exclusively in the client's IndexedDB. -**signed_prekeys** (`models/SignedPrekey.py:20-50`) +**signed_prekeys** (`backend/app/models/SignedPrekey.py`) ``` ┌────────────────────────────────────────────────────────────┐ -│ signed_prekeys │ +│ signed_prekeys │ ├──────────────────────┬──────────────────┬──────────────────┤ -│ Column │ Type │ Constraints │ +│ Column │ Type │ Constraints │ ├──────────────────────┼──────────────────┼──────────────────┤ -│ id │ INTEGER │ PK, autoincrement│ -│ user_id │ UUID │ FK → users.id │ -│ key_id │ INTEGER │ INDEX │ -│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ -│ private_key │ VARCHAR(64) │ NOT NULL │ -│ signature │ VARCHAR(128) │ NOT NULL (Ed2551)│ -│ is_active │ BOOLEAN │ default True │ -│ expires_at │ TIMESTAMP(tz) │ nullable │ -│ created_at │ TIMESTAMP(tz) │ NOT NULL │ -│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +│ id │ INTEGER │ PK, autoincrement│ +│ user_id │ UUID │ FK → users.id │ +│ key_id │ INTEGER │ INDEX │ +│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ +│ signature │ VARCHAR(128) │ NOT NULL (Ed25519)│ +│ is_active │ BOOLEAN │ default True │ +│ expires_at │ TIMESTAMP(tz) │ nullable │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ └────────────────────────────────────────────────────────────┘ ``` -Rotation: New signed prekey every 48 hours (`SIGNED_PREKEY_ROTATION_HOURS = 48`, `config.py:76`). Old inactive prekeys retained for 7 days (`SIGNED_PREKEY_RETENTION_DAYS = 7`, `config.py:77`) then cleaned up by `cleanup_old_signed_prekeys()` (`prekey_service.py:428-465`). +Rotation: the client generates a new signed prekey roughly every 48 hours (`SIGNED_PREKEY_ROTATION_HOURS`) and re-runs `uploadKeys` to publish the new public + signature. The previous active SPK is marked `is_active=False` by `store_client_keys` so the bundle endpoint always serves the latest. -**one_time_prekeys** (`models/OneTimePrekey.py:18-45`) +**one_time_prekeys** (`backend/app/models/OneTimePrekey.py`) ``` ┌────────────────────────────────────────────────────────────┐ -│ one_time_prekeys │ +│ one_time_prekeys │ ├──────────────────────┬──────────────────┬──────────────────┤ -│ Column │ Type │ Constraints │ +│ Column │ Type │ Constraints │ ├──────────────────────┼──────────────────┼──────────────────┤ -│ id │ INTEGER │ PK, autoincrement│ -│ user_id │ UUID │ FK → users.id │ -│ key_id │ INTEGER │ INDEX │ -│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ -│ private_key │ VARCHAR(64) │ NOT NULL │ -│ is_used │ BOOLEAN │ default False │ -│ │ │ INDEX │ -│ created_at │ TIMESTAMP(tz) │ NOT NULL │ -│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +│ id │ INTEGER │ PK, autoincrement│ +│ user_id │ UUID │ FK → users.id │ +│ key_id │ INTEGER │ INDEX │ +│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ +│ is_used │ BOOLEAN │ default False │ +│ │ │ INDEX │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ └────────────────────────────────────────────────────────────┘ ``` -Each user starts with 100 OPKs (`DEFAULT_ONE_TIME_PREKEY_COUNT = 100`, `config.py:75`). When the count drops below 20, the `get_prekey_bundle` endpoint auto-replenishes (`encryption.py:47-49`). +The client uploads OPK publics in batches of 100 (`DEFAULT_ONE_TIME_PREKEY_COUNT` in `frontend/src/types/encryption.ts`) and re-uploads when its local unused-OPK count drops below half. When `get_prekey_bundle` hands one out, it sets `is_used=True` so the same OPK is never offered to two senders. -**ratchet_states** (`models/RatchetState.py:18-68`) - -``` -┌────────────────────────────────────────────────────────────┐ -│ ratchet_states │ -├────────────────────────────┬──────────────┬────────────────┤ -│ Column │ Type │ Constraints │ -├────────────────────────────┼──────────────┼────────────────┤ -│ id │ INTEGER │ PK │ -│ user_id │ UUID │ FK → users.id │ -│ peer_user_id │ UUID │ FK → users.id │ -│ dh_private_key │ VARCHAR(100K)│ nullable │ -│ dh_public_key │ VARCHAR(100K)│ nullable │ -│ dh_peer_public_key │ VARCHAR(100K)│ nullable │ -│ root_key │ VARCHAR(100K)│ NOT NULL │ -│ sending_chain_key │ VARCHAR(100K)│ NOT NULL │ -│ receiving_chain_key │ VARCHAR(100K)│ NOT NULL │ -│ sending_message_number │ INTEGER │ default 0 │ -│ receiving_message_number │ INTEGER │ default 0 │ -│ previous_sending_chain_len │ INTEGER │ default 0 │ -│ created_at │ TIMESTAMP(tz)│ NOT NULL │ -│ updated_at │ TIMESTAMP(tz)│ NOT NULL │ -└────────────────────────────────────────────────────────────┘ -``` - -There is one ratchet state per (user_id, peer_user_id) pair. The relationship is directional: Alice's ratchet state for talking to Bob is a separate row from Bob's ratchet state for talking to Alice. - -**skipped_message_keys** (`models/SkippedMessageKey.py:17-51`) - -``` -┌────────────────────────────────────────────────────────────┐ -│ skipped_message_keys │ -├──────────────────────┬──────────────────┬──────────────────┤ -│ Column │ Type │ Constraints │ -├──────────────────────┼──────────────────┼──────────────────┤ -│ id │ INTEGER │ PK, autoincrement│ -│ ratchet_state_id │ INTEGER │ FK → ratchet_ │ -│ │ │ states.id, INDEX │ -│ dh_public_key │ VARCHAR(100000) │ NOT NULL, INDEX │ -│ message_number │ INTEGER │ NOT NULL, INDEX │ -│ message_key │ VARCHAR(100000) │ NOT NULL │ -│ created_at │ TIMESTAMP(tz) │ NOT NULL │ -│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +> Note: there are no `ratchet_states` or `skipped_message_keys` tables. The Double Ratchet — including its skipped-key cache — runs entirely in the browser and is persisted in IndexedDB by `frontend/src/crypto/key-store.ts:saveRatchetState`. Putting that state on the server would mean the server held the symmetric keys, defeating E2EE. +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ └────────────────────────────────────────────────────────────┘ ``` @@ -917,68 +868,68 @@ These store message keys for out-of-order delivery. When the receiving ratchet a SurrealDB is schemaless, but these are the document structures the application creates: -**messages** (created via `surreal_manager.py:112-122`) +**messages** (created via `surreal_manager.py`) ``` { - id: "messages:ulid_here", // SurrealDB auto-generated - sender_id: "uuid_string", - recipient_id: "uuid_string", - room_id: "rooms:ulid_here" | null, - ciphertext: "base64url_encoded_bytes", - nonce: "base64url_encoded_bytes", - header: "{\"dh_public_key\":\"...\",\"message_number\":0,...}", - sender_username: "alice", - created_at: "2026-01-15T10:30:00Z", - updated_at: "2026-01-15T10:30:00Z" + id: "messages:ulid_here", // SurrealDB auto-generated + sender_id: "uuid_string", + recipient_id: "uuid_string", + room_id: "rooms:ulid_here" | null, + ciphertext: "base64url_encoded_bytes", + nonce: "base64url_encoded_bytes", + header: "{\"dh_public_key\":\"...\",\"message_number\":0,...}", + sender_username: "alice", + created_at: "2026-01-15T10:30:00Z", + updated_at: "2026-01-15T10:30:00Z" } ``` -**presence** (created via `surreal_manager.py:287-305`) +**presence** (created via `surreal_manager.py`) ``` { - id: "presence:`user_uuid`", // user-specific record ID - user_id: "uuid_string", - status: "online" | "away" | "offline", - last_seen: "2026-01-15T10:30:00Z", - updated_at: "time::now()" + id: "presence:`user_uuid`", // user-specific record ID + user_id: "uuid_string", + status: "online" | "away" | "offline", + last_seen: "2026-01-15T10:30:00Z", + updated_at: "time::now" } ``` -**rooms** (created via `surreal_manager.py:155-182`) +**rooms** (created via `surreal_manager.py`) ``` { - id: "rooms:ulid_here", - name: "Room Name", - type: "direct" | "group", - members: ["uuid1", "uuid2"], - created_at: "2026-01-15T10:30:00Z", - updated_at: "2026-01-15T10:30:00Z" + id: "rooms:ulid_here", + name: "Room Name", + type: "direct" | "group", + members: ["uuid1", "uuid2"], + created_at: "2026-01-15T10:30:00Z", + updated_at: "2026-01-15T10:30:00Z" } ``` -**room_participants** (created via `surreal_manager.py:184-221`) +**room_participants** (created via `surreal_manager.py`) ``` { - id: "room_participants:ulid_here", - room_id: "rooms:ulid_here", - user_id: "uuid_string", - role: "member" | "admin", - joined_at: "2026-01-15T10:30:00Z" + id: "room_participants:ulid_here", + room_id: "rooms:ulid_here", + user_id: "uuid_string", + role: "member" | "admin", + joined_at: "2026-01-15T10:30:00Z" } ``` ### Redis Keys ``` -webauthn:reg_challenge:{username} → 32 bytes (hex-encoded), TTL 600s -webauthn:auth_challenge:{username} → 32 bytes (hex-encoded), TTL 600s +webauthn:reg_challenge:{username} → 32 bytes (hex-encoded), TTL 600s +webauthn:auth_challenge:{username} → 32 bytes (hex-encoded), TTL 600s ``` -These are the only two key patterns currently in use. Both use the Redis pipeline GET+DELETE pattern (`redis_manager.py:86-95`) for atomic one-time consumption. The TTL ensures stale challenges are automatically cleaned up even if the client never completes the flow. +These are the only two key patterns currently in use. Both use the Redis pipeline GET+DELETE pattern (`redis_manager.py`) for atomic one-time consumption. The TTL ensures stale challenges are automatically cleaned up even if the client never completes the flow. --- @@ -995,8 +946,8 @@ These are the only two key patterns currently in use. Both use the Redis pipelin | Stolen database dump | Key separation | PostgreSQL has public keys only (client-side path). Private keys live in browser IndexedDB. OPKs are single-use. | | Phishing / credential theft | WebAuthn | Credentials are origin-bound. Cannot be replayed on a different domain. | | Replay attacks | Nonces + counters | AES-GCM nonces are random. Message numbers are sequential. WebAuthn challenges have TTL. | -| Authenticator cloning | Signature counter | `passkey_manager.py:184-193` checks that the counter strictly increases. If it does not, authentication fails with a clone detection error. | -| Message tampering | AEAD | AES-256-GCM provides authenticated encryption. Tampered ciphertext fails the GCM tag check (`double_ratchet.py:143-153`). | +| Authenticator cloning | Signature counter | `backend/app/core/passkey/passkey_manager.py` checks that the counter strictly increases. If it does not, authentication fails with a clone detection error. | +| Message tampering | AEAD | AES-256-GCM provides authenticated encryption. Tampered ciphertext fails the GCM tag check (`frontend/src/crypto/double-ratchet.ts`). | | Key compromise (single key) | Forward secrecy | Double Ratchet generates new DH keys regularly. Compromising one chain key reveals only future messages in that chain, not past messages. | **What is out of scope:** @@ -1004,7 +955,7 @@ These are the only two key patterns currently in use. Both use the Redis pipelin | Threat | Why | |---|---| | Compromised client device | If the attacker has access to the browser, they have access to IndexedDB (private keys) and can read plaintext. There is no defense against a fully compromised endpoint. | -| Side-channel attacks on crypto | The `constantTimeEqual()` function in `primitives.ts:388-397` is the extent of timing attack mitigation. Comprehensive side-channel resistance would require constant-time implementations of all crypto primitives, which the WebCrypto API generally provides but does not guarantee. | +| Side-channel attacks on crypto | The `constantTimeEqual` function in `frontend/src/crypto/primitives.ts` is the extent of timing attack mitigation. Comprehensive side-channel resistance would require constant-time implementations of all crypto primitives, which the WebCrypto API generally provides but does not guarantee. | | Metadata analysis | The server knows who messages whom, when, how often, and message sizes. Only content is protected, not metadata. Protecting metadata would require something like mixnets or onion routing, which is not implemented. | | Quantum computing | X25519 and Ed25519 are not post-quantum. A sufficiently powerful quantum computer could break them. Post-quantum key exchange (e.g., ML-KEM/Kyber) is not implemented. | | Compromised authenticator supply chain | If the authenticator hardware itself is backdoored, WebAuthn cannot detect this. The AAGUID field can identify the authenticator model but not verify its integrity. | @@ -1013,64 +964,64 @@ These are the only two key patterns currently in use. Both use the Redis pipelin ``` Layer 1: Transport Security (TLS/HTTPS via Nginx) - │ - │ Protects: Data in transit between client and server - │ Mechanism: TLS certificate, HTTPS enforcement - │ Configuration: Nginx reverse proxy with SSL termination - │ - ▼ + │ + │ Protects: Data in transit between client and server + │ Mechanism: TLS certificate, HTTPS enforcement + │ Configuration: Nginx reverse proxy with SSL termination + │ + ▼ Layer 2: Authentication (WebAuthn/Passkeys) - │ - │ Protects: Identity verification, prevents impersonation - │ Mechanism: Public-key cryptography, hardware-bound keys - │ Key files: passkey_manager.py:43-210, auth_service.py:331-598 - │ Redis: Challenge storage with 600s TTL, one-time consumption - │ - ▼ + │ + │ Protects: Identity verification, prevents impersonation + │ Mechanism: Public-key cryptography, hardware-bound keys + │ Key files: backend/app/core/passkey/passkey_manager.py, backend/app/services/auth_service.py + │ Redis: Challenge storage with 600s TTL, one-time consumption + │ + ▼ Layer 3: Key Exchange (X3DH Protocol) - │ - │ Protects: Initial shared secret establishment - │ Mechanism: 3-4 Diffie-Hellman operations + HKDF - │ Key files: x3dh_manager.py:208-350, prekey_service.py:293-361 - │ Properties: Asynchronous (works even if recipient offline) - │ Deniable (either party could have forged the exchange) - │ - ▼ + │ + │ Protects: Initial shared secret establishment + │ Mechanism: 3-4 Diffie-Hellman operations + HKDF + │ Key files: frontend/src/crypto/x3dh.ts, backend/app/services/prekey_service.py + │ Properties: Asynchronous (works even if recipient offline) + │ Deniable (either party could have forged the exchange) + │ + ▼ Layer 4: Message Encryption (Double Ratchet + AES-256-GCM) - │ - │ Protects: Message confidentiality and integrity - │ Mechanism: Symmetric ratchet (HMAC chains) + DH ratchet - │ Key files: double_ratchet.py:64-419, primitives.ts:1-397 - │ Properties: Forward secrecy per-message - │ Future secrecy (self-healing after compromise) - │ - ▼ + │ + │ Protects: Message confidentiality and integrity + │ Mechanism: Symmetric ratchet (HMAC chains) + DH ratchet + │ Key files: frontend/src/crypto/double-ratchet.ts, frontend/src/crypto/primitives.ts + │ Properties: Forward secrecy per-message + │ Future secrecy (self-healing after compromise) + │ + ▼ Layer 5: Key Lifecycle Management - │ - │ Protects: Limits blast radius of any single key compromise - │ Mechanism: SPK rotation every 48h, OPK single-use, old SPK cleanup - │ Key files: prekey_service.py:221-291 (rotation), - │ prekey_service.py:428-465 (cleanup), - │ prekey_service.py:363-407 (replenishment) - │ Constants: SIGNED_PREKEY_ROTATION_HOURS=48 (config.py:76) - │ SIGNED_PREKEY_RETENTION_DAYS=7 (config.py:77) - │ DEFAULT_ONE_TIME_PREKEY_COUNT=100 (config.py:75) - │ - ▼ + │ + │ Protects: Limits blast radius of any single key compromise + │ Mechanism: SPK rotation every 48h, OPK single-use, old SPK cleanup + │ Key files: backend/app/services/prekey_service.py (rotation), + │ backend/app/services/prekey_service.py (cleanup), + │ backend/app/services/prekey_service.py (replenishment) + │ Constants: SIGNED_PREKEY_ROTATION_HOURS=48 (config.py) + │ SIGNED_PREKEY_RETENTION_DAYS=7 (config.py) + │ DEFAULT_ONE_TIME_PREKEY_COUNT=100 (config.py) + │ + ▼ Layer 6: Rate Limiting and Abuse Prevention - │ - Protects: Against brute force and DoS - Mechanism: Per-user message rate limits, auth attempt limits - Constants: RATE_LIMIT_MESSAGES_PER_MINUTE=60 (config.py:146) - RATE_LIMIT_AUTH_ATTEMPTS=5 (config.py:147) - WS_MAX_CONNECTIONS_PER_USER=5 (config.py:141) + │ + Protects: Against brute force and DoS + Mechanism: Per-user message rate limits, auth attempt limits + Constants: RATE_LIMIT_MESSAGES_PER_MINUTE=60 (config.py) + RATE_LIMIT_AUTH_ATTEMPTS=5 (config.py) + WS_MAX_CONNECTIONS_PER_USER=5 (config.py) ``` --- ## Configuration -All configuration lives in `config.py`. The `Settings` class (lines 96-218) inherits from `pydantic_settings.BaseSettings` and loads values from environment variables and the `.env` file. +All configuration lives in `config.py`. The `Settings` class inherits from `pydantic_settings.BaseSettings` and loads values from environment variables and the `.env` file. ### Application Settings @@ -1090,11 +1041,11 @@ All configuration lives in `config.py`. The `Settings` class (lines 96-218) inhe | `POSTGRES_DB` | `"chat_auth"` | Database name | | `POSTGRES_USER` | `"chat_user"` | Database user | | `POSTGRES_PASSWORD` | `""` | Database password | -| `DATABASE_URL` | (auto-built) | Full connection string, assembled by `field_validator` at line 149 | +| `DATABASE_URL` | (auto-built) | Full connection string, assembled by `field_validator` | | `DB_POOL_SIZE` | `20` | SQLAlchemy connection pool size | | `DB_MAX_OVERFLOW` | `40` | Pool overflow connections | -The `assemble_db_connection` validator (`config.py:149-161`) builds the URL from components if `DATABASE_URL` is not explicitly set. The URL uses the `postgresql+asyncpg://` scheme for async connections. +The `assemble_db_connection` validator (`config.py`) builds the URL from components if `DATABASE_URL` is not explicitly set. The URL uses the `postgresql+asyncpg://` scheme for async connections. ### SurrealDB Settings @@ -1106,7 +1057,7 @@ The `assemble_db_connection` validator (`config.py:149-161`) builds the URL from | `SURREAL_PASSWORD` | (required) | SurrealDB password | | `SURREAL_NAMESPACE` | `"chat"` | SurrealDB namespace | | `SURREAL_DATABASE` | `"production"` | SurrealDB database | -| `SURREAL_URL` | (auto-built) | WebSocket URL, assembled at line 163 | +| `SURREAL_URL` | (auto-built) | WebSocket URL, assembled | ### Redis Settings @@ -1115,7 +1066,7 @@ The `assemble_db_connection` validator (`config.py:149-161`) builds the URL from | `REDIS_HOST` | `"localhost"` | Redis host | | `REDIS_PORT` | `6379` | Redis port | | `REDIS_PASSWORD` | `""` | Redis password (optional) | -| `REDIS_URL` | (auto-built) | Connection URL, assembled at line 178 | +| `REDIS_URL` | (auto-built) | Connection URL, assembled | ### WebAuthn Settings @@ -1145,42 +1096,42 @@ The `RP_ID` must match the domain the browser sees. In production, this would be ### Database Connection Pooling -PostgreSQL pool: 20 base connections + 40 overflow = 60 max simultaneous connections (`config.py:118-119`). The engine uses `pool_pre_ping=True` (`models/Base.py:43`) to detect and replace stale connections before queries fail. +PostgreSQL pool: 20 base connections + 40 overflow = 60 max simultaneous connections (`config.py`). The engine uses `pool_pre_ping=True` (`models/Base.py`) to detect and replace stale connections before queries fail. -SQLAlchemy async sessions are created per-request via the `get_session()` dependency (`models/Base.py:54-59`). Sessions are recycled automatically after each request. +SQLAlchemy async sessions are created per-request via the `get_session` dependency (`models/Base.py`). Sessions are recycled automatically after each request. -Redis pool: 50 connections (`redis_manager.py:39`). Connection pooling is handled by `redis.asyncio.ConnectionPool`. +Redis pool: 50 connections (`redis_manager.py`). Connection pooling is handled by `redis.asyncio.ConnectionPool`. -SurrealDB: Single persistent WebSocket connection per application instance (`surreal_manager.py:47-48`). The `ensure_connected()` method (`surreal_manager.py:73-78`) lazily reconnects if the connection drops. +SurrealDB: Single persistent WebSocket connection per application instance (`surreal_manager.py`). The `ensure_connected` method (`surreal_manager.py`) lazily reconnects if the connection drops. ### Message Throughput -60 messages per minute per user rate limit (`config.py:146`). This translates to one message per second sustained, which is reasonable for a chat application. +60 messages per minute per user rate limit (`config.py`). This translates to one message per second sustained, which is reasonable for a chat application. -WebSocket heartbeat every 30 seconds (`config.py:140`). This is frequent enough to detect dead connections quickly but not so frequent as to waste bandwidth on idle connections. +WebSocket heartbeat every 30 seconds (`config.py`). This is frequent enough to detect dead connections quickly but not so frequent as to waste bandwidth on idle connections. -SurrealDB live queries are push-based (`surreal_manager.py:341-359`). When a new message is created, SurrealDB pushes it to the subscribed callback immediately. There is no polling interval, so latency is limited to network round-trip plus SurrealDB processing time. +SurrealDB live queries are push-based (`surreal_manager.py`). When a new message is created, SurrealDB pushes it to the subscribed callback immediately. There is no polling interval, so latency is limited to network round-trip plus SurrealDB processing time. ### Encryption Overhead Approximate per-message encryption cost (based on typical X25519/AES-256-GCM performance): ``` -X25519 DH exchange: ~0.1ms +X25519 DH exchange: ~0.1ms HMAC-SHA256 chain step: ~0.01ms -AES-256-GCM encrypt: ~0.001ms per KB (for typical chat messages) -HKDF derivation: ~0.01ms +AES-256-GCM encrypt: ~0.001ms per KB (for typical chat messages) +HKDF derivation: ~0.01ms -Total per-message: < 1ms +Total per-message: < 1ms ``` This is negligible compared to network latency (typically 10-100ms). Encryption overhead is not a bottleneck for this application. ### Connection Limits -5 WebSocket connections per user (`config.py:141`). This supports a reasonable number of devices (phone, laptop, tablet, desktop, secondary browser) without allowing a single user to exhaust server WebSocket capacity. +5 WebSocket connections per user (`config.py`). This supports a reasonable number of devices (phone, laptop, tablet, desktop, secondary browser) without allowing a single user to exhaust server WebSocket capacity. -The `ConnectionManager` delivers each message to all of a user's connections (`websocket_manager.py:135-153`), so the cost of multi-device scales linearly with the number of connections per user. +The `ConnectionManager` delivers each message to all of a user's connections (`websocket_manager.py`), so the cost of multi-device scales linearly with the number of connections per user. --- @@ -1212,13 +1163,11 @@ SolidJS also has a smaller bundle size than React, which helps with initial load The tradeoff is ecosystem size. React has far more libraries, tutorials, and Stack Overflow answers. SolidJS is smaller and you occasionally need to build something that React would have a library for. -### Why Separate Server and Client Encryption Paths? +### Why Only a Client-Side Encryption Path? -`message_service.py` has both `send_encrypted_message` (server-side encryption, line 316, marked `[DEPRECATED]`) and `store_encrypted_message` (client-side passthrough, line 269). The server-side path exists because the system was originally built with server-side encryption and is being migrated to full client-side encryption. +An earlier draft of this project carried two encryption paths — a server-side one that loaded ratchet state from PostgreSQL and encrypted on the backend, and a client-side passthrough that stored opaque ciphertext. The audit (see `docs/plans/`) deleted the server-side path entirely, including its `RatchetState` and `SkippedMessageKey` tables and the entire `app/core/encryption/` Python module. -In the server-side path, the server loads the ratchet state, encrypts the plaintext, and stores the ciphertext. This means the server momentarily has access to the plaintext. In the client-side path, the server never sees the plaintext; it receives and stores ciphertext as-is. - -The client-side path is the correct production path. The server-side path remains for backwards compatibility and as a fallback during the migration. +Why? Because as long as a server-side X3DH/Double Ratchet implementation exists, the schema columns it needs (private keys, ratchet roots) exist, and any caller can hit it — the server has the technical capability to read user messages. "Marked deprecated" is a comment, not a defense. The only way to honestly claim end-to-end encryption is to make it physically impossible for the server to participate. So now `message_service.py` has exactly one method, `store_encrypted_message`, and it does nothing but pass `ciphertext` / `nonce` / `header` through to SurrealDB. --- @@ -1230,25 +1179,25 @@ The client-side path is the correct production path. The server-side path remain docker compose -f dev.compose.yml up ┌───────────────────────────────────────────────────────────────┐ -│ Docker Network (chat_network_dev) │ -│ │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ PostgreSQL │ │ SurrealDB │ │ Redis │ │ -│ │ 16-alpine │ │ latest │ │ 8-alpine │ │ -│ │ :5432→5432 │ │ :8000→8001 │ │ :6379→6379 │ │ -│ │ │ │ file://data│ │ appendonly │ │ -│ └─────────────┘ └─────────────┘ └─────────────┘ │ -│ │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ FastAPI │ │ Vite Dev │ │ Nginx │ │ -│ │ :8000→8000 │ │ :5173→5173 │ │ :80→80 │ │ -│ │ uvicorn │ │ HMR enabled│ │ reverse │ │ -│ │ --reload │ │ hot module │ │ proxy │ │ -│ │ vol: ./back │ │ vol: ./fro │ │ dev.nginx │ │ -│ │ end │ │ ntend │ │ config │ │ -│ └─────────────┘ └─────────────┘ └─────────────┘ │ -│ │ -│ Source: dev.compose.yml │ +│ Docker Network (chat_network_dev) │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ PostgreSQL │ │ SurrealDB │ │ Redis │ │ +│ │ 16-alpine │ │ latest │ │ 8-alpine │ │ +│ │ :5432→5432 │ │ :8000→8001 │ │ :6379→6379 │ │ +│ │ │ │ file://data│ │ appendonly │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ FastAPI │ │ Vite Dev │ │ Nginx │ │ +│ │ :8000→8000 │ │ :5173→5173 │ │ :80→80 │ │ +│ │ uvicorn │ │ HMR enabled│ │ reverse │ │ +│ │ --reload │ │ hot module │ │ proxy │ │ +│ │ vol: ./back │ │ vol: ./fro │ │ dev.nginx │ │ +│ │ end │ │ ntend │ │ config │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ Source: dev.compose.yml │ └───────────────────────────────────────────────────────────────┘ ``` @@ -1260,39 +1209,39 @@ In development, all services expose ports to the host for direct access and debu docker compose up ┌───────────────────────────────────────────────────────────────┐ -│ Docker Network (chat_network) │ -│ │ -│ ┌──────────────────────────────────────────────────────────┐ │ -│ │ Nginx (:80, :443) │ │ -│ │ │ │ -│ │ /api/* ────────► upstream: backend:8000 (HTTP) │ │ -│ │ /ws ────────► upstream: backend:8000 (WebSocket) │ │ -│ │ /* ────────► Static SolidJS build (from image) │ │ -│ │ │ │ -│ └──────────────────────────┬───────────────────────────────┘ │ -│ │ │ -│ ┌──────────────────────────┼──────────────────────────────┐ │ -│ │ │ │ │ -│ │ ┌─────────────┐ ┌─────┴─────┐ ┌─────────────┐ │ │ -│ │ │ PostgreSQL │ │ FastAPI │ │ SurrealDB │ │ │ -│ │ │ 16-alpine │ │ gunicorn │ │ latest │ │ │ -│ │ │ vol: data │ │ workers │ │ vol: data │ │ │ -│ │ │ healthcheck │ │ no ports │ │ healthcheck│ │ │ -│ │ │ restart: │ │ exposed │ │ restart: │ │ │ -│ │ │ always │ │ to host │ │ always │ │ │ -│ │ └─────────────┘ └───────────┘ └─────────────┘ │ │ -│ │ │ │ -│ │ ┌─────────────┐ │ │ -│ │ │ Redis │ │ │ -│ │ │ 8-alpine │ │ │ -│ │ │ vol: data │ │ │ -│ │ │ maxmem 2gb │ │ │ -│ │ │ LRU evict │ │ │ -│ │ └─────────────┘ │ │ -│ │ │ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ │ -│ Source: compose.yml │ +│ Docker Network (chat_network) │ +│ │ +│ ┌──────────────────────────────────────────────────────────┐ │ +│ │ Nginx (:80, :443) │ │ +│ │ │ │ +│ │ /api/* ────────► upstream: backend:8000 (HTTP) │ │ +│ │ /ws ────────► upstream: backend:8000 (WebSocket) │ │ +│ │ /* ────────► Static SolidJS build (from image) │ │ +│ │ │ │ +│ └──────────────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ┌──────────────────────────┼──────────────────────────────┐ │ +│ │ │ │ │ +│ │ ┌─────────────┐ ┌─────┴─────┐ ┌─────────────┐ │ │ +│ │ │ PostgreSQL │ │ FastAPI │ │ SurrealDB │ │ │ +│ │ │ 16-alpine │ │ gunicorn │ │ latest │ │ │ +│ │ │ vol: data │ │ workers │ │ vol: data │ │ │ +│ │ │ healthcheck │ │ no ports │ │ healthcheck│ │ │ +│ │ │ restart: │ │ exposed │ │ restart: │ │ │ +│ │ │ always │ │ to host │ │ always │ │ │ +│ │ └─────────────┘ └───────────┘ └─────────────┘ │ │ +│ │ │ │ +│ │ ┌─────────────┐ │ │ +│ │ │ Redis │ │ │ +│ │ │ 8-alpine │ │ │ +│ │ │ vol: data │ │ │ +│ │ │ maxmem 2gb │ │ │ +│ │ │ LRU evict │ │ │ +│ │ └─────────────┘ │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ Source: compose.yml │ └───────────────────────────────────────────────────────────────┘ ``` @@ -1306,30 +1255,28 @@ The compose file (`compose.yml`) uses `depends_on` with `condition: service_heal ### Custom Exception Hierarchy -Defined in `core/exceptions.py` (lines 1-95): +Defined in `backend/app/core/exceptions.py`: ``` AppException (base) - ├── UserExistsError → 409 Conflict - ├── UserNotFoundError → 404 Not Found - ├── UserInactiveError → 403 Forbidden - ├── CredentialNotFoundError → 404 Not Found - ├── CredentialVerificationError → 401 Unauthorized - ├── ChallengeExpiredError → 400 Bad Request - ├── DatabaseError → 500 Internal Server Error - ├── AuthenticationError → 401 Unauthorized - ├── InvalidDataError → 400 Bad Request - ├── EncryptionError → 500 Internal Server Error - ├── DecryptionError → 500 Internal Server Error - ├── RatchetStateNotFoundError → 404 Not Found - └── KeyExchangeError → 500 Internal Server Error + ├── UserExistsError → 409 Conflict + ├── UserNotFoundError → 404 Not Found + ├── UserInactiveError → 403 Forbidden + ├── CredentialNotFoundError → 404 Not Found + ├── CredentialVerificationError → 401 Unauthorized + ├── ChallengeExpiredError → 400 Bad Request + ├── DatabaseError → 500 Internal Server Error + ├── AuthenticationError → 401 Unauthorized + └── InvalidDataError → 400 Bad Request ``` +`slowapi.errors.RateLimitExceeded` is also handled here and maps to 429 Too Many Requests. + ### Exception Handlers -Registered in `core/exception_handlers.py` (lines 221-246) via `register_exception_handlers(app)`, which is called from `factory.py:89`. Each exception type maps to a handler function that returns a `JSONResponse` with the appropriate HTTP status code. +Registered in `backend/app/core/exception_handlers.py` via `register_exception_handlers(app)`, called from `factory.py`. Each exception type maps to a handler function that returns a `JSONResponse` with the appropriate HTTP status code. -For security-sensitive errors (`DatabaseError`, `EncryptionError`, `DecryptionError`, `KeyExchangeError`), the response body contains a generic message ("Internal server error", "Encryption failed", etc.) rather than the actual error detail. The detail is logged server-side but not exposed to the client. +For security-sensitive errors (`DatabaseError`), the response body contains a generic message ("Internal server error") rather than the actual error detail. The detail is logged server-side but not exposed to the client. ### WebSocket Error Format @@ -1337,13 +1284,13 @@ WebSocket errors are sent as JSON to the client: ```json { - "type": "error", - "error_code": "invalid_json", - "error_message": "Invalid JSON format" + "type": "error", + "error_code": "invalid_json", + "error_message": "Invalid JSON format" } ``` -Error codes include: `max_connections`, `database_error`, `invalid_json`, `missing_type`, `unknown_type`, `processing_error`. These are defined inline in `websocket.py` (lines 62-76) and `websocket_manager.py` (lines 58-62). +Error codes include: `max_connections`, `database_error`, `invalid_json`, `missing_type`, `unknown_type`, `processing_error`. These are defined inline in `websocket.py` and `websocket_manager.py` . --- @@ -1353,10 +1300,10 @@ Where to add new features: | What | Where | Steps | |---|---|---| -| New API endpoint | `api/` directory | 1. Create router in `api/new_feature.py` 2. Create service in `services/new_feature_service.py` 3. Register router in `factory.py` (after line 113) | -| New database model | `models/` directory | 1. Create model class inheriting `BaseDBModel` 2. Import it somewhere that loads at startup (e.g., `models/__init__.py`) 3. `init_db()` auto-creates the table via `SQLModel.metadata.create_all` | -| New encryption algorithm | `core/encryption/` | 1. Add implementation in `core/encryption/` 2. Add config constants in `config.py` 3. Wire into `message_service.py` or `crypto-service.ts` | -| New WebSocket message type | `websocket_service.py` | 1. Add constant in `config.py` (WS_MESSAGE_TYPE_*) 2. Add handler method in `WebSocketService` 3. Add routing case in `route_message()` | +| New API endpoint | `api/` directory | 1. Create router in `api/new_feature.py` 2. Create service in `services/new_feature_service.py` 3. Register router in `factory.py` (after) | +| New database model | `models/` directory | 1. Create model class inheriting `BaseDBModel` 2. Import it somewhere that loads at startup (e.g., `models/__init__.py`) 3. `init_db` auto-creates the table via `SQLModel.metadata.create_all` | +| New encryption algorithm | `frontend/src/crypto/` | 1. Add implementation in `frontend/src/crypto/` 2. Add config constants in `config.py` 3. Wire into `message_service.py` or `crypto-service.ts` | +| New WebSocket message type | `websocket_service.py` | 1. Add constant in `config.py` (WS_MESSAGE_TYPE_*) 2. Add handler method in `WebSocketService` 3. Add routing case in `route_message` | | New SurrealDB collection | `surreal_manager.py` | 1. Add CRUD methods in `SurrealDBManager` 2. Add response schema in `schemas/surreal.py` | --- @@ -1371,62 +1318,48 @@ These are known architectural limitations, not bugs: 3. **Metadata not protected.** The server knows who sends messages to whom, when, how frequently, and the approximate size of each message. Only the content is encrypted. Metadata protection would require techniques like onion routing, padding, or dummy traffic, all of which add significant complexity and performance cost. -4. **No message deletion or expiry.** Once a message is stored in SurrealDB, it stays there indefinitely. There is no TTL on messages and no "delete for everyone" feature. SurrealDB does support the `_schedule_room_deletion` method for ephemeral rooms (`surreal_manager.py:393-425`), but this is room-level, not message-level. +4. **No message deletion or expiry.** Once a message is stored in SurrealDB, it stays there indefinitely. There is no TTL on messages and no "delete for everyone" feature. SurrealDB does support the `_schedule_room_deletion` method for ephemeral rooms (`surreal_manager.py`), but this is room-level, not message-level. 5. **Single-region deployment.** The Docker Compose setup assumes all services run on one machine or one cluster. There is no geo-distribution, no CDN for the frontend, and no database replication. For a production deployment serving users across regions, you would need to add these. -6. **No offline message queue.** If a recipient is offline when a message is sent, the message is stored in SurrealDB but only delivered when the recipient connects and the live query fires. There is no explicit mechanism for fetching missed messages on reconnect beyond the live query catching up. The `get_room_messages` method (`surreal_manager.py:124-153`) exists for fetching message history, but the client must explicitly call it. +6. **No offline message queue.** If a recipient is offline when a message is sent, the message is stored in SurrealDB but only delivered when the recipient connects and the live query fires. There is no explicit mechanism for fetching missed messages on reconnect beyond the live query catching up. The `get_room_messages` method (`surreal_manager.py`) exists for fetching message history, but the client must explicitly call it. --- ## Key Files Reference -| What | Where | Lines | -|---|---|---| -| App factory | `factory.py` | 63-115 | -| Lifespan (DB init) | `factory.py` | 39-61 | -| All settings | `config.py` | 96-218 | -| Crypto constants | `config.py` | 64-77 | -| X3DH protocol | `core/encryption/x3dh_manager.py` | 56-353 | -| X3DH sender | `core/encryption/x3dh_manager.py` | 208-281 | -| X3DH receiver | `core/encryption/x3dh_manager.py` | 283-350 | -| Double Ratchet | `core/encryption/double_ratchet.py` | 64-419 | -| Encrypt message | `core/encryption/double_ratchet.py` | 323-362 | -| Decrypt message | `core/encryption/double_ratchet.py` | 364-416 | -| WebAuthn manager | `core/passkey/passkey_manager.py` | 43-210 | -| Clone detection | `core/passkey/passkey_manager.py` | 184-193 | -| WebSocket pool | `core/websocket_manager.py` | 31-296 | -| Heartbeat loop | `core/websocket_manager.py` | 177-201 | -| Live query sub | `core/websocket_manager.py` | 203-223 | -| Live msg handler | `core/websocket_manager.py` | 225-251 | -| SurrealDB client | `core/surreal_manager.py` | 28-428 | -| Redis client | `core/redis_manager.py` | 19-174 | -| Exception types | `core/exceptions.py` | 7-95 | -| Exception handlers | `core/exception_handlers.py` | 221-246 | -| Message storage | `services/message_service.py` | 269-314 | -| Conversation init | `services/message_service.py` | 48-166 | -| Key management | `services/prekey_service.py` | 41-468 | -| Client key upload | `services/prekey_service.py` | 45-150 | -| Prekey bundle | `services/prekey_service.py` | 293-361 | -| SPK rotation | `services/prekey_service.py` | 221-291 | -| OPK replenish | `services/prekey_service.py` | 363-407 | -| Auth service | `services/auth_service.py` | 47-601 | -| WS message router | `services/websocket_service.py` | 35-324 | -| Auth endpoints | `api/auth.py` | 31-103 | -| Encryption endpoints | `api/encryption.py` | 21-127 | -| WebSocket endpoint | `api/websocket.py` | 25-84 | -| User model | `models/User.py` | 24-68 | -| Credential model | `models/Credential.py` | 27-78 | -| Identity key model | `models/IdentityKey.py` | 18-48 | -| Signed prekey model | `models/SignedPrekey.py` | 20-50 | -| One-time prekey model | `models/OneTimePrekey.py` | 18-45 | -| Ratchet state model | `models/RatchetState.py` | 18-68 | -| Skipped keys model | `models/SkippedMessageKey.py` | 17-51 | -| DB engine + sessions | `models/Base.py` | 37-67 | -| Client crypto prims | `frontend/src/crypto/primitives.ts` | 1-397 | -| Client crypto svc | `frontend/src/crypto/crypto-service.ts` | - | -| Client double ratchet | `frontend/src/crypto/double-ratchet.ts` | - | -| Client X3DH | `frontend/src/crypto/x3dh.ts` | - | -| Client key store | `frontend/src/crypto/key-store.ts` | - | -| Production compose | `compose.yml` | 1-118 | -| Development compose | `dev.compose.yml` | 1-151 | +| What | Where | +|---|---| +| App factory | `backend/app/factory.py` | +| Lifespan (DB init) | `backend/app/factory.py` (`lifespan`) | +| Settings + crypto constants | `backend/app/config.py` | +| Auth dependency | `backend/app/core/dependencies.py` (`current_user`) | +| Session helpers | `backend/app/core/dependencies.py` (`issue_session`, `revoke_session`) | +| WebAuthn manager | `backend/app/core/passkey/passkey_manager.py` | +| Clone detection | `passkey_manager.PasskeyManager.verify_authentication` | +| WebSocket pool | `backend/app/core/websocket_manager.py` | +| SurrealDB client | `backend/app/core/surreal_manager.py` | +| Redis client (sessions, challenges) | `backend/app/core/redis_manager.py` | +| Exception types | `backend/app/core/exceptions.py` | +| Exception handlers | `backend/app/core/exception_handlers.py` | +| Pass-through message storage | `backend/app/services/message_service.py` (`store_encrypted_message`) | +| Public prekey storage + bundle lookup | `backend/app/services/prekey_service.py` | +| Auth service | `backend/app/services/auth_service.py` | +| WS message router (membership-checked) | `backend/app/services/websocket_service.py` | +| Auth endpoints (incl. `/auth/me`, `/auth/logout`) | `backend/app/api/auth.py` | +| Encryption endpoints (bundle + upload) | `backend/app/api/encryption.py` | +| Rooms (membership-gated CRUD) | `backend/app/api/rooms.py` | +| WebSocket endpoint (cookie-authenticated) | `backend/app/api/websocket.py` | +| User model (with `webauthn_user_handle`) | `backend/app/models/User.py` | +| Credential model | `backend/app/models/Credential.py` | +| Identity public keys | `backend/app/models/IdentityKey.py` | +| Signed prekey publics | `backend/app/models/SignedPrekey.py` | +| One-time prekey publics | `backend/app/models/OneTimePrekey.py` | +| DB engine + sessions | `backend/app/models/Base.py` | +| Client crypto primitives | `frontend/src/crypto/primitives.ts` | +| Client crypto orchestration | `frontend/src/crypto/crypto-service.ts` | +| Client Double Ratchet | `frontend/src/crypto/double-ratchet.ts` | +| Client X3DH | `frontend/src/crypto/x3dh.ts` | +| Client IndexedDB key store | `frontend/src/crypto/key-store.ts` | +| Production compose | `compose.yml` | +| Development compose | `dev.compose.yml` | diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md index a3760484..c170c7cf 100644 --- a/PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md @@ -8,99 +8,95 @@ This document walks through every significant piece of the encrypted P2P chat, f ``` backend/ - app/ - api/ - auth.py # WebAuthn registration/login endpoints - encryption.py # Key upload, prekey bundle retrieval - rooms.py # Chat room creation and listing - websocket.py # WebSocket endpoint (receives raw frames) - core/ - encryption/ - x3dh_manager.py # X3DH key exchange: generate, sign, exchange - double_ratchet.py # Double Ratchet: KDF chains, encrypt, decrypt - passkey/ - passkey_manager.py # WebAuthn: registration, auth, clone detection - websocket_manager.py # Connection pool, heartbeats, live query sub - surreal_manager.py # SurrealDB client (messages, live queries) - redis_manager.py # Redis client (challenges, presence) - exceptions.py # Custom exception hierarchy - exception_handlers.py # FastAPI exception-to-HTTP mappings - enums.py # Presence status enum - models/ - Base.py # SQLModel engine, async session maker - User.py # User table (id, username, display_name) - Credential.py # WebAuthn credential storage - IdentityKey.py # X25519 + Ed25519 identity keypairs - SignedPrekey.py # Signed prekeys with rotation/expiry - OneTimePrekey.py # Single-use one-time prekeys - RatchetState.py # Serialized Double Ratchet state per conversation - SkippedMessageKey.py # Persisted skipped message keys - schemas/ - auth.py # Pydantic schemas for registration/auth flows - websocket.py # Pydantic schemas for WS message types - surreal.py # Pydantic schemas for SurrealDB documents - rooms.py # Pydantic schemas for room operations - common.py # Shared response schemas - services/ - auth_service.py # User creation, credential management - prekey_service.py # Key initialization, rotation, bundle retrieval - message_service.py # Conversation init, encrypt, store, decrypt - websocket_service.py # WS message routing (type dispatch) - presence_service.py # Online/offline/away status via Redis - config.py # All constants + Settings (env vars) - factory.py # FastAPI app factory with middleware - main.py # Uvicorn entry point + app/ + api/ + auth.py # WebAuthn registration/login endpoints + encryption.py # Key upload, prekey bundle retrieval + rooms.py # Chat room creation and listing + websocket.py # WebSocket endpoint (receives raw frames) + core/ + passkey/ + passkey_manager.py # WebAuthn: registration, auth, clone detection + dependencies.py # current_user, issue_session, revoke_session + websocket_manager.py # Connection pool, heartbeats, live query sub + surreal_manager.py # SurrealDB client (messages, live queries) + redis_manager.py # Redis client (sessions, challenges) + exceptions.py # Custom exception hierarchy + exception_handlers.py # FastAPI exception-to-HTTP mappings + enums.py # Presence status enum + models/ + Base.py # SQLModel engine, async session maker + User.py # User table (id, username, webauthn_user_handle, ...) + Credential.py # WebAuthn credential storage (BE/BS, sign_count) + IdentityKey.py # X25519 + Ed25519 identity public keys (no privates) + SignedPrekey.py # Signed prekey publics + signature + OneTimePrekey.py # Single-use prekey publics (is_used flag) + schemas/ + auth.py # Pydantic schemas for registration/auth flows + websocket.py # Pydantic schemas for WS message types + surreal.py # Pydantic schemas for SurrealDB documents + rooms.py # Pydantic schemas for room operations + common.py # Shared response schemas + services/ + auth_service.py # User creation, credential management, search + prekey_service.py # Public-only prekey storage + bundle lookup + message_service.py # Pass-through ciphertext writer + websocket_service.py # WS routing + per-user rate cap + membership + presence_service.py # Online/offline/away status via SurrealDB + config.py # All constants + Settings (env vars) + factory.py # FastAPI app factory with middleware + main.py # Uvicorn entry point frontend/ - src/ - crypto/ - primitives.ts # Low-level WebCrypto: X25519, Ed25519, AES-GCM, HKDF, HMAC - x3dh.ts # Client-side X3DH: generate keys, initiate, receive - double-ratchet.ts # Client-side Double Ratchet: init, encrypt, decrypt, serialize - crypto-service.ts # High-level orchestrator: session management, key upload - key-store.ts # IndexedDB persistence for all key material - message-store.ts # Message history persistence - websocket/ - websocket-manager.ts # WS connection, reconnect, heartbeat, message queue - message-handlers.ts # Dispatch incoming WS messages to stores - stores/ - auth.store.ts # Authentication state (nanostores) - session.store.ts # Session token management - messages.store.ts # Message list per room - rooms.store.ts # Room list and active room - presence.store.ts # Online status per user - typing.store.ts # Typing indicator state - ui.store.ts # UI state (sidebar open, modals) - settings.store.ts # User preferences - services/ - auth.service.ts # WebAuthn browser API calls - room.service.ts # Room CRUD via API - components/ - Auth/ # Login, Register, PasskeyButton - Chat/ # MessageList, ChatInput, ConversationList, etc. - Layout/ # AppShell, Sidebar, Header, ProtectedRoute - UI/ # Button, Modal, Input, Tooltip, etc. - pages/ - Chat.tsx # Main chat page (conversation + message pane) - Login.tsx # Login page - Register.tsx # Registration page - Home.tsx # Landing page - NotFound.tsx # 404 - types/ - encryption.ts # Type definitions for crypto structures - chat.ts # Message and room types - websocket.ts # WS message types - auth.ts # Auth types - guards.ts # Runtime type guards for WS messages - api.ts # API response types - components.ts # Component prop types - index.ts # Re-exports + constants - lib/ - api-client.ts # HTTP client for REST endpoints - base64.ts # Base64 helpers - validators.ts # Input validation - date.ts # Date formatting - config.ts # Frontend constants (URLs, timeouts) + src/ + crypto/ + primitives.ts # Low-level WebCrypto: X25519, Ed25519, AES-GCM, HKDF, HMAC + x3dh.ts # Client-side X3DH: generate keys, initiate, receive + double-ratchet.ts # Client-side Double Ratchet: init, encrypt, decrypt, serialize + crypto-service.ts # High-level orchestrator: session management, key upload + key-store.ts # IndexedDB persistence for all key material + message-store.ts # Message history persistence + websocket/ + websocket-manager.ts # WS connection, reconnect, heartbeat, message queue + message-handlers.ts # Dispatch incoming WS messages to stores + stores/ + auth.store.ts # Authentication state (nanostores) + session.store.ts # Session token management + messages.store.ts # Message list per room + rooms.store.ts # Room list and active room + presence.store.ts # Online status per user + typing.store.ts # Typing indicator state + ui.store.ts # UI state (sidebar open, modals) + settings.store.ts # User preferences + services/ + auth.service.ts # WebAuthn browser API calls + room.service.ts # Room CRUD via API + components/ + Auth/ # Login, Register, PasskeyButton + Chat/ # MessageList, ChatInput, ConversationList, etc. + Layout/ # AppShell, Sidebar, Header, ProtectedRoute + UI/ # Button, Modal, Input, Tooltip, etc. + pages/ + Chat.tsx # Main chat page (conversation + message pane) + Login.tsx # Login page + Register.tsx # Registration page + Home.tsx # Landing page + NotFound.tsx # 404 + types/ + encryption.ts # Type definitions for crypto structures + chat.ts # Message and room types + websocket.ts # WS message types + auth.ts # Auth types + guards.ts # Runtime type guards for WS messages + api.ts # API response types + components.ts # Component prop types + index.ts # Re-exports + constants + lib/ + api-client.ts # HTTP client for REST endpoints + base64.ts # Base64 helpers + validators.ts # Input validation + date.ts # Date formatting + config.ts # Frontend constants (URLs, timeouts) ``` --- @@ -111,30 +107,30 @@ X3DH (Extended Triple Diffie-Hellman) solves a specific problem: Alice wants to ### Step 1: Key Generation -The foundation of X3DH is generating two types of identity keypairs per user. Here is the X25519 identity key generation from `x3dh_manager.py:60-86`: +The foundation of X3DH is generating two types of identity keypairs per user. Here is the X25519 identity key generation from `frontend/src/crypto/x3dh.ts`: ```python def generate_identity_keypair_x25519(self) -> tuple[str, str]: - private_key = X25519PrivateKey.generate() - public_key = private_key.public_key() + private_key = X25519PrivateKey.generate + public_key = private_key.public_key - private_bytes = private_key.private_bytes( - encoding = serialization.Encoding.Raw, - format = serialization.PrivateFormat.Raw, - encryption_algorithm = serialization.NoEncryption() - ) - public_bytes = public_key.public_bytes( - encoding = serialization.Encoding.Raw, - format = serialization.PublicFormat.Raw - ) + private_bytes = private_key.private_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PrivateFormat.Raw, + encryption_algorithm = serialization.NoEncryption + ) + public_bytes = public_key.public_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PublicFormat.Raw + ) - return ( - bytes_to_base64url(private_bytes), - bytes_to_base64url(public_bytes) - ) + return ( + bytes_to_base64url(private_bytes), + bytes_to_base64url(public_bytes) + ) ``` -There is also `generate_identity_keypair_ed25519` at `x3dh_manager.py:88-114`, which follows the exact same pattern but uses `Ed25519PrivateKey.generate()` instead. +There is also `generate_identity_keypair_ed25519` at `frontend/src/crypto/x3dh.ts`, which follows the exact same pattern but uses `Ed25519PrivateKey.generate` instead. **Why Raw encoding?** PEM and DER formats include metadata headers, algorithm identifiers, and ASN.1 structure. That is wasted bytes when you already know the algorithm on both sides. Raw encoding gives you exactly 32 bytes for X25519 and 32 bytes for Ed25519. That is the minimum footprint, and it maps directly to what the WebCrypto API expects on the frontend when importing with `importKey("raw", ...)`. @@ -144,88 +140,88 @@ There is also `generate_identity_keypair_ed25519` at `x3dh_manager.py:88-114`, w ### Step 2: Signed Prekey Creation -The Signed Prekey (SPK) is a semi-static X25519 key that gets rotated every 48 hours. Here is `generate_signed_prekey` from `x3dh_manager.py:116-152`: +The Signed Prekey (SPK) is a semi-static X25519 key that gets rotated every 48 hours. Here is `generate_signed_prekey` from `frontend/src/crypto/x3dh.ts`: ```python def generate_signed_prekey(self, - identity_private_key_ed25519: str) -> tuple[str, - str, - str]: - spk_private = X25519PrivateKey.generate() - spk_public = spk_private.public_key() + identity_private_key_ed25519: str) -> tuple[str, + str, + str]: + spk_private = X25519PrivateKey.generate + spk_public = spk_private.public_key - spk_private_bytes = spk_private.private_bytes( - encoding = serialization.Encoding.Raw, - format = serialization.PrivateFormat.Raw, - encryption_algorithm = serialization.NoEncryption() - ) - spk_public_bytes = spk_public.public_bytes( - encoding = serialization.Encoding.Raw, - format = serialization.PublicFormat.Raw - ) + spk_private_bytes = spk_private.private_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PrivateFormat.Raw, + encryption_algorithm = serialization.NoEncryption + ) + spk_public_bytes = spk_public.public_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PublicFormat.Raw + ) - identity_private_bytes = base64url_to_bytes(identity_private_key_ed25519) - identity_private = Ed25519PrivateKey.from_private_bytes( - identity_private_bytes - ) + identity_private_bytes = base64url_to_bytes(identity_private_key_ed25519) + identity_private = Ed25519PrivateKey.from_private_bytes( + identity_private_bytes + ) - signature = identity_private.sign(spk_public_bytes) + signature = identity_private.sign(spk_public_bytes) - return ( - bytes_to_base64url(spk_private_bytes), - bytes_to_base64url(spk_public_bytes), - bytes_to_base64url(signature) - ) + return ( + bytes_to_base64url(spk_private_bytes), + bytes_to_base64url(spk_public_bytes), + bytes_to_base64url(signature) + ) ``` **Why sign with Ed25519?** The signature proves that Bob's long-term Ed25519 identity key endorses this particular SPK. When Alice fetches Bob's prekey bundle, she verifies this signature before performing the DH operations. Without the signature, a man-in-the-middle could substitute their own SPK public key into Bob's bundle. Alice would compute a shared secret with the attacker instead of Bob, and the attacker could relay messages between them while reading everything. **What does the signature prove?** It proves exactly one thing: "The entity controlling the Ed25519 private key corresponding to Bob's published Ed25519 public key chose to sign this specific SPK public key." That binds the SPK to Bob's identity. -**Why not self-sign with X25519?** X25519 is a DH function, not a signature scheme. There is no `sign()` method on an X25519 key. You could theoretically convert between Curve25519 and Ed25519 key formats (they share the same underlying curve), but doing that conversion is a footgun. It is error-prone, not all libraries support it, and it violates the principle of key separation. Using a dedicated Ed25519 signing key is simpler and safer. +**Why not self-sign with X25519?** X25519 is a DH function, not a signature scheme. There is no `sign` method on an X25519 key. You could theoretically convert between Curve25519 and Ed25519 key formats (they share the same underlying curve), but doing that conversion is a footgun. It is error-prone, not all libraries support it, and it violates the principle of key separation. Using a dedicated Ed25519 signing key is simpler and safer. ### Step 3: Prekey Bundle Assembly -When Alice wants to message Bob, the server assembles Bob's prekey bundle. This happens in `prekey_service.py:293-361`: +When Alice wants to message Bob, the server assembles Bob's prekey bundle. This happens in `backend/app/services/prekey_service.py`: ```python async def get_prekey_bundle( - self, - session: AsyncSession, - user_id: UUID + self, + session: AsyncSession, + user_id: UUID ) -> PreKeyBundle: - # Fetch identity key - ik_statement = select(IdentityKey).where(IdentityKey.user_id == user_id) - ... - # Fetch active signed prekey - spk_statement = select(SignedPrekey).where( - SignedPrekey.user_id == user_id, - SignedPrekey.is_active - ).order_by(SignedPrekey.created_at.desc()) - ... - if not signed_prekey: - signed_prekey = await self.rotate_signed_prekey(session, user_id) + # Fetch identity key + ik_statement = select(IdentityKey).where(IdentityKey.user_id == user_id) + ... + # Fetch active signed prekey + spk_statement = select(SignedPrekey).where( + SignedPrekey.user_id == user_id, + SignedPrekey.is_active + ).order_by(SignedPrekey.created_at.desc) + ... + if not signed_prekey: + signed_prekey = await self.rotate_signed_prekey(session, user_id) - # Fetch one unused OPK - opk_statement = select(OneTimePrekey).where( - OneTimePrekey.user_id == user_id, - not OneTimePrekey.is_used - ).limit(1) - ... - if one_time_prekey: - one_time_prekey.is_used = True - ... + # Fetch one unused OPK + opk_statement = select(OneTimePrekey).where( + OneTimePrekey.user_id == user_id, + not OneTimePrekey.is_used + ).limit(1) + ... + if one_time_prekey: + one_time_prekey.is_used = True + ... ``` -**Why auto-rotate?** If Bob's signed prekey expired and there are no active ones, the bundle fetch would fail. Auto-rotation at `prekey_service.py:316-321` ensures there is always an active SPK. The rotation generates a new X25519 keypair, signs it with Bob's Ed25519 identity key, marks old SPKs as inactive, and sets a 48-hour expiry (`SIGNED_PREKEY_ROTATION_HOURS = 48` in `config.py:76`). +**Why auto-rotate?** If Bob's signed prekey expired and there are no active ones, the bundle fetch would fail. Auto-rotation at `backend/app/services/prekey_service.py` ensures there is always an active SPK. The rotation generates a new X25519 keypair, signs it with Bob's Ed25519 identity key, marks old SPKs as inactive, and sets a 48-hour expiry (`SIGNED_PREKEY_ROTATION_HOURS = 48` in `config.py`). **Why mark OPK as used?** One-Time Prekeys provide forward secrecy for the initial handshake. Each OPK is consumed exactly once. When Alice fetches Bob's bundle, the server marks that OPK as `is_used = True` and commits the change. If a second person also tries to start a conversation with Bob around the same time, they get a different OPK (or none at all). This single-use guarantee means that even if Bob's long-term keys are later compromised, an attacker cannot retroactively compute the shared secret for conversations that used an OPK, because the OPK private key was consumed and the DH output from it cannot be reconstructed from the other keys alone. -**What happens if no OPK is available?** The protocol still works. The `PreKeyBundle` dataclass has `one_time_prekey: str | None = None`. When the bundle is assembled without an OPK, the X3DH sender side simply computes `dh1 + dh2 + dh3` instead of `dh1 + dh2 + dh3 + dh4`. This gives slightly weaker forward secrecy for the initial handshake (it relies on the signed prekey not being compromised), but the Double Ratchet provides forward secrecy for all subsequent messages regardless. The system compensates by calling `replenish_one_time_prekeys` periodically to maintain a pool of 100 OPKs (`DEFAULT_ONE_TIME_PREKEY_COUNT = 100` in `config.py:75`). +**What happens if no OPK is available?** The protocol still works. The `PreKeyBundle` dataclass has `one_time_prekey: str | None = None`. When the bundle is assembled without an OPK, the X3DH sender side simply computes `dh1 + dh2 + dh3` instead of `dh1 + dh2 + dh3 + dh4`. This gives slightly weaker forward secrecy for the initial handshake (it relies on the signed prekey not being compromised), but the Double Ratchet provides forward secrecy for all subsequent messages regardless. The system compensates by calling `replenish_one_time_prekeys` periodically to maintain a pool of 100 OPKs (`DEFAULT_ONE_TIME_PREKEY_COUNT = 100` in `config.py`). ### Step 4: X3DH Sender Side -This is the core of the initial key agreement. Here is the DH operations from `x3dh_manager.py:241-264`: +This is the core of the initial key agreement. Here is the DH operations from `frontend/src/crypto/x3dh.ts`: ```python dh1 = alice_ik_private.exchange(bob_spk_public) @@ -234,25 +230,25 @@ dh3 = alice_ek_private.exchange(bob_spk_public) used_one_time_prekey = False if bob_bundle.one_time_prekey: - bob_opk_public_bytes = base64url_to_bytes(bob_bundle.one_time_prekey) - bob_opk_public = X25519PublicKey.from_public_bytes(bob_opk_public_bytes) - dh4 = alice_ek_private.exchange(bob_opk_public) - key_material = dh1 + dh2 + dh3 + dh4 - used_one_time_prekey = True + bob_opk_public_bytes = base64url_to_bytes(bob_bundle.one_time_prekey) + bob_opk_public = X25519PublicKey.from_public_bytes(bob_opk_public_bytes) + dh4 = alice_ek_private.exchange(bob_opk_public) + key_material = dh1 + dh2 + dh3 + dh4 + used_one_time_prekey = True else: - key_material = dh1 + dh2 + dh3 + key_material = dh1 + dh2 + dh3 f = b'\xff' * X25519_KEY_SIZE hkdf = HKDF( - algorithm = hashes.SHA256(), - length = X25519_KEY_SIZE, - salt = b'\x00' * X25519_KEY_SIZE, - info = b'X3DH', + algorithm = hashes.SHA256, + length = X25519_KEY_SIZE, + salt = b'\x00' * X25519_KEY_SIZE, + info = b'X3DH', ) shared_key = hkdf.derive(f + key_material) ``` -Before this code runs, the sender verifies the signed prekey signature at `x3dh_manager.py:217-220`. If verification fails, a `ValueError` is raised and the exchange aborts. This is the MITM protection. +Before this code runs, the sender verifies the signed prekey signature at `frontend/src/crypto/x3dh.ts`. If verification fails, a `ValueError` is raised and the exchange aborts. This is the MITM protection. **Each DH operation explained:** @@ -267,22 +263,22 @@ Before this code runs, the sender verifies the signed prekey signature at `x3dh_ **Why info="X3DH"?** The `info` parameter in HKDF is a domain separation string. It ensures that the derived key is bound to this specific protocol. If the same key material were accidentally reused in a different context (say, a TLS handshake), the info string would produce a different derived key. This prevents cross-protocol attacks. -**Associated data: IK_A || IK_B.** After deriving the shared key, the code computes `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` at `x3dh_manager.py:272`. This associated data is later passed to AES-GCM as the AAD (Additional Authenticated Data). It binds the encrypted messages to the specific pair of identity keys that performed the handshake. If an attacker tries to redirect messages between different users, the AAD check will fail during decryption. +**Associated data: IK_A || IK_B.** After deriving the shared key, the code computes `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` at `frontend/src/crypto/x3dh.ts`. This associated data is later passed to AES-GCM as the AAD (Additional Authenticated Data). It binds the encrypted messages to the specific pair of identity keys that performed the handshake. If an attacker tries to redirect messages between different users, the AAD check will fail during decryption. ### Step 5: X3DH Receiver Side -The receiver-side logic lives at `x3dh_manager.py:283-350`. The operations are mirrored: +The receiver-side logic lives at `frontend/src/crypto/x3dh.ts`. The operations are mirrored: ```python -dh1 = bob_spk_private.exchange(alice_ik_public) # mirrors alice_ik.exchange(bob_spk) -dh2 = bob_ik_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_ik) -dh3 = bob_spk_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_spk) -dh4 = bob_opk_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_opk) +dh1 = bob_spk_private.exchange(alice_ik_public) # mirrors alice_ik.exchange(bob_spk) +dh2 = bob_ik_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_ik) +dh3 = bob_spk_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_spk) +dh4 = bob_opk_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_opk) ``` This works because of how Diffie-Hellman is defined: `A_private.exchange(B_public) == B_private.exchange(A_public)`. The math is commutative. Both sides compute the same four 32-byte DH outputs, concatenate them in the same order, and derive the same shared key through the same HKDF parameters (zero salt, `X3DH` info, `0xFF` padding). -The receiver also constructs `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` in the same order (Alice's IK first, Bob's IK second) at `x3dh_manager.py:341`. Order matters. If Alice put hers first on the sender side but Bob reversed the order on the receiver side, the associated data would not match and decryption would fail. +The receiver also constructs `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` in the same order (Alice's IK first, Bob's IK second) at `frontend/src/crypto/x3dh.ts`. Order matters. If Alice put hers first on the sender side but Bob reversed the order on the receiver side, the associated data would not match and decryption would fail. ### Common Mistakes @@ -291,7 +287,7 @@ The receiver also constructs `associated_data = alice_ik_public_bytes + bob_ik_p ```python # WRONG - breaks compatibility between sender and receiver salt = os.urandom(32) -hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=salt, info=b'X3DH') +hkdf = HKDF(algorithm=hashes.SHA256, length=32, salt=salt, info=b'X3DH') ``` If you use a random salt, the sender and receiver will derive different shared keys. The sender would need to transmit the salt alongside the ephemeral key, and at that point you have added complexity for no security gain since the DH outputs already provide the randomness. @@ -301,12 +297,12 @@ If you use a random salt, the sender and receiver will derive different shared k ```python # WRONG - allows MITM to substitute their own SPK def perform_x3dh_sender(self, alice_identity_private_x25519, bob_bundle, ...): - # Skipped: self.verify_signed_prekey(...) - alice_ik_private_bytes = base64url_to_bytes(alice_identity_private_x25519) - ... + # Skipped: self.verify_signed_prekey(...) + alice_ik_private_bytes = base64url_to_bytes(alice_identity_private_x25519) + ... ``` -Without signature verification, an attacker who controls the network can replace Bob's SPK public key with their own. Alice would compute a shared secret with the attacker. The attacker then performs X3DH with Bob using Bob's real SPK, and relays messages between them. Neither Alice nor Bob detects the interception. The fix is on `x3dh_manager.py:217-220`: verify before exchanging. +Without signature verification, an attacker who controls the network can replace Bob's SPK public key with their own. Alice would compute a shared secret with the attacker. The attacker then performs X3DH with Bob using Bob's real SPK, and relays messages between them. Neither Alice nor Bob detects the interception. The fix is on `frontend/src/crypto/x3dh.ts`: verify before exchanging. **BAD: Reusing one-time prekeys** @@ -317,53 +313,53 @@ opk = get_opk_for_user(user_id) # another conversation uses the same OPK ``` -If two conversations use the same OPK, an attacker who later compromises Bob's OPK private key can compute the DH4 output for both conversations. That defeats the purpose of single-use keys. The fix is at `prekey_service.py:332`: `one_time_prekey.is_used = True` immediately upon retrieval, followed by `session.commit()`. +If two conversations use the same OPK, an attacker who later compromises Bob's OPK private key can compute the DH4 output for both conversations. That defeats the purpose of single-use keys. The fix is at `backend/app/services/prekey_service.py`: `one_time_prekey.is_used = True` immediately upon retrieval, followed by `session.commit`. ### Frontend X3DH Implementation -The frontend mirrors the backend X3DH in TypeScript using the WebCrypto API. The client-side sender logic lives in `x3dh.ts:115-173`: +The frontend mirrors the backend X3DH in TypeScript using the WebCrypto API. The client-side sender logic lives in `x3dh.ts`: ```typescript export async function initiateX3DH( - identityKeyPair: IdentityKeyPair, - recipientBundle: PreKeyBundle + identityKeyPair: IdentityKeyPair, + recipientBundle: PreKeyBundle ): Promise { - const signatureValid = await verifySignedPreKey( - recipientBundle.identity_key_ed25519, - recipientBundle.signed_prekey, - recipientBundle.signed_prekey_signature - ) - if (!signatureValid) { - throw new Error("Invalid signed prekey signature") - } - ... - const dh1 = await x25519DeriveSharedSecret(senderIdentityPrivate, recipientSignedPreKeyPublic) - const dh2 = await x25519DeriveSharedSecret(ephemeralKeyPair.privateKey, recipientIdentityPublic) - const dh3 = await x25519DeriveSharedSecret(ephemeralKeyPair.privateKey, recipientSignedPreKeyPublic) - ... - const sharedKey = await hkdfDerive(concatenated, EMPTY_SALT, X3DH_INFO, 32) + const signatureValid = await verifySignedPreKey( + recipientBundle.identity_key_ed25519, + recipientBundle.signed_prekey, + recipientBundle.signed_prekey_signature + ) + if (!signatureValid) { + throw new Error("Invalid signed prekey signature") + } + ... + const dh1 = await x25519DeriveSharedSecret(senderIdentityPrivate, recipientSignedPreKeyPublic) + const dh2 = await x25519DeriveSharedSecret(ephemeralKeyPair.privateKey, recipientIdentityPublic) + const dh3 = await x25519DeriveSharedSecret(ephemeralKeyPair.privateKey, recipientSignedPreKeyPublic) + ... + const sharedKey = await hkdfDerive(concatenated, EMPTY_SALT, X3DH_INFO, 32) ``` -The HKDF derivation used here is from `primitives.ts:166-192`: +The HKDF derivation used here is from `frontend/src/crypto/primitives.ts`: ```typescript export async function hkdfDerive( - inputKeyMaterial: Uint8Array, - salt: Uint8Array, - info: Uint8Array, - outputLength: number = HKDF_OUTPUT_SIZE + inputKeyMaterial: Uint8Array, + salt: Uint8Array, + info: Uint8Array, + outputLength: number = HKDF_OUTPUT_SIZE ): Promise { - const baseKey = await subtle.importKey( - "raw", inputKeyMaterial.buffer as ArrayBuffer, - { name: "HKDF" }, false, ["deriveBits"] - ) - const derivedBits = await subtle.deriveBits( - { name: "HKDF", hash: "SHA-256", - salt: salt.buffer as ArrayBuffer, - info: info.buffer as ArrayBuffer }, - baseKey, outputLength * 8 - ) - return new Uint8Array(derivedBits) + const baseKey = await subtle.importKey( + "raw", inputKeyMaterial.buffer as ArrayBuffer, + { name: "HKDF" }, false, ["deriveBits"] + ) + const derivedBits = await subtle.deriveBits( + { name: "HKDF", hash: "SHA-256", + salt: salt.buffer as ArrayBuffer, + info: info.buffer as ArrayBuffer }, + baseKey, outputLength * 8 + ) + return new Uint8Array(derivedBits) } ``` @@ -371,7 +367,7 @@ There is an important difference between the backend and frontend HKDF calls. Th The WebCrypto API's `subtle.deriveBits` with the `HKDF` algorithm handles the extract-then-expand steps internally. You provide the input key material, a salt, an info string, and the desired output length in bits (not bytes, which is why `outputLength * 8` appears). The `importKey` call with `{ name: "HKDF" }` creates a non-extractable key object suitable only for derivation, which prevents the raw key material from being accidentally exported or misused. -The receiver-side X3DH in `x3dh.ts:175-215` mirrors the sender with the same key swaps as the Python version. The commutative property of ECDH ensures both sides derive the same 32-byte shared key. +The receiver-side X3DH in `x3dh.ts` mirrors the sender with the same key swaps as the Python version. The commutative property of ECDH ensures both sides derive the same 32-byte shared key. --- @@ -381,130 +377,132 @@ The Double Ratchet provides ongoing forward secrecy and break-in recovery after ### Initialization -The sender initializes at `double_ratchet.py:279-302`: +The sender initializes via `initializeRatchetSender` in `frontend/src/crypto/double-ratchet.ts`: -```python -def initialize_sender( - self, - shared_key: bytes, - peer_public_key: bytes -) -> DoubleRatchetState: - dh_private = X25519PrivateKey.generate() - peer_public = X25519PublicKey.from_public_bytes(peer_public_key) - dh_output = dh_private.exchange(peer_public) +```ts +export async function initializeRatchetSender( + peerId: string, + sharedKey: Uint8Array, + peerPublicKey: Uint8Array, +): Promise { + const dhKeyPair = await generateX25519KeyPair + const dhPublicKey = await exportPublicKey(dhKeyPair.publicKey) + const peerKey = await importX25519PublicKey(peerPublicKey) + const dhOutput = await x25519DeriveSharedSecret(dhKeyPair.privateKey, peerKey) - root_key, sending_chain_key = self._kdf_rk(shared_key, dh_output) + // KDF_RK per spec: HKDF(salt = previous_root_key, IKM = dh_output, info = "DoubleRatchet") + const derivedKeys = await hkdfDerive(dhOutput, sharedKey, RATCHET_INFO, 64) + const rootKey = derivedKeys.slice(0, 32) + const sendingChainKey = derivedKeys.slice(32, 64) - state = DoubleRatchetState( - root_key = root_key, - sending_chain_key = sending_chain_key, - receiving_chain_key = b'\x00' * HKDF_OUTPUT_SIZE, - dh_private_key = dh_private, - dh_peer_public_key = peer_public_key - ) - return state + return { peer_id: peerId, root_key: rootKey, sending_chain_key: sendingChainKey, /* ... */ } +} ``` -The receiver initializes at `double_ratchet.py:304-321`: +The receiver initializes via `initializeRatchetReceiver`: -```python -def initialize_receiver( - self, - shared_key: bytes, - own_private_key: X25519PrivateKey -) -> DoubleRatchetState: - state = DoubleRatchetState( - root_key = shared_key, - sending_chain_key = b'\x00' * HKDF_OUTPUT_SIZE, - receiving_chain_key = b'\x00' * HKDF_OUTPUT_SIZE, - dh_private_key = own_private_key, - dh_peer_public_key = None - ) - return state +```ts +export async function initializeRatchetReceiver( + peerId: string, + sharedKey: Uint8Array, + dhKeyPair: CryptoKeyPair, +): Promise { + const dhPublicKey = await exportPublicKey(dhKeyPair.publicKey) + return { + peer_id: peerId, + root_key: sharedKey, + sending_chain_key: new Uint8Array(0), + receiving_chain_key: null, + dh_private_key: dhKeyPair, + dh_public_key: dhPublicKey, + dh_peer_public_key: null, + /* ... */ + } +} ``` **Why does the sender do an extra DH step?** The sender generates a fresh DH keypair and immediately performs a DH exchange with Bob's SPK public key. This produces a `dh_output` which gets fed into `_kdf_rk` along with the X3DH `shared_key` to derive the first `root_key` and `sending_chain_key`. This extra step means the first message already has a DH ratchet contribution beyond the X3DH output. The receiver, by contrast, does not perform this step during initialization. It simply stores the X3DH shared key as the root key and waits for the first message, which will carry the sender's new DH public key and trigger the receiver's first ratchet step. ### KDF Chain Operations -The root key chain derivation at `double_ratchet.py:79-94`: +The root key chain derivation at `frontend/src/crypto/double-ratchet.ts`: ```python def _kdf_rk(self, root_key: bytes, dh_output: bytes) -> tuple[bytes, bytes]: - hkdf = HKDF( - algorithm = hashes.SHA256(), - length = HKDF_OUTPUT_SIZE * 2, - salt = root_key, - info = b'', - ) - output = hkdf.derive(dh_output) - new_root_key = output[: HKDF_OUTPUT_SIZE] - new_chain_key = output[HKDF_OUTPUT_SIZE :] - return new_root_key, new_chain_key + hkdf = HKDF( + algorithm = hashes.SHA256, + length = HKDF_OUTPUT_SIZE * 2, + salt = root_key, + info = b'', + ) + output = hkdf.derive(dh_output) + new_root_key = output[: HKDF_OUTPUT_SIZE] + new_chain_key = output[HKDF_OUTPUT_SIZE :] + return new_root_key, new_chain_key ``` `_kdf_rk` is the root chain KDF. It runs every time a DH ratchet step occurs (when the sender or receiver generates a new DH keypair). The `root_key` is used as the HKDF salt, and the fresh DH output is the input key material. HKDF produces 64 bytes (`HKDF_OUTPUT_SIZE * 2 = 64`), which is split in half: the first 32 bytes become the new root key, and the second 32 bytes become the new chain key. The root key never directly encrypts anything. It acts as a "master key" that seeds each new chain. Because the root key is updated with fresh DH output every ratchet step, even if an attacker compromises the current root key, the next ratchet step will incorporate a new DH output that the attacker does not know (assuming the DH private key is secure), and the root key will be "healed." This is the break-in recovery property. The empty `info` parameter is intentional. The Signal spec does not use an info string for the root chain KDF. Domain separation happens implicitly because the root chain HKDF always uses the root key as salt, whereas the X3DH HKDF uses a zero salt. Different salts produce different outputs even with the same input, so there is no cross-contamination. -The chain key derivation at `double_ratchet.py:96-109`: +The chain key derivation at `frontend/src/crypto/double-ratchet.ts`: ```python def _kdf_ck(self, chain_key: bytes) -> tuple[bytes, bytes]: - h_chain = hmac.HMAC(chain_key, hashes.SHA256()) - h_chain.update(b'\x01') - next_chain_key = h_chain.finalize() + h_chain = hmac.HMAC(chain_key, hashes.SHA256) + h_chain.update(b'\x01') + next_chain_key = h_chain.finalize - h_message = hmac.HMAC(chain_key, hashes.SHA256()) - h_message.update(b'\x02') - message_key = h_message.finalize() + h_message = hmac.HMAC(chain_key, hashes.SHA256) + h_message.update(b'\x02') + message_key = h_message.finalize - return next_chain_key, message_key + return next_chain_key, message_key ``` **Why HMAC with 0x01 and 0x02?** The chain key needs to produce two outputs: the next chain key and the message key. Using HMAC with different single-byte constants is the simplest way to derive two independent values from one input. `0x01` produces the next chain key, `0x02` produces the message key. The constants are arbitrary but fixed by convention (the Signal spec uses these exact values). The important thing is that they are different, so the two HMAC outputs are cryptographically independent. -**Why two separate HMACs?** You cannot reuse an HMAC object after calling `finalize()` in the Python `cryptography` library. Each HMAC computation is a fresh instance. Even if you could reuse it, producing two different outputs from the same key requires two different inputs, which means two operations. +**Why two separate HMACs?** You cannot reuse an HMAC object after calling `finalize` in the Python `cryptography` library. Each HMAC computation is a fresh instance. Even if you could reuse it, producing two different outputs from the same key requires two different inputs, which means two operations. **What is the one-way property?** Given `next_chain_key = HMAC(chain_key, 0x01)`, you cannot compute `chain_key` from `next_chain_key`. HMAC is a one-way function. This means that if an attacker obtains a message key for message N, they cannot derive the chain key that produced it, and therefore cannot derive message keys for messages 0 through N-1. This is forward secrecy within a single chain. ### Message Encryption -Here is the encrypt path from `double_ratchet.py:323-362`: +Here is the encrypt path from `frontend/src/crypto/double-ratchet.ts`: ```python def encrypt_message(self, state, plaintext, associated_data): - state.sending_chain_key, message_key = self._kdf_ck(state.sending_chain_key) - nonce, ciphertext = self._encrypt_with_message_key(message_key, plaintext, associated_data) + state.sending_chain_key, message_key = self._kdf_ck(state.sending_chain_key) + nonce, ciphertext = self._encrypt_with_message_key(message_key, plaintext, associated_data) - if state.dh_private_key: - dh_public = state.dh_private_key.public_key() - dh_public_bytes = dh_public.public_bytes( - encoding = serialization.Encoding.Raw, - format = serialization.PublicFormat.Raw - ) - else: - dh_public_bytes = b'\x00' * X25519_KEY_SIZE + if state.dh_private_key: + dh_public = state.dh_private_key.public_key + dh_public_bytes = dh_public.public_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PublicFormat.Raw + ) + else: + dh_public_bytes = b'\x00' * X25519_KEY_SIZE - encrypted_msg = EncryptedMessage( - ciphertext = ciphertext, - nonce = nonce, - dh_public_key = dh_public_bytes, - message_number = state.sending_message_number, - previous_chain_length = state.previous_sending_chain_length - ) - state.sending_message_number += 1 - return encrypted_msg + encrypted_msg = EncryptedMessage( + ciphertext = ciphertext, + nonce = nonce, + dh_public_key = dh_public_bytes, + message_number = state.sending_message_number, + previous_chain_length = state.previous_sending_chain_length + ) + state.sending_message_number += 1 + return encrypted_msg ``` -And the underlying AES-GCM encryption at `double_ratchet.py:111-130`: +And the underlying AES-GCM encryption at `frontend/src/crypto/double-ratchet.ts`: ```python def _encrypt_with_message_key(self, message_key, plaintext, associated_data): - aesgcm = AESGCM(message_key) - nonce = os.urandom(AES_GCM_NONCE_SIZE) - ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data) - return nonce, ciphertext + aesgcm = AESGCM(message_key) + nonce = os.urandom(AES_GCM_NONCE_SIZE) + ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data) + return nonce, ciphertext ``` **Why include dh_public_key in the header?** The recipient needs the sender's current DH public key to determine whether to perform a DH ratchet step. If the DH public key in the message header differs from the last known peer DH key, the recipient knows the sender has ratcheted, and the recipient must also ratchet to derive the correct receiving chain key. Without this header field, the recipient would have no way to know which chain to use. @@ -513,41 +511,41 @@ def _encrypt_with_message_key(self, message_key, plaintext, associated_data): ### Message Decryption -The decrypt path at `double_ratchet.py:364-416` handles three distinct cases: +The decrypt path at `frontend/src/crypto/double-ratchet.ts` handles three distinct cases: ```python def decrypt_message(self, state, encrypted_msg, associated_data): - # Case 1: Check if we already stored this message's key (out-of-order) - skipped_key = self._try_skipped_message_key( - state, encrypted_msg.dh_public_key, encrypted_msg.message_number - ) - if skipped_key: - return self._decrypt_with_message_key( - skipped_key, encrypted_msg.nonce, encrypted_msg.ciphertext, associated_data - ) + # Case 1: Check if we already stored this message's key (out-of-order) + skipped_key = self._try_skipped_message_key( + state, encrypted_msg.dh_public_key, encrypted_msg.message_number + ) + if skipped_key: + return self._decrypt_with_message_key( + skipped_key, encrypted_msg.nonce, encrypted_msg.ciphertext, associated_data + ) - # Case 2: New DH ratchet key means sender ratcheted - if encrypted_msg.dh_public_key != state.dh_peer_public_key: - if state.dh_peer_public_key: - self._store_skipped_message_keys( - state, encrypted_msg.previous_chain_length, state.dh_peer_public_key - ) - self._dh_ratchet_receive(state, encrypted_msg.dh_public_key) + # Case 2: New DH ratchet key means sender ratcheted + if encrypted_msg.dh_public_key != state.dh_peer_public_key: + if state.dh_peer_public_key: + self._store_skipped_message_keys( + state, encrypted_msg.previous_chain_length, state.dh_peer_public_key + ) + self._dh_ratchet_receive(state, encrypted_msg.dh_public_key) - # Case 3: Message number ahead of current position means skipped messages - if encrypted_msg.message_number > state.receiving_message_number: - self._store_skipped_message_keys( - state, encrypted_msg.message_number, encrypted_msg.dh_public_key - ) + # Case 3: Message number ahead of current position means skipped messages + if encrypted_msg.message_number > state.receiving_message_number: + self._store_skipped_message_keys( + state, encrypted_msg.message_number, encrypted_msg.dh_public_key + ) - # Advance chain and decrypt - state.receiving_chain_key, message_key = self._kdf_ck(state.receiving_chain_key) - state.receiving_message_number += 1 + # Advance chain and decrypt + state.receiving_chain_key, message_key = self._kdf_ck(state.receiving_chain_key) + state.receiving_message_number += 1 - plaintext = self._decrypt_with_message_key( - message_key, encrypted_msg.nonce, encrypted_msg.ciphertext, associated_data - ) - return plaintext + plaintext = self._decrypt_with_message_key( + message_key, encrypted_msg.nonce, encrypted_msg.ciphertext, associated_data + ) + return plaintext ``` **Case 1: Skipped key hit.** The message key was already computed and stored during a previous skip operation. Pop it from the dict, decrypt, done. This handles previously skipped messages arriving late. @@ -558,32 +556,32 @@ def decrypt_message(self, state, encrypted_msg, associated_data): ### Out-of-Order Handling -The skip mechanism at `double_ratchet.py:215-244`: +The skip mechanism at `frontend/src/crypto/double-ratchet.ts`: ```python def _store_skipped_message_keys(self, state, until_message_number, dh_public_key): - num_to_skip = until_message_number - state.receiving_message_number + num_to_skip = until_message_number - state.receiving_message_number - if num_to_skip > self.max_skip: - raise ValueError( - f"Cannot skip {num_to_skip} messages " - f"(MAX_SKIP={self.max_skip})" - ) + if num_to_skip > self.max_skip: + raise ValueError( + f"Cannot skip {num_to_skip} messages " + f"(MAX_SKIP={self.max_skip})" + ) - if len(state.skipped_message_keys) + num_to_skip > self.max_cache: - self._evict_oldest_skipped_keys(state, num_to_skip) + if len(state.skipped_message_keys) + num_to_skip > self.max_cache: + self._evict_oldest_skipped_keys(state, num_to_skip) - chain_key = state.receiving_chain_key - for msg_num in range(state.receiving_message_number, until_message_number): - chain_key, message_key = self._kdf_ck(chain_key) - state.skipped_message_keys[(dh_public_key, msg_num)] = message_key + chain_key = state.receiving_chain_key + for msg_num in range(state.receiving_message_number, until_message_number): + chain_key, message_key = self._kdf_ck(chain_key) + state.skipped_message_keys[(dh_public_key, msg_num)] = message_key - state.receiving_chain_key = chain_key + state.receiving_chain_key = chain_key ``` -**MAX_SKIP = 1000** (`config.py:73`). If an attacker sends a message claiming `message_number = 999999999`, the receiver would need to compute 999999999 HMAC operations to derive all the intermediate keys. That is a denial-of-service attack. The limit of 1000 means the receiver will reject any message that would require skipping more than 1000 positions. In practice, even aggressive out-of-order delivery rarely exceeds a few dozen skipped messages. +**MAX_SKIP = 1000** (`config.py`). If an attacker sends a message claiming `message_number = 999999999`, the receiver would need to compute 999999999 HMAC operations to derive all the intermediate keys. That is a denial-of-service attack. The limit of 1000 means the receiver will reject any message that would require skipping more than 1000 positions. In practice, even aggressive out-of-order delivery rarely exceeds a few dozen skipped messages. -**MAX_CACHE = 2000** (`config.py:74`). Even with the skip limit, a determined attacker could slowly accumulate skipped keys by sending many small gaps. The cache limit prevents unbounded memory growth. When the cache is full, `_evict_oldest_skipped_keys` removes the oldest entries to make room. Oldest entries are the ones most likely to be genuinely lost (if a message has not arrived after 2000 other skipped keys were stored, it is probably never coming). +**MAX_CACHE = 2000** (`config.py`). Even with the skip limit, a determined attacker could slowly accumulate skipped keys by sending many small gaps. The cache limit prevents unbounded memory growth. When the cache is full, `_evict_oldest_skipped_keys` removes the oldest entries to make room. Oldest entries are the ones most likely to be genuinely lost (if a message has not arrived after 2000 other skipped keys were stored, it is probably never coming). ### Common Mistakes @@ -591,7 +589,7 @@ def _store_skipped_message_keys(self, state, until_message_number, dh_public_key **Not evicting old skipped keys (memory exhaustion).** Without eviction, the `skipped_message_keys` dict grows without bound. An attacker can cause OOM by sending many messages with incrementing but non-sequential message numbers across multiple ratchet steps. -**Reusing nonces (catastrophic for AES-GCM).** AES-GCM uses a 12-byte nonce. If the same nonce is used twice with the same key, an attacker can XOR the two ciphertexts to eliminate the keystream and recover both plaintexts. The code uses `os.urandom(AES_GCM_NONCE_SIZE)` at `double_ratchet.py:122` for a fresh random nonce every time. Since each message key is used exactly once (each chain key advances after derivation), and each message key gets a random nonce, the probability of a collision is negligible. But if someone modified the code to reuse message keys, the random nonce would be the only defense, and with 12 bytes, birthday collisions become likely after about 2^48 messages under the same key. +**Reusing nonces (catastrophic for AES-GCM).** AES-GCM uses a 12-byte nonce. If the same nonce is used twice with the same key, an attacker can XOR the two ciphertexts to eliminate the keystream and recover both plaintexts. The code uses `os.urandom(AES_GCM_NONCE_SIZE)` at `frontend/src/crypto/double-ratchet.ts` for a fresh random nonce every time. Since each message key is used exactly once (each chain key advances after derivation), and each message key gets a random nonce, the probability of a collision is negligible. But if someone modified the code to reuse message keys, the random nonce would be the only defense, and with 12 bytes, birthday collisions become likely after about 2^48 messages under the same key. --- @@ -599,58 +597,58 @@ def _store_skipped_message_keys(self, state, until_message_number, dh_public_key ### Registration Flow -Registration is handled in `passkey_manager.py:55-94`. The server generates registration options: +Registration is handled in `backend/app/core/passkey/passkey_manager.py`. The server generates registration options: ```python def generate_registration_options( - self, user_id, username, display_name, exclude_credentials + self, user_id, username, display_name, exclude_credentials ): - challenge = secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES) - ... - options = generate_registration_options( - rp_id = self.rp_id, - rp_name = self.rp_name, - user_id = user_id, - user_name = username, - challenge = challenge, - attestation = AttestationConveyancePreference.NONE, - authenticator_selection = AuthenticatorSelectionCriteria( - resident_key = ResidentKeyRequirement.REQUIRED, - user_verification = UserVerificationRequirement.PREFERRED, - ), - exclude_credentials = exclude_creds, - ) + challenge = secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES) + ... + options = generate_registration_options( + rp_id = self.rp_id, + rp_name = self.rp_name, + user_id = user_id, + user_name = username, + challenge = challenge, + attestation = AttestationConveyancePreference.NONE, + authenticator_selection = AuthenticatorSelectionCriteria( + resident_key = ResidentKeyRequirement.REQUIRED, + user_verification = UserVerificationRequirement.PREFERRED, + ), + exclude_credentials = exclude_creds, + ) ``` **ResidentKeyRequirement.REQUIRED** means the authenticator must store the credential on the device itself (a "discoverable credential" or "passkey"). This allows usernameless login: the user just taps their security key or uses biometrics, and the authenticator presents the credential without the user typing a username. Without REQUIRED, the server would need to provide a credential ID hint during authentication, which means the user must provide their username first. -**Challenge generation** uses `secrets.token_bytes(32)` at `passkey_manager.py:65`. The `secrets` module is Python's CSPRNG interface, equivalent to `/dev/urandom`. The challenge must be unpredictable to prevent replay attacks. The challenge is stored in Redis with a TTL of 600 seconds (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600` in `config.py:84`), so it expires before an attacker could realistically capture and replay it. +**Challenge generation** uses `secrets.token_bytes(32)` at `backend/app/core/passkey/passkey_manager.py`. The `secrets` module is Python's CSPRNG interface, equivalent to `/dev/urandom`. The challenge must be unpredictable to prevent replay attacks. The challenge is stored in Redis with a TTL of 600 seconds (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600` in `config.py`), so it expires before an attacker could realistically capture and replay it. -**Attestation verification** at `passkey_manager.py:96-130` calls `verify_registration_response` from py_webauthn. This checks that the authenticator's response was correctly signed, that the RP ID matches, that the origin matches, and that the challenge matches. The `expected_origin = self.rp_origin` check at `passkey_manager.py:109` is critical: it ensures the registration happened from the legitimate frontend, not a phishing page on a different domain. +**Attestation verification** at `backend/app/core/passkey/passkey_manager.py` calls `verify_registration_response` from py_webauthn. This checks that the authenticator's response was correctly signed, that the RP ID matches, that the origin matches, and that the challenge matches. The `expected_origin = self.rp_origin` check at `backend/app/core/passkey/passkey_manager.py` is critical: it ensures the registration happened from the legitimate frontend, not a phishing page on a different domain. ### Authentication Flow -Authentication follows the same challenge-response pattern. The server generates options at `passkey_manager.py:132-160`, sends the challenge to the client, the client's authenticator signs the challenge with its private key, and the server verifies at `passkey_manager.py:162-207`. +Authentication follows the same challenge-response pattern. The server generates options at `backend/app/core/passkey/passkey_manager.py`, sends the challenge to the client, the client's authenticator signs the challenge with its private key, and the server verifies at `backend/app/core/passkey/passkey_manager.py`. **Why challenge-response?** Without a fresh challenge, an attacker who intercepts one authentication response could replay it forever. The challenge ensures each authentication is unique. The authenticator signs the challenge (along with other data like the RP ID and origin) with its private key. The server verifies the signature using the stored public key. Even if the signed response is intercepted, it cannot be reused because the next authentication will have a different challenge. -**Why check origin?** The origin check (`expected_origin = self.rp_origin`) at `passkey_manager.py:177` prevents credential phishing. If an attacker sets up a fake site at `evil-chat.com` and tricks the user into authenticating, the authenticator will include `evil-chat.com` as the origin in the signed data. When the attacker tries to forward that signed response to the real server, the origin check fails because the server expects `http://localhost:3000` (or whatever the production origin is). The authenticator binds the signature to the origin, making phishing structurally impossible. +**Why check origin?** The origin check (`expected_origin = self.rp_origin`) at `backend/app/core/passkey/passkey_manager.py` prevents credential phishing. If an attacker sets up a fake site at `evil-chat.com` and tricks the user into authenticating, the authenticator will include `evil-chat.com` as the origin in the signed data. When the attacker tries to forward that signed response to the real server, the origin check fails because the server expects `http://localhost:3000` (or whatever the production origin is). The authenticator binds the signature to the origin, making phishing structurally impossible. ### Clone Detection -From `passkey_manager.py:184-193`: +From `backend/app/core/passkey/passkey_manager.py`: ```python if (credential_current_sign_count != 0 and new_sign_count != 0 - and new_sign_count <= credential_current_sign_count): - logger.error( - "Signature counter did not increase: current=%s, new=%s. Possible cloned authenticator detected!", - credential_current_sign_count, - new_sign_count - ) - raise ValueError( - "Signature counter anomaly detected - potential cloned authenticator" - ) + and new_sign_count <= credential_current_sign_count): + logger.error( + "Signature counter did not increase: current=%s, new=%s. Possible cloned authenticator detected!", + credential_current_sign_count, + new_sign_count + ) + raise ValueError( + "Signature counter anomaly detected - potential cloned authenticator" + ) ``` **What cloning means.** If someone physically copies a hardware security key (or extracts the private key from a software authenticator), both the original and the clone have the same credential. Each has its own signature counter starting from the same value. When the original authenticates, its counter increments to N+1. When the clone authenticates, its counter also increments, but from N, producing N+1 (or a different value depending on when each was used). If the server sees a counter that is less than or equal to the last recorded counter, it knows two devices are using the same credential. @@ -665,68 +663,68 @@ if (credential_current_sign_count != 0 and new_sign_count != 0 ### Connection Management -From `websocket_manager.py:43-95`: +From `websocket_manager.py`: ```python async def connect(self, websocket, user_id): - await websocket.accept() - if user_id not in self.active_connections: - self.active_connections[user_id] = [] - if len(self.active_connections[user_id]) >= WS_MAX_CONNECTIONS_PER_USER: - await self._send_error(websocket, "max_connections", ...) - await websocket.close() - return False - self.active_connections[user_id].append(websocket) - await presence_service.set_user_online(user_id) - self.heartbeat_tasks[user_id] = asyncio.create_task(self._heartbeat_loop(websocket, user_id)) - await self._subscribe_to_messages(user_id) - return True + await websocket.accept + if user_id not in self.active_connections: + self.active_connections[user_id] = [] + if len(self.active_connections[user_id]) >= WS_MAX_CONNECTIONS_PER_USER: + await self._send_error(websocket, "max_connections", ...) + await websocket.close + return False + self.active_connections[user_id].append(websocket) + await presence_service.set_user_online(user_id) + self.heartbeat_tasks[user_id] = asyncio.create_task(self._heartbeat_loop(websocket, user_id)) + await self._subscribe_to_messages(user_id) + return True ``` **Why per-user connection lists?** A user might have the chat open on their phone and their laptop simultaneously. Each device creates its own WebSocket connection. The `active_connections` dict maps `UUID -> list[WebSocket]`, so when a message arrives for a user, it gets sent to all of their active connections. Without this, multi-device support would not work. -**Why max 5?** `WS_MAX_CONNECTIONS_PER_USER = 5` (from `config.py:142`). Without a limit, a malicious client could open thousands of WebSocket connections and exhaust server memory. Five is generous enough for legitimate multi-device usage (phone, tablet, laptop, desktop, maybe one more) while capping resource consumption. The specific number is configurable via environment variable. +**Why max 5?** `WS_MAX_CONNECTIONS_PER_USER = 5` (from `config.py`). Without a limit, a malicious client could open thousands of WebSocket connections and exhaust server memory. Five is generous enough for legitimate multi-device usage (phone, tablet, laptop, desktop, maybe one more) while capping resource consumption. The specific number is configurable via environment variable. -**Why heartbeat?** WebSocket connections can silently die. A user closes their laptop lid, their WiFi drops, or a NAT gateway times out the connection. The TCP stack might not detect the dead connection for minutes or hours. The heartbeat loop at `websocket_manager.py:177-201` sends a ping every `WS_HEARTBEAT_INTERVAL` (30 seconds by default). If the send fails, it means the connection is dead, and `disconnect` cleans it up. This keeps the `active_connections` dict accurate and ensures presence status reflects reality. +**Why heartbeat?** WebSocket connections can silently die. A user closes their laptop lid, their WiFi drops, or a NAT gateway times out the connection. The TCP stack might not detect the dead connection for minutes or hours. The heartbeat loop at `websocket_manager.py` sends a ping every `WS_HEARTBEAT_INTERVAL` (30 seconds by default). If the send fails, it means the connection is dead, and `disconnect` cleans it up. This keeps the `active_connections` dict accurate and ensures presence status reflects reality. ### Live Query Subscription -From `websocket_manager.py:203-223`: +From `websocket_manager.py`: ```python async def _subscribe_to_messages(self, user_id): - def message_callback(update: LiveMessageUpdate): - asyncio.create_task(self._handle_live_message(user_id, update)) + def message_callback(update: LiveMessageUpdate): + asyncio.create_task(self._handle_live_message(user_id, update)) - live_id = await surreal_db.live_messages_for_user( - user_id = str(user_id), - callback = message_callback - ) - self.live_query_ids[user_id] = live_id + live_id = await surreal_db.live_messages_for_user( + user_id = str(user_id), + callback = message_callback + ) + self.live_query_ids[user_id] = live_id ``` -**SurrealDB pushes new messages to the server, server forwards to client. No polling needed.** SurrealDB has a native live query feature. When you register a live query like `LIVE SELECT * FROM messages WHERE recipient_id = $user_id`, SurrealDB pushes every new matching record to the callback in real time. The server does not need to poll the database on a timer. When a new message is created in SurrealDB (by `message_service.store_encrypted_message`), the live query fires, `_handle_live_message` at `websocket_manager.py:225-251` picks it up, wraps it in a WebSocket message schema, and sends it to all of that user's active connections. +**SurrealDB pushes new messages to the server, server forwards to client. No polling needed.** SurrealDB has a native live query feature. When you register a live query like `LIVE SELECT * FROM messages WHERE recipient_id = $user_id`, SurrealDB pushes every new matching record to the callback in real time. The server does not need to poll the database on a timer. When a new message is created in SurrealDB (by `message_service.store_encrypted_message`), the live query fires, `_handle_live_message` at `websocket_manager.py` picks it up, wraps it in a WebSocket message schema, and sends it to all of that user's active connections. This is more efficient than polling because there is zero latency (the callback fires the instant the record is created) and zero wasted queries (no empty poll cycles when no messages are pending). ### Dead Connection Cleanup -The `disconnect` method at `websocket_manager.py:97-133` handles cleanup: +The `disconnect` method at `websocket_manager.py` handles cleanup: ```python async def disconnect(self, websocket, user_id): - if user_id in self.active_connections: - if websocket in self.active_connections[user_id]: - self.active_connections[user_id].remove(websocket) + if user_id in self.active_connections: + if websocket in self.active_connections[user_id]: + self.active_connections[user_id].remove(websocket) - if not self.active_connections[user_id]: - del self.active_connections[user_id] - await presence_service.set_user_offline(user_id) - # Kill live query - # Cancel heartbeat task + if not self.active_connections[user_id]: + del self.active_connections[user_id] + await presence_service.set_user_offline(user_id) + # Kill live query + # Cancel heartbeat task ``` -Dead connections are also detected during `send_message` at `websocket_manager.py:135-153`. If sending to a WebSocket raises an exception, that WebSocket is added to a `dead_connections` list, and each dead connection is cleaned up via `disconnect` after the send loop completes. This lazy cleanup means the system self-heals: even if a heartbeat fails to detect a dead connection, the next message send attempt will catch it. +Dead connections are also detected during `send_message` at `websocket_manager.py`. If sending to a WebSocket raises an exception, that WebSocket is added to a `dead_connections` list, and each dead connection is cleaned up via `disconnect` after the send loop completes. This lazy cleanup means the system self-heals: even if a heartbeat fails to detect a dead connection, the next message send attempt will catch it. The cleanup only sets the user offline and kills the live query when the last connection for that user is removed. If they still have another tab open, their other connection keeps working and they stay online. @@ -736,24 +734,24 @@ The cleanup only sets the user offline and kills the live query when the last co ### Server-Side Storage (Passthrough) -From `message_service.py:269-314`: +From `backend/app/services/message_service.py`: ```python async def store_encrypted_message( - self, session, sender_id, recipient_id, ciphertext, nonce, header, room_id + self, session, sender_id, recipient_id, ciphertext, nonce, header, room_id ): - ... - surreal_message = { - "sender_id": str(sender_id), - "recipient_id": str(recipient_id), - "room_id": room_id, - "ciphertext": ciphertext, - "nonce": nonce, - "header": header, - "sender_username": sender_user.username, - ... - } - result = await surreal_db.create_message(surreal_message) + ... + surreal_message = { + "sender_id": str(sender_id), + "recipient_id": str(recipient_id), + "room_id": room_id, + "ciphertext": ciphertext, + "nonce": nonce, + "header": header, + "sender_username": sender_user.username, + ... + } + result = await surreal_db.create_message(surreal_message) ``` The server stores encrypted data without any decryption capability. The `ciphertext`, `nonce`, and `header` fields are base64-encoded strings that arrive from the client and get stored as-is. The server never sees the plaintext. The server never sees the encryption keys. The server could be fully compromised and the attacker still could not read messages, because the message keys exist only on the clients' devices in IndexedDB. @@ -762,30 +760,30 @@ This is the correct E2E approach. Many "encrypted" chat systems encrypt on the s ### Conversation Initialization -From `message_service.py:48-166`: +From `backend/app/services/message_service.py`: ```python async def initialize_conversation(self, session, sender_id, recipient_id): - # Check for existing ratchet state - # Fetch sender's identity key - # Fetch recipient's prekey bundle - # Perform X3DH sender-side - x3dh_result = x3dh_manager.perform_x3dh_sender( - alice_identity_private_x25519 = sender_ik.private_key, - bob_bundle = recipient_bundle, - bob_identity_public_ed25519 = recipient_ik.public_key_ed25519 - ) - # Initialize Double Ratchet with shared key - dr_state = double_ratchet.initialize_sender( - shared_key = x3dh_result.shared_key, - peer_public_key = recipient_spk_public_bytes - ) - # Serialize and persist ratchet state to PostgreSQL + # Check for existing ratchet state + # Fetch sender's identity key + # Fetch recipient's prekey bundle + # Perform X3DH sender-side + x3dh_result = initiateX3DH (frontend/src/crypto/x3dh.ts)( + alice_identity_private_x25519 = sender_ik.private_key, + bob_bundle = recipient_bundle, + bob_identity_public_ed25519 = recipient_ik.public_key_ed25519 + ) + # Initialize Double Ratchet with shared key + dr_state = double_ratchet.initialize_sender( + shared_key = x3dh_result.shared_key, + peer_public_key = recipient_spk_public_bytes + ) + # Serialize and persist ratchet state to PostgreSQL ``` This is the bridge between key exchange and ongoing encryption. The method orchestrates the entire flow: check if a conversation already exists (idempotency), fetch both users' keys from PostgreSQL, perform the X3DH exchange, initialize the Double Ratchet with the resulting shared key, serialize the ratchet state, and persist it. After this method completes, `encrypt_message` and `decrypt_message` can operate using the stored ratchet state. -The check at `message_service.py:60-73` is important: if a ratchet state already exists for this sender-recipient pair, the method returns it without doing anything. This prevents accidental re-initialization, which would generate a new shared secret and break the existing conversation. +The check at `backend/app/services/message_service.py` is important: if a ratchet state already exists for this sender-recipient pair, the method returns it without doing anything. This prevents accidental re-initialization, which would generate a new shared secret and break the existing conversation. --- @@ -793,16 +791,16 @@ The check at `message_service.py:60-73` is important: if a ratchet state already ### Constant-Time Comparison -From `primitives.ts:388-397`: +From `frontend/src/crypto/primitives.ts`: ```typescript export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean { - if (a.length !== b.length) return false - let result = 0 - for (let i = 0; i < a.length; i++) { - result |= a[i] ^ b[i] - } - return result === 0 + if (a.length !== b.length) return false + let result = 0 + for (let i = 0; i < a.length; i++) { + result |= a[i] ^ b[i] + } + return result === 0 } ``` @@ -814,40 +812,40 @@ The length check at the top does leak whether the lengths are equal. This is gen ### Random Number Generation -On the Python backend, `os.urandom(AES_GCM_NONCE_SIZE)` at `double_ratchet.py:122` reads from the operating system's cryptographically secure random number generator (CSPRNG). On Linux, this is backed by `/dev/urandom`, which draws entropy from hardware events (disk timing, network interrupts, etc.) and maintains a CSPRNG state that is computationally indistinguishable from true randomness. +On the Python backend, `os.urandom(AES_GCM_NONCE_SIZE)` at `frontend/src/crypto/double-ratchet.ts` reads from the operating system's cryptographically secure random number generator (CSPRNG). On Linux, this is backed by `/dev/urandom`, which draws entropy from hardware events (disk timing, network interrupts, etc.) and maintains a CSPRNG state that is computationally indistinguishable from true randomness. -On the browser frontend, the `generateRandomBytes` function at `primitives.ts:294-298`: +On the browser frontend, the `generateRandomBytes` function at `frontend/src/crypto/primitives.ts`: ```typescript export function generateRandomBytes(length: number): Uint8Array { - const bytes = new Uint8Array(length) - crypto.getRandomValues(bytes) - return bytes + const bytes = new Uint8Array(length) + crypto.getRandomValues(bytes) + return bytes } ``` This calls the WebCrypto API's CSPRNG. Every major browser implements this using the OS CSPRNG under the hood. Chrome uses BoringSSL's CSPRNG, Firefox uses NSS, Safari uses CommonCrypto. All ultimately draw from the kernel's entropy pool. -The same function is used for nonce generation in `aesGcmEncrypt` at `primitives.ts:243`: `const nonce = generateRandomBytes(AES_GCM_NONCE_SIZE)`. It is also used for WebAuthn challenge generation on the server side via `secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)` at `passkey_manager.py:65`, which is Python's wrapper around `os.urandom`. +The same function is used for nonce generation in `aesGcmEncrypt` at `frontend/src/crypto/primitives.ts`: `const nonce = generateRandomBytes(AES_GCM_NONCE_SIZE)`. It is also used for WebAuthn challenge generation on the server side via `secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)` at `backend/app/core/passkey/passkey_manager.py`, which is Python's wrapper around `os.urandom`. **Why CSPRNG?** Nonces, challenges, and ephemeral keys must be unpredictable. If an attacker can predict the next nonce, they can precompute the keystream for AES-GCM and decrypt messages in real time. If they can predict the next ephemeral key, they can compute the DH shared secret. If they can predict the next WebAuthn challenge, they can pre-sign a response and replay it. -**Why not Math.random()?** `Math.random()` uses a PRNG (pseudorandom number generator) seeded from a low-entropy source (often just the current time or a simple counter). Most JavaScript engines use xorshift128+ or a similar fast PRNG. The internal state is typically 128 bits, and it can be reconstructed by observing as few as 4-5 consecutive outputs. An attacker who knows the internal state of `Math.random()` can predict every future output, which would compromise every nonce and every ephemeral key generated by the application. In 2015, researchers demonstrated recovering the full xorshift128+ state from V8's `Math.random()` by observing just a few outputs. `Math.random()` is fine for shuffling a deck of cards in a UI animation. It is catastrophic for cryptography. +**Why not Math.random?** `Math.random` uses a PRNG (pseudorandom number generator) seeded from a low-entropy source (often just the current time or a simple counter). Most JavaScript engines use xorshift128+ or a similar fast PRNG. The internal state is typically 128 bits, and it can be reconstructed by observing as few as 4-5 consecutive outputs. An attacker who knows the internal state of `Math.random` can predict every future output, which would compromise every nonce and every ephemeral key generated by the application. In 2015, researchers demonstrated recovering the full xorshift128+ state from V8's `Math.random` by observing just a few outputs. `Math.random` is fine for shuffling a deck of cards in a UI animation. It is catastrophic for cryptography. ### Key Serialization -All keys are serialized as base64url-encoded raw bytes. For example, in `x3dh_manager.py:67-85`: +All keys are serialized as base64url-encoded raw bytes. For example, in `frontend/src/crypto/x3dh.ts`: ```python private_bytes = private_key.private_bytes( - encoding = serialization.Encoding.Raw, - format = serialization.PrivateFormat.Raw, - encryption_algorithm = serialization.NoEncryption() + encoding = serialization.Encoding.Raw, + format = serialization.PrivateFormat.Raw, + encryption_algorithm = serialization.NoEncryption ) ... return ( - bytes_to_base64url(private_bytes), - bytes_to_base64url(public_bytes) + bytes_to_base64url(private_bytes), + bytes_to_base64url(public_bytes) ) ``` @@ -863,53 +861,53 @@ Here is what happens when Alice types "Hello Bob" and it reaches Bob's screen. E **1. Alice types "Hello Bob" in Chat.tsx.** The `ChatInput` component captures the text in a form submit handler. The handler creates a temporary message ID (a random string for optimistic UI updates) and calls into the message sending logic. The plaintext exists only in the browser's JavaScript heap at this point. -**2. crypto-service.ts `encrypt("bob-uuid", "Hello Bob")`.** The `CryptoService.encrypt` method at `crypto-service.ts:217-249` is invoked. It first checks whether a ratchet session exists with Bob. +**2. crypto-service.ts `encrypt("bob-uuid", "Hello Bob")`.** The `CryptoService.encrypt` method at `frontend/src/crypto/crypto-service.ts` is invoked. It first checks whether a ratchet session exists with Bob. -**3. Load ratchet state from IndexedDB.** `getRatchetState(peerId)` at `crypto-service.ts:303-315` checks the in-memory `Map` first (keyed by peer ID). If the state is not in memory, it queries IndexedDB via `key-store.ts`, deserializes the stored JSON back into a `DoubleRatchetState` object (reconstructing CryptoKey objects from base64-encoded bytes), and caches it. If no state exists at all, `establishSession(peerId)` triggers the full X3DH flow: fetch Bob's prekey bundle from the server via `api.encryption.getPrekeyBundle`, perform client-side X3DH via `x3dh.ts:initiateX3DH`, and initialize the sending ratchet via `double-ratchet.ts:initializeRatchetSender`. +**3. Load ratchet state from IndexedDB.** `getRatchetState(peerId)` at `frontend/src/crypto/crypto-service.ts` checks the in-memory `Map` first (keyed by peer ID). If the state is not in memory, it queries IndexedDB via `key-store.ts`, deserializes the stored JSON back into a `DoubleRatchetState` object (reconstructing CryptoKey objects from base64-encoded bytes), and caches it. If no state exists at all, `establishSession(peerId)` triggers the full X3DH flow: fetch Bob's prekey bundle from the server via `api.encryption.getPrekeyBundle`, perform client-side X3DH via `x3dh.ts:initiateX3DH`, and initialize the sending ratchet via `double-ratchet.ts:initializeRatchetSender`. -**4. double-ratchet.ts `encryptMessage`: advance chain key via HMAC.** At `double-ratchet.ts:167`, `deriveMessageKey(state.sending_chain_key)` computes two values: - - `messageKey = HMAC-SHA-256(chainKey, "MessageKey")` - the key that will encrypt this specific message - - `nextChainKey = HMAC-SHA-256(chainKey, "ChainKey")` - the chain key for the next message +**4. double-ratchet.ts `encryptMessage`: advance chain key via HMAC.** At `double-ratchet.ts`, `deriveMessageKey(state.sending_chain_key)` computes two values: + - `messageKey = HMAC-SHA-256(chainKey, "MessageKey")` - the key that will encrypt this specific message + - `nextChainKey = HMAC-SHA-256(chainKey, "ChainKey")` - the chain key for the next message The old `sending_chain_key` is immediately overwritten with `nextChainKey`. The `messageKey` is used once and never stored. This is the forward secrecy mechanism: after encryption, the message key cannot be recovered from the updated chain state. -**5. primitives.ts `aesGcmEncrypt` with message key.** At `double-ratchet.ts:178`, the 32-byte `messageKey` is imported as an AES-GCM CryptoKey. The `plaintext` bytes (`TextEncoder.encode("Hello Bob")` = 9 bytes) are encrypted. A 12-byte random nonce is generated via `crypto.getRandomValues`. The message header (DH public key, message number, previous chain length) is serialized to JSON and used as Additional Authenticated Data (AAD). AES-GCM produces the ciphertext (9 bytes of encrypted data + 16 bytes of authentication tag = 25 bytes total). +**5. primitives.ts `aesGcmEncrypt` with message key.** At `double-ratchet.ts`, the 32-byte `messageKey` is imported as an AES-GCM CryptoKey. The `plaintext` bytes (`TextEncoder.encode("Hello Bob")` = 9 bytes) are encrypted. A 12-byte random nonce is generated via `crypto.getRandomValues`. The message header (DH public key, message number, previous chain length) is serialized to JSON and used as Additional Authenticated Data (AAD). AES-GCM produces the ciphertext (9 bytes of encrypted data + 16 bytes of authentication tag = 25 bytes total). -**6. Return {ciphertext, nonce, header}.** The `EncryptedMessage` object at `double-ratchet.ts:183-187` contains raw `Uint8Array` ciphertext and nonce, plus a typed `MessageHeader` object. Back in `crypto-service.ts:244-248`, the ciphertext and nonce are base64-encoded to strings, and the header (including any pending X3DH header for first messages) is JSON-stringified into a single string. The result is three strings: `ciphertext`, `nonce`, and `header`. All three are opaque to anyone without the message key. +**6. Return {ciphertext, nonce, header}.** The `EncryptedMessage` object at `double-ratchet.ts` contains raw `Uint8Array` ciphertext and nonce, plus a typed `MessageHeader` object. Back in `frontend/src/crypto/crypto-service.ts`, the ciphertext and nonce are base64-encoded to strings, and the header (including any pending X3DH header for first messages) is JSON-stringified into a single string. The result is three strings: `ciphertext`, `nonce`, and `header`. All three are opaque to anyone without the message key. -**7. WebSocket sends JSON payload.** `websocket-manager.ts:108-124` calls `sendEncryptedMessage`, which constructs the outgoing message: +**7. WebSocket sends JSON payload.** `websocket-manager.ts` calls `sendEncryptedMessage`, which constructs the outgoing message: ```json { - "type": "encrypted_message", - "recipient_id": "bob-uuid", - "room_id": "room-uuid", - "ciphertext": "base64...", - "nonce": "base64...", - "header": "{\"ratchet\":{\"dh_public_key\":\"...\",\"message_number\":0,...}}", - "temp_id": "temp-abc123" + "type": "encrypted_message", + "recipient_id": "bob-uuid", + "room_id": "room-uuid", + "ciphertext": "base64...", + "nonce": "base64...", + "header": "{\"ratchet\":{\"dh_public_key\":\"...\",\"message_number\":0,...}}", + "temp_id": "temp-abc123" } ``` This JSON is sent via `ws.send(JSON.stringify(message))`. If the WebSocket is not connected, the message is queued in `messageQueue` and sent when the connection is restored. -**8. websocket.py:46-54 receives.** The `websocket_endpoint` at `websocket.py:46` calls `data = await websocket.receive_text()`, parses the JSON with `json.loads(data)`, and passes the resulting dict to `websocket_service.route_message`. At this point the server has the encrypted blob. It cannot decrypt it. It does not have the message key, the chain key, or any ratchet state for Alice and Bob's conversation (in the E2E model). +**8. websocket.py receives.** The `websocket_endpoint` at `websocket.py` calls `data = await websocket.receive_text`, parses the JSON with `json.loads(data)`, and passes the resulting dict to `websocket_service.route_message`. At this point the server has the encrypted blob. It cannot decrypt it. It does not have the message key, the chain key, or any ratchet state for Alice and Bob's conversation (in the E2E model). -**9. websocket_service.py routes to handler.** At `websocket_service.py:63`, the message type `"encrypted_message"` dispatches to `handle_encrypted_message` at `websocket_service.py:87-179`. The handler extracts `recipient_id`, `room_id`, `ciphertext`, `nonce`, and `header` from the message dict. +**9. websocket_service.py routes to handler.** At `backend/app/services/websocket_service.py`, the message type `"encrypted_message"` dispatches to `handle_encrypted_message` at `backend/app/services/websocket_service.py`. The handler extracts `recipient_id`, `room_id`, `ciphertext`, `nonce`, and `header` from the message dict. -**10. message_service.py:269-314 stores in SurrealDB.** The handler calls `message_service.store_encrypted_message`, which constructs a SurrealDB document: +**10. backend/app/services/message_service.py stores in SurrealDB.** The handler calls `message_service.store_encrypted_message`, which constructs a SurrealDB document: ```python surreal_message = { - "sender_id": str(sender_id), - "recipient_id": str(recipient_id), - "room_id": room_id, - "ciphertext": ciphertext, # still base64, still encrypted - "nonce": nonce, # still base64 - "header": header, # still JSON string - "sender_username": sender_user.username, - "created_at": now.isoformat(), - "updated_at": now.isoformat(), + "sender_id": str(sender_id), + "recipient_id": str(recipient_id), + "room_id": room_id, + "ciphertext": ciphertext, # still base64, still encrypted + "nonce": nonce, # still base64 + "header": header, # still JSON string + "sender_username": sender_user.username, + "created_at": now.isoformat, + "updated_at": now.isoformat, } ``` @@ -917,38 +915,38 @@ The document is written to SurrealDB via `surreal_db.create_message(surreal_mess **11. SurrealDB live query fires.** The instant the message record is created, SurrealDB pushes a `CREATE` event to any registered live query matching `recipient_id = bob_uuid`. This is a push, not a pull. There is no polling interval. The latency from write to callback is measured in single-digit milliseconds. -**12. websocket_manager.py:225-251 picks up the update.** The `_handle_live_message` callback at `websocket_manager.py:225` receives the `LiveMessageUpdate`. It checks that `update.action == "CREATE"` (ignoring updates and deletes). It wraps the message data in an `EncryptedMessageWS` Pydantic schema and calls `self.send_message(user_id, ws_message.model_dump(mode="json"))`. +**12. websocket_manager.py picks up the update.** The `_handle_live_message` callback at `websocket_manager.py` receives the `LiveMessageUpdate`. It checks that `update.action == "CREATE"` (ignoring updates and deletes). It wraps the message data in an `EncryptedMessageWS` Pydantic schema and calls `self.send_message(user_id, ws_message.model_dump(mode="json"))`. -**13. ConnectionManager sends to Bob's WebSocket(s).** At `websocket_manager.py:135-153`, the manager iterates over all of Bob's active connections (he might have the app open on his phone and laptop) and calls `websocket.send_json(message)` on each. If any send fails, that connection is added to a `dead_connections` list and cleaned up after the loop. +**13. ConnectionManager sends to Bob's WebSocket(s).** At `websocket_manager.py`, the manager iterates over all of Bob's active connections (he might have the app open on his phone and laptop) and calls `websocket.send_json(message)` on each. If any send fails, that connection is added to a `dead_connections` list and cleaned up after the loop. -**14. Bob's websocket-manager.ts receives.** The `handleMessage` method at `websocket-manager.ts:166-178` parses the JSON from `event.data`. The `routeMessage` method at `websocket-manager.ts:180-201` runs through type guards. `isEncryptedMessageWS(message)` matches because the message has `type: "encrypted_message"` and the required fields. The message is dispatched to `handleWSMessage` in `message-handlers.ts`. +**14. Bob's websocket-manager.ts receives.** The `handleMessage` method at `websocket-manager.ts` parses the JSON from `event.data`. The `routeMessage` method at `websocket-manager.ts` runs through type guards. `isEncryptedMessageWS(message)` matches because the message has `type: "encrypted_message"` and the required fields. The message is dispatched to `handleWSMessage` in `message-handlers.ts`. -**15. crypto-service.ts `decrypt(alice_id, ciphertext, nonce, header)`.** The message handler calls `CryptoService.decrypt` at `crypto-service.ts:251-301`. The `header` string is parsed to extract the ratchet header (DH public key, message number, previous chain length) and optionally an X3DH header (if this is the first message in a new conversation). +**15. crypto-service.ts `decrypt(alice_id, ciphertext, nonce, header)`.** The message handler calls `CryptoService.decrypt` at `frontend/src/crypto/crypto-service.ts`. The `header` string is parsed to extract the ratchet header (DH public key, message number, previous chain length) and optionally an X3DH header (if this is the first message in a new conversation). -**16. Load ratchet state from IndexedDB.** Same mechanism as step 3, but for Bob's side. If no ratchet state exists for Alice, and the message includes an X3DH header, `handleIncomingSession` at `crypto-service.ts:168-215` triggers: - - Look up Alice's identity key from the X3DH header - - Look up Bob's signed prekey private key from IndexedDB - - Look up the consumed one-time prekey if one was used - - Call `receiveX3DH` from `x3dh.ts:175-215` to compute the same shared secret Alice computed - - Call `initializeRatchetReceiver` from `double-ratchet.ts:68-88` with the shared secret and Bob's signed prekey pair +**16. Load ratchet state from IndexedDB.** Same mechanism as step 3, but for Bob's side. If no ratchet state exists for Alice, and the message includes an X3DH header, `handleIncomingSession` at `frontend/src/crypto/crypto-service.ts` triggers: + - Look up Alice's identity key from the X3DH header + - Look up Bob's signed prekey private key from IndexedDB + - Look up the consumed one-time prekey if one was used + - Call `receiveX3DH` from `x3dh.ts` to compute the same shared secret Alice computed + - Call `initializeRatchetReceiver` from `double-ratchet.ts` with the shared secret and Bob's signed prekey pair -**17. Check skipped keys, advance receiving chain.** `decryptMessage` at `double-ratchet.ts:190-238` runs through the three-case logic: - - Check if this message's key was previously stored (skipped key cache hit) - - Check if Alice's DH public key in the header differs from the last known key (DH ratchet step needed) - - Check if the message number is ahead of the current position (skip intermediate keys) - Then derive the message key: `messageKey = HMAC-SHA-256(receivingChainKey, "MessageKey")`. +**17. Check skipped keys, advance receiving chain.** `decryptMessage` at `double-ratchet.ts` runs through the three-case logic: + - Check if this message's key was previously stored (skipped key cache hit) + - Check if Alice's DH public key in the header differs from the last known key (DH ratchet step needed) + - Check if the message number is ahead of the current position (skip intermediate keys) + Then derive the message key: `messageKey = HMAC-SHA-256(receivingChainKey, "MessageKey")`. -**18. primitives.ts `aesGcmDecrypt`.** At `double-ratchet.ts:232`, AES-256-GCM decryption is performed. The 32-byte message key is imported as a CryptoKey. The 12-byte nonce from the message is used as the IV. The header JSON is reconstructed and used as AAD. `subtle.decrypt` verifies the 16-byte authentication tag and produces the 9 plaintext bytes. If the tag does not match (tampered message, wrong key, wrong nonce, wrong AAD), the WebCrypto API throws an `OperationError`. +**18. primitives.ts `aesGcmDecrypt`.** At `double-ratchet.ts`, AES-256-GCM decryption is performed. The 32-byte message key is imported as a CryptoKey. The 12-byte nonce from the message is used as the IV. The header JSON is reconstructed and used as AAD. `subtle.decrypt` verifies the 16-byte authentication tag and produces the 9 plaintext bytes. If the tag does not match (tampered message, wrong key, wrong nonce, wrong AAD), the WebCrypto API throws an `OperationError`. -**19. Return "Hello Bob".** The `Uint8Array` plaintext bytes are decoded via `new TextDecoder().decode(plaintextBytes)` at `crypto-service.ts:300`, producing the string `"Hello Bob"`. The ratchet state is serialized and saved back to IndexedDB. +**19. Return "Hello Bob".** The `Uint8Array` plaintext bytes are decoded via `new TextDecoder.decode(plaintextBytes)` at `frontend/src/crypto/crypto-service.ts`, producing the string `"Hello Bob"`. The ratchet state is serialized and saved back to IndexedDB. **20. Chat.tsx renders plaintext.** The decrypted message is added to the message store. The reactive SolidJS component re-renders the message list. Bob sees "Hello Bob" on screen. The entire journey from Alice's keypress to Bob's screen took: - - ~1ms for client-side encryption (HMAC + AES-GCM) - - ~10-50ms for WebSocket round trip (depends on network) - - ~1ms for SurrealDB write + live query fire - - ~10-50ms for WebSocket delivery to Bob - - ~1ms for client-side decryption - Total: roughly 25-100ms end to end. + - ~1ms for client-side encryption (HMAC + AES-GCM) + - ~10-50ms for WebSocket round trip (depends on network) + - ~1ms for SurrealDB write + live query fire + - ~10-50ms for WebSocket delivery to Bob + - ~1ms for client-side decryption + Total: roughly 25-100ms end to end. The plaintext "Hello Bob" existed in memory on Alice's device and Bob's device. At no other point in the pipeline, not on the WebSocket server, not in SurrealDB, not in PostgreSQL, not in Redis, did the plaintext exist. That is what end-to-end encryption means in practice. @@ -958,23 +956,23 @@ The plaintext "Hello Bob" existed in memory on Alice's device and Bob's device. ### Encryption Failures -When encryption fails (corrupted ratchet state, invalid key, etc.), an `EncryptionError` is raised at `message_service.py:366`. The exception hierarchy in `exceptions.py:73-76` derives from `AppException`. The exception handler returns the error to the client via the WebSocket error message type. The message is never stored, because the encryption failed before the store call. +When encryption fails (corrupted ratchet state, invalid key, etc.), an `EncryptionError` is raised at `backend/app/services/message_service.py`. The exception hierarchy in `exceptions.py` derives from `AppException`. The exception handler returns the error to the client via the WebSocket error message type. The message is never stored, because the encryption failed before the store call. ### Database Transaction Failures -`IntegrityError` is caught in every service method that writes to the database. For example, at `message_service.py:161-164`: +`IntegrityError` is caught in every service method that writes to the database. For example, at `backend/app/services/message_service.py`: ```python except IntegrityError as e: - await session.rollback() - raise DatabaseError("Failed to initialize conversation") from e + await session.rollback + raise DatabaseError("Failed to initialize conversation") from e ``` The pattern is consistent: catch the SQLAlchemy integrity error, roll back the session to a clean state, then raise a custom `DatabaseError`. This ensures the database is never left in a partially committed state. The `from e` chain preserves the original exception for logging. ### WebSocket Disconnection -When a WebSocket disconnects (user closes the tab, network drops), `WebSocketDisconnect` is caught at `websocket.py:79`. The `finally` block at `websocket.py:84` always runs `connection_manager.disconnect(websocket, user_uuid)`, which: +When a WebSocket disconnects (user closes the tab, network drops), `WebSocketDisconnect` is caught at `websocket.py`. The `finally` block at `websocket.py` always runs `connection_manager.disconnect(websocket, user_uuid)`, which: - Removes the WebSocket from the connection pool - If it was the last connection for that user: kills the SurrealDB live query, cancels the heartbeat task, and sets presence to offline via Redis - If other connections remain: just removes this one and logs the remaining count @@ -985,7 +983,7 @@ When a WebSocket disconnects (user closes the tab, network drops), `WebSocketDis ### Unit Tests -**Test X3DH key agreement.** Generate identity keys for Alice and Bob. Generate a signed prekey for Bob. Generate an OPK for Bob. Assemble Bob's prekey bundle. Perform sender-side X3DH as Alice via `x3dh_manager.perform_x3dh_sender`. Perform receiver-side X3DH as Bob via `x3dh_manager.perform_x3dh_receiver`, passing Alice's ephemeral public key and identity key from the sender result. Assert that `alice_result.shared_key == bob_result.shared_key`. This verifies the commutative DH math. Also assert that `alice_result.associated_data == bob_result.associated_data`, confirming both sides compute the same AAD. +**Test X3DH key agreement.** Generate identity keys for Alice and Bob. Generate a signed prekey for Bob. Generate an OPK for Bob. Assemble Bob's prekey bundle. Perform sender-side X3DH as Alice via `initiateX3DH (frontend/src/crypto/x3dh.ts)`. Perform receiver-side X3DH as Bob via `receiveX3DH (frontend/src/crypto/x3dh.ts)`, passing Alice's ephemeral public key and identity key from the sender result. Assert that `alice_result.shared_key == bob_result.shared_key`. This verifies the commutative DH math. Also assert that `alice_result.associated_data == bob_result.associated_data`, confirming both sides compute the same AAD. Test the OPK-absent case separately: set `bob_bundle.one_time_prekey = None`, pass `bob_one_time_prekey_private = None` on the receiver side. Verify the shared keys still match (they will, because both sides fall back to the 3-DH variant). @@ -1009,15 +1007,15 @@ For clone detection, call `verify_authentication` with `credential_current_sign_ ### Integration Tests -**Full message flow.** Register two users with WebAuthn mocks. Initialize encryption keys for both via `prekey_service.initialize_user_keys`. Fetch user B's prekey bundle. Initialize a conversation from A to B via `message_service.initialize_conversation`. Send an encrypted message from A. On user B's side, load the ratchet state (performing receiver-side X3DH), and decrypt the message. Verify the plaintext matches. This exercises the entire pipeline: key generation, prekey bundle assembly, X3DH, Double Ratchet initialization, encryption, SurrealDB storage, and decryption. +**Full message flow (frontend).** Stand up two clean clients in `vitest`, run `generateIdentityKeyPair` / `generateSignedPreKey` / `generateOneTimePreKeys` for each, hand Alice Bob's bundle, run `initiateX3DH` and `initializeRatchetSender`, encrypt a message, then on Bob's side run `receiveX3DH` + `initializeRatchetReceiver` and `decryptMessage`. Assert the plaintext matches. The existing `frontend/src/crypto/x3dh.test.ts` and `frontend/src/crypto/double-ratchet.test.ts` already cover this end-to-end. -**WebSocket lifecycle.** Start the FastAPI test server. Connect two test WebSocket clients (simulating Alice and Bob). Send a typed message from Alice via `ws.send_json({"type": "encrypted_message", ...})`. Assert Bob's WebSocket receives the forwarded message. Send a typing indicator from Alice. Assert Bob receives it. Disconnect Alice. Assert Bob receives a presence update (or verify via the presence service that Alice's status is now offline). Reconnect Alice. Verify the heartbeat starts and presence is restored. +**WebSocket lifecycle.** Spin up the FastAPI test app, register two users via the auth flow (which sets a session cookie), open two WebSocket connections carrying the cookies, and exercise the full message round-trip. The cookie-authenticated WebSocket is the only auth path — there is no `?user_id=` query string anymore. -**Database rollback on failure.** Create an async session. Start an operation that will fail (e.g., insert a `RatchetState` with a `user_id` that violates a foreign key constraint). Assert that the session rolls back cleanly and raises `DatabaseError`. Verify that no partial data was committed by querying the table afterward. +**Database rollback on failure.** Create an async session. Start an operation that will fail (for example, inserting an `IdentityKey` row whose `user_id` doesn't exist). Assert that the session rolls back cleanly and raises `DatabaseError`. Verify that no partial data was committed by querying the table afterward. -**Key rotation.** Initialize keys for a user. Verify they have an active signed prekey. Call `prekey_service.rotate_signed_prekey`. Verify the old SPK is now inactive. Verify the new SPK is active. Fetch the prekey bundle. Verify the bundle contains the new SPK, not the old one. Verify the signature on the new SPK validates against the user's Ed25519 identity key. +**Key rotation.** Generate a fresh signed prekey on the client (`generateSignedPreKey`), call `uploadKeys` again with it, then re-fetch the bundle as a peer. Verify the bundle contains the new SPK and that its signature validates against the user's Ed25519 identity key. Confirm that the previously-active SPK row is now `is_active = False` in PostgreSQL. -**OPK exhaustion.** Initialize a user with 2 OPKs. Fetch the prekey bundle twice (consuming both OPKs). Fetch a third time. Verify the bundle is returned with `one_time_prekey = None`. Verify X3DH still succeeds without the OPK (the 3-DH fallback). +**OPK exhaustion.** Upload a user with 2 OPKs. Fetch their prekey bundle twice (consuming both, marking them `is_used = True`). Fetch a third time and verify the bundle returns with `one_time_prekey = None`. Verify X3DH still succeeds without the OPK (the 3-DH fallback). --- @@ -1029,7 +1027,7 @@ For clone detection, call `verify_authentication` with `credential_current_sign_ **Cause:** Reusing the same nonce with the same key completely breaks GCM's security. AES-GCM uses the nonce to generate a unique keystream via CTR mode. If two messages share the same nonce and key, XORing the two ciphertexts cancels out the keystream and produces the XOR of the two plaintexts. An attacker can then use frequency analysis or known-plaintext attacks to recover both messages. Additionally, the authentication tag becomes forgeable. -**Fix:** Always use `os.urandom(12)` for a fresh nonce, as done at `double_ratchet.py:122`. Since each message key is derived from a unique chain key position, nonce reuse across different message keys is not a concern. The risk would only arise if the same message key were used twice, which the chain advancement prevents. +**Fix:** Always use `os.urandom(12)` for a fresh nonce, as done at `frontend/src/crypto/double-ratchet.ts`. Since each message key is derived from a unique chain key position, nonce reuse across different message keys is not a concern. The risk would only arise if the same message key were used twice, which the chain advancement prevents. ### Pitfall 2: Not Verifying Signed Prekey @@ -1037,13 +1035,13 @@ For clone detection, call `verify_authentication` with `credential_current_sign_ **Cause:** Skipping signature verification allows an attacker to substitute their own SPK into Bob's prekey bundle. Alice computes a shared secret with the attacker. The attacker independently computes a shared secret with Bob using Bob's real SPK. The attacker can now read and modify all messages. -**Fix:** Always verify before exchange, as done at `x3dh_manager.py:217-220`: +**Fix:** Always verify before exchange, as done at `frontend/src/crypto/x3dh.ts`: ```python if not self.verify_signed_prekey(bob_bundle.signed_prekey, - bob_bundle.signed_prekey_signature, - bob_identity_public_ed25519): - raise ValueError("Invalid signed prekey signature") + bob_bundle.signed_prekey_signature, + bob_identity_public_ed25519): + raise ValueError("Invalid signed prekey signature") ``` ### Pitfall 3: Storing Private Keys on Server @@ -1052,7 +1050,7 @@ if not self.verify_signed_prekey(bob_bundle.signed_prekey, **Cause:** If private key material is accessible to the server, anyone who compromises the server (SQL injection, SSH access, backup leak) can decrypt every message ever sent. -**Fix:** Client-side key generation and IndexedDB storage. The server only stores public keys. Look at `prekey_service.py:45-150`: the `store_client_keys` method receives only public keys from the client. The `private_key` fields on `IdentityKey` and `SignedPrekey` models are set to empty strings `""` when storing client-uploaded keys. Private keys live exclusively in the browser's IndexedDB, which is sandboxed per origin. +**Fix:** Client-side key generation and IndexedDB storage. The server only stores public keys. Look at `backend/app/services/prekey_service.py`: the `store_client_keys` method receives only public keys from the client. The `private_key` fields on `IdentityKey` and `SignedPrekey` models are set to empty strings `""` when storing client-uploaded keys. Private keys live exclusively in the browser's IndexedDB, which is sandboxed per origin. ### Pitfall 4: Non-Constant-Time Comparison @@ -1060,16 +1058,16 @@ if not self.verify_signed_prekey(bob_bundle.signed_prekey, **Cause:** Using `===` or `==` for byte array comparison in JavaScript short-circuits on the first mismatch. An attacker who can measure response times with sufficient precision (microsecond resolution is achievable over a network) can determine how many leading bytes are correct and brute-force the rest. -**Fix:** XOR-based comparison as in `primitives.ts:388-397`: +**Fix:** XOR-based comparison as in `frontend/src/crypto/primitives.ts`: ```typescript export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean { - if (a.length !== b.length) return false - let result = 0 - for (let i = 0; i < a.length; i++) { - result |= a[i] ^ b[i] - } - return result === 0 + if (a.length !== b.length) return false + let result = 0 + for (let i = 0; i < a.length; i++) { + result |= a[i] ^ b[i] + } + return result === 0 } ``` @@ -1079,19 +1077,19 @@ export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean { The codebase follows a strict layering that separates concerns and minimizes the blast radius of bugs. -- **Encryption logic in `core/encryption/`** (never in the API layer). The `x3dh_manager.py` and `double_ratchet.py` files have zero knowledge of HTTP, WebSocket, or databases. They take bytes in and return bytes out. `X3DHManager` does not import SQLModel. `DoubleRatchet` does not import FastAPI. This makes them independently testable: you can instantiate `X3DHManager()` in a test file, call `generate_identity_keypair_x25519()`, and verify the output without setting up a database, a Redis connection, or a web server. It also makes them reusable: if you wanted to use the same X3DH implementation in a CLI tool or a different server framework, you could import `core/encryption/` without pulling in any web dependencies. +- **Encryption logic in `frontend/src/crypto/`** (never in the API layer). The `frontend/src/crypto/x3dh.ts` and `frontend/src/crypto/double-ratchet.ts` files have zero knowledge of HTTP, WebSocket, or databases. They take bytes in and return bytes out. `X3DHManager` does not import SQLModel. `DoubleRatchet` does not import FastAPI. This makes them independently testable: you can instantiate `X3DHManager` in a test file, call `generate_identity_keypair_x25519`, and verify the output without setting up a database, a Redis connection, or a web server. It also makes them reusable: if you wanted to use the same X3DH implementation in a CLI tool or a different server framework, you could import `frontend/src/crypto/` without pulling in any web dependencies. -- **Database operations in `services/`** (never in core). The `prekey_service.py`, `message_service.py`, and `auth_service.py` files handle all SQLModel queries and SurrealDB operations. They receive an `AsyncSession` as a parameter (dependency injection) and call into `core/encryption/` for crypto operations. The crypto layer never sees a session object, and the service layer never calls OpenSSL directly. This means a database schema change only affects service files and models, not the encryption logic. +- **Database operations in `services/`** (never in core). The `prekey_service.py`, `message_service.py`, and `auth_service.py` files handle all SQLModel queries and SurrealDB operations. They receive an `AsyncSession` as a parameter (dependency injection) and call into `frontend/src/crypto/` for crypto operations. The crypto layer never sees a session object, and the service layer never calls OpenSSL directly. This means a database schema change only affects service files and models, not the encryption logic. -- **Pydantic schemas for ALL API boundaries.** Every WebSocket message type has a corresponding schema in `schemas/websocket.py` (e.g., `EncryptedMessageWS`, `TypingIndicatorWS`, `ErrorMessageWS`). Every HTTP request/response has a schema in `schemas/auth.py`, `schemas/rooms.py`, etc. This provides runtime validation: if a client sends a malformed payload (missing fields, wrong types, oversized strings), Pydantic rejects it before the data reaches service logic. For example, `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` in `config.py:44` caps the ciphertext field length, preventing a client from sending a 10GB ciphertext that would consume server memory. +- **Pydantic schemas for ALL API boundaries.** Every WebSocket message type has a corresponding schema in `schemas/websocket.py` (e.g., `EncryptedMessageWS`, `TypingIndicatorWS`, `ErrorMessageWS`). Every HTTP request/response has a schema in `schemas/auth.py`, `schemas/rooms.py`, etc. This provides runtime validation: if a client sends a malformed payload (missing fields, wrong types, oversized strings), Pydantic rejects it before the data reaches service logic. For example, `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` in `config.py` caps the ciphertext field length, preventing a client from sending a 10GB ciphertext that would consume server memory. -- **Module-level singletons for stateless managers.** At the bottom of `x3dh_manager.py:353`: `x3dh_manager = X3DHManager()`. At the bottom of `double_ratchet.py:419`: `double_ratchet = DoubleRatchet()`. At the bottom of `websocket_manager.py:296`: `connection_manager = ConnectionManager()`. These classes either hold no per-request state (X3DHManager, DoubleRatchet) or manage global connection state (ConnectionManager). A single instance per process is safe because: (a) stateless managers have no mutable state, so concurrent async tasks cannot interfere with each other; (b) the ConnectionManager's mutable state (`active_connections` dict) is accessed within a single event loop, so there are no threading concerns. +- **Module-level singletons for stateless managers.** At the bottom of `backend/app/core/websocket_manager.py`: `connection_manager = ConnectionManager`. At the bottom of `backend/app/core/redis_manager.py` and `backend/app/core/surreal_manager.py`: `redis_manager` and `surreal_db`. The X3DH and Double Ratchet implementations live in the browser as plain TypeScript modules with exported functions, so there's no equivalent server-side singleton for them. The Python singletons either manage shared connection pools or are stateless wrappers; they're safe to share across the process because (a) stateless managers have no mutable state, so concurrent async tasks cannot interfere with each other, and (b) `connection_manager.active_connections` is touched only inside the single event loop. -- **Async all the way through (no blocking I/O).** Every database query uses `await session.execute(...)`. Every WebSocket operation uses `await websocket.send_json(...)`. Every Redis call uses async methods. Every SurrealDB call uses async methods. The only synchronous operations are the CPU-bound cryptographic computations (HMAC, HKDF, AES-GCM, X25519 DH exchange), which complete in microseconds and do not benefit from async. If a crypto operation ever became slow (e.g., Argon2 password hashing taking 500ms), it would need to be offloaded to a thread pool via `asyncio.to_thread()` to avoid blocking the event loop. But the operations used here are all sub-millisecond. +- **Async all the way through (no blocking I/O).** Every database query uses `await session.execute(...)`. Every WebSocket operation uses `await websocket.send_json(...)`. Every Redis call uses async methods. Every SurrealDB call uses async methods. The only synchronous operations are the CPU-bound cryptographic computations (HMAC, HKDF, AES-GCM, X25519 DH exchange), which complete in microseconds and do not benefit from async. If a crypto operation ever became slow (e.g., Argon2 password hashing taking 500ms), it would need to be offloaded to a thread pool via `asyncio.to_thread` to avoid blocking the event loop. But the operations used here are all sub-millisecond. - **Custom exception hierarchy in `exceptions.py`.** Every error type has its own exception class (`EncryptionError`, `DecryptionError`, `KeyExchangeError`, `DatabaseError`, `UserNotFoundError`, etc.), all inheriting from `AppException`. This allows `exception_handlers.py` to map each exception type to an appropriate HTTP status code and error response. A `UserNotFoundError` returns 404. An `EncryptionError` returns 500. A `ChallengeExpiredError` returns 401. Without the hierarchy, every error would be a generic 500, and the client would have no idea what went wrong. -- **Frontend mirrors the backend structure.** The frontend's `crypto/` directory parallels the backend's `core/encryption/`. `primitives.ts` provides the same low-level operations as Python's `cryptography` library. `x3dh.ts` parallels `x3dh_manager.py`. `double-ratchet.ts` parallels `double_ratchet.py`. `crypto-service.ts` parallels `message_service.py` (the orchestration layer). This symmetry makes it easier to trace a bug: if decryption fails, you compare the frontend and backend implementations step by step, checking that the same constants, the same byte orderings, and the same HKDF parameters are used on both sides. +- **Frontend mirrors the backend structure.** The frontend's `crypto/` directory parallels the backend's `frontend/src/crypto/`. `primitives.ts` provides the same low-level operations as Python's `cryptography` library. `x3dh.ts` parallels `frontend/src/crypto/x3dh.ts`. `double-ratchet.ts` parallels `frontend/src/crypto/double-ratchet.ts`. `crypto-service.ts` parallels `message_service.py` (the orchestration layer). This symmetry makes it easier to trace a bug: if decryption fails, you compare the frontend and backend implementations step by step, checking that the same constants, the same byte orderings, and the same HKDF parameters are used on both sides. --- @@ -1103,7 +1101,7 @@ The codebase follows a strict layering that separates concerns and minimizes the - **py_webauthn:** Implements the WebAuthn/FIDO2 server-side protocol. Handles the complex ASN.1 parsing of attestation objects, CBOR decoding of authenticator data, signature verification across multiple algorithm families (ES256, RS256, EdDSA), and challenge lifecycle management. The WebAuthn specification is over 200 pages. Writing a compliant implementation from scratch would take months and invite subtle bugs in the CBOR parsing alone. py_webauthn also provides convenient helpers like `bytes_to_base64url` and `base64url_to_bytes` that we use throughout the encryption code. -- **sqlmodel:** Combines SQLAlchemy's query builder with Pydantic's validation. You define a model once (like `RatchetState` or `IdentityKey`) and get both a database table schema and a Pydantic serialization schema. The `AsyncSession` support via `sqlmodel.ext.asyncio.session` integrates cleanly with FastAPI's async request handlers. The alternative would be using raw SQLAlchemy models plus separate Pydantic schemas, which doubles the model definitions and introduces synchronization risk (changing a column in the SQLAlchemy model but forgetting to update the Pydantic schema). +- **sqlmodel:** Combines SQLAlchemy's query builder with Pydantic's validation. You define a model once (like `User` or `IdentityKey`) and get both a database table schema and a Pydantic serialization schema. The `AsyncSession` support via `sqlmodel.ext.asyncio.session` integrates cleanly with FastAPI's async request handlers. The alternative would be using raw SQLAlchemy models plus separate Pydantic schemas, which doubles the model definitions and introduces synchronization risk (changing a column in the SQLAlchemy model but forgetting to update the Pydantic schema). - **surrealdb:** The Python client for SurrealDB's WebSocket protocol. The key feature is live query support: `LIVE SELECT * FROM messages WHERE recipient_id = $user_id` creates a persistent subscription that pushes new matching records to a callback. This eliminates the need for a separate pub/sub system (like Redis Pub/Sub or RabbitMQ) for real-time message delivery. SurrealDB serves as both the message store and the real-time event bus. diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md index bfb42821..d1e315ef 100644 --- a/PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md @@ -25,23 +25,23 @@ Each challenge references actual files in this project. The line numbers point y The backend already has the scaffolding for this. Look at these files: -- `backend/app/config.py:54` defines `WS_MESSAGE_TYPE_RECEIPT = "receipt"` already -- `backend/app/services/websocket_service.py:69-70` already routes receipt messages to `handle_read_receipt` -- `backend/app/services/websocket_service.py:272-314` has a working `handle_read_receipt` implementation -- `backend/app/schemas/websocket.py:62-69` defines `ReadReceiptWS` with `message_id`, `user_id`, and `read_at` +- `backend/app/config.py` defines `WS_MESSAGE_TYPE_RECEIPT = "receipt"` already +- `backend/app/services/backend/app/services/websocket_service.py` already routes receipt messages to `handle_read_receipt` +- `backend/app/services/backend/app/services/websocket_service.py` has a working `handle_read_receipt` implementation +- `backend/app/schemas/websocket.py` defines `ReadReceiptWS` with `message_id`, `user_id`, and `read_at` So the backend is done. The work is on the frontend. **Implementation approach:** 1. When Bob's client decrypts and displays a message from Alice, emit a receipt through the WebSocket: - ```json - { - "type": "receipt", - "message_id": "messages:abc123", - "sender_id": "alice-uuid-here" - } - ``` + ```json + { + "type": "receipt", + "message_id": "messages:abc123", + "sender_id": "alice-uuid-here" + } + ``` 2. Alice's client receives the `ReadReceiptWS` via the WebSocket message handler at `frontend/src/websocket/message-handlers.ts`. Add a handler for the `"receipt"` message type. @@ -79,11 +79,11 @@ So the backend is done. The work is on the frontend. This feature is substantially built already. Study what exists: -- `backend/app/config.py:52` defines `WS_MESSAGE_TYPE_TYPING = "typing"` -- `backend/app/services/websocket_service.py:65-66` routes typing messages to `handle_typing_indicator` -- `backend/app/services/websocket_service.py:181-224` broadcasts typing events to the room via `connection_manager.broadcast_to_room` -- `backend/app/schemas/websocket.py:42-49` defines `TypingIndicatorWS` with `user_id`, `room_id`, `is_typing` -- `frontend/src/stores/typing.store.ts:1-113` is a complete typing store with auto-clear timeouts (5 seconds, line 10) +- `backend/app/config.py` defines `WS_MESSAGE_TYPE_TYPING = "typing"` +- `backend/app/services/backend/app/services/websocket_service.py` routes typing messages to `handle_typing_indicator` +- `backend/app/services/backend/app/services/websocket_service.py` broadcasts typing events to the room via `connection_manager.broadcast_to_room` +- `backend/app/schemas/websocket.py` defines `TypingIndicatorWS` with `user_id`, `room_id`, `is_typing` +- `frontend/src/stores/typing.store.ts` is a complete typing store with auto-clear timeouts (5 seconds,) - `frontend/src/components/Chat/TypingIndicator.tsx` already renders the typing indicator UI The backend and most of the frontend state management are done. What is missing: @@ -100,11 +100,11 @@ The backend and most of the frontend state management are done. What is missing: **Throttling pattern:** ``` -Keystroke at t=0ms -> send "typing: true" +Keystroke at t=0ms -> send "typing: true" Keystroke at t=500ms -> suppress (within 3s window) -Keystroke at t=1.2s -> suppress -Keystroke at t=4.5s -> send "typing: true" (new window) -No input for 3s -> send "typing: false" +Keystroke at t=1.2s -> suppress +Keystroke at t=4.5s -> send "typing: true" (new window) +No input for 3s -> send "typing: false" ``` **Do not encrypt typing events.** They are ephemeral metadata. Running them through the Double Ratchet would advance the ratchet state for throwaway data, wasting chain keys and increasing the risk of desynchronization. @@ -113,7 +113,7 @@ No input for 3s -> send "typing: false" - Open two tabs as different users in the same room - Start typing in Tab A - Verify "typing..." appears in Tab B within 1 second -- Stop typing and verify the indicator disappears after 5 seconds (the `TYPING_TIMEOUT_MS` constant at `typing.store.ts:10`) +- Stop typing and verify the indicator disappears after 5 seconds (the `TYPING_TIMEOUT_MS` constant at `typing.store.ts`) - Type rapidly and verify only one WebSocket event per 3-second window (check WS frames in DevTools) - Switch rooms and verify typing state resets @@ -123,7 +123,7 @@ No input for 3s -> send "typing: false" **What to build:** Display human-readable timestamps on messages ("2:34 PM", "Yesterday", "Jan 15") and ensure messages display in correct chronological order even when they arrive out of order over the WebSocket. -**Why it matters:** The Double Ratchet protocol handles cryptographic out-of-order delivery (see `double_ratchet.py:215-244` for the skipped message key logic), but the UI also needs to handle display ordering correctly. This is a common source of bugs in messaging apps. If Alice sends message A then message B, but B arrives at Bob first over the WebSocket, the UI must still display A before B. +**Why it matters:** The Double Ratchet protocol handles cryptographic out-of-order delivery (see `frontend/src/crypto/double-ratchet.ts` for the skipped message key logic), but the UI also needs to handle display ordering correctly. This is a common source of bugs in messaging apps. If Alice sends message A then message B, but B arrives at Bob first over the WebSocket, the UI must still display A before B. **What you will learn:** - Client-side message sorting with stable ordering @@ -132,22 +132,22 @@ No input for 3s -> send "typing: false" **Where to start reading:** -- `backend/app/schemas/websocket.py:26-39` shows `EncryptedMessageWS` includes a `timestamp` field -- `frontend/src/crypto/message-store.ts:95-119` already sorts messages by `created_at` (line 106-107) +- `backend/app/schemas/websocket.py` shows `EncryptedMessageWS` includes a `timestamp` field +- `frontend/src/crypto/message-store.ts` already sorts messages by `created_at` - `frontend/src/lib/date.ts` is the existing date utility file - `frontend/src/components/Chat/MessageBubble.tsx` renders individual messages **Implementation approach:** -1. The message store at `message-store.ts:106-107` already sorts by `created_at`. Verify this works when messages arrive out of order by checking that new messages insert at the correct position, not just appended at the end. +1. The message store at `message-store.ts` already sorts by `created_at`. Verify this works when messages arrive out of order by checking that new messages insert at the correct position, not just appended at the end. 2. Create a timestamp formatter. Use `Intl.RelativeTimeFormat` for recent messages and `Intl.DateTimeFormat` for older ones: - - Under 1 minute: "Just now" - - Under 1 hour: "12 min ago" - - Same day: "2:34 PM" - - Yesterday: "Yesterday at 2:34 PM" - - Same year: "Jan 15 at 2:34 PM" - - Older: "Jan 15, 2025 at 2:34 PM" + - Under 1 minute: "Just now" + - Under 1 hour: "12 min ago" + - Same day: "2:34 PM" + - Yesterday: "Yesterday at 2:34 PM" + - Same year: "Jan 15 at 2:34 PM" + - Older: "Jan 15, 2025 at 2:34 PM" 3. Add date separator headers between messages from different days. When rendering the message list, compare each message's date with the previous message. If they differ, insert a "--- January 15, 2025 ---" separator. @@ -175,20 +175,20 @@ No input for 3s -> send "typing: false" The backend infrastructure is already built: -- `backend/app/core/websocket_manager.py:74` calls `presence_service.set_user_online(user_id)` on WebSocket connect -- `backend/app/core/websocket_manager.py:108` calls `presence_service.set_user_offline(user_id)` on disconnect (only when the last connection for that user closes) -- `backend/app/services/presence_service.py:22-35` sets online status in SurrealDB -- `backend/app/services/presence_service.py:37-50` sets offline status -- `backend/app/services/websocket_service.py:226-270` handles presence update messages +- `backend/app/core/websocket_manager.py` calls `presence_service.set_user_online(user_id)` on WebSocket connect +- `backend/app/core/websocket_manager.py` calls `presence_service.set_user_offline(user_id)` on disconnect (only when the last connection for that user closes) +- `backend/app/services/presence_service.py` sets online status in SurrealDB +- `backend/app/services/presence_service.py` sets offline status +- `backend/app/services/backend/app/services/websocket_service.py` handles presence update messages The frontend store is also ready: -- `frontend/src/stores/presence.store.ts:1-72` has `$presenceByUser` map, `setUserPresence`, `getUserStatus`, `isUserOnline` +- `frontend/src/stores/presence.store.ts` has `$presenceByUser` map, `setUserPresence`, `getUserStatus`, `isUserOnline` - `frontend/src/components/Chat/OnlineStatus.tsx` already renders the status indicator **Implementation approach:** -1. When a user connects via WebSocket, the backend already updates SurrealDB presence. What is missing is broadcasting that change to the user's contacts. In `websocket_manager.py`, after `presence_service.set_user_online(user_id)` succeeds (line 74), broadcast a presence update to all rooms that user belongs to. +1. When a user connects via WebSocket, the backend already updates SurrealDB presence. What is missing is broadcasting that change to the user's contacts. In `websocket_manager.py`, after `presence_service.set_user_online(user_id)` succeeds , broadcast a presence update to all rooms that user belongs to. 2. Add a `PresenceUpdateWS` handler in the frontend WebSocket message handler. When a `"presence"` message arrives, call `setUserPresence` from the presence store. @@ -223,30 +223,30 @@ Signal allows searching decrypted messages on-device. WhatsApp backs up messages **Where to start reading:** -- `frontend/src/crypto/message-store.ts:1-211` is the existing IndexedDB message store with `decrypted_messages` object store -- `frontend/src/crypto/message-store.ts:34-39` creates indexes on `room_id`, `created_at`, and a compound `room_created` index -- `frontend/src/crypto/message-store.ts:95-119` shows how messages are queried by room -- `frontend/src/crypto/key-store.ts:392-412` shows `clearAllKeys()` which wipes all crypto state on logout +- `frontend/src/crypto/message-store.ts` is the existing IndexedDB message store with `decrypted_messages` object store +- `frontend/src/crypto/message-store.ts` creates indexes on `room_id`, `created_at`, and a compound `room_created` index +- `frontend/src/crypto/message-store.ts` shows how messages are queried by room +- `frontend/src/crypto/key-store.ts` shows `clearAllKeys` which wipes all crypto state on logout **Implementation approach:** -1. Decrypted messages are already stored in IndexedDB at `message-store.ts:61-67` (`saveDecryptedMessage`). The plaintext is already persisted locally. You do not need to add a separate plaintext store. +1. Decrypted messages are already stored in IndexedDB at `message-store.ts` (`saveDecryptedMessage`). The plaintext is already persisted locally. You do not need to add a separate plaintext store. 2. Add a `searchMessages` function to `message-store.ts`. IndexedDB does not support full-text search natively. You have two options: - Option A (simple): Load all messages for the user, filter with `String.prototype.includes()` or a regex. This works for small message histories (under 10,000 messages). + Option A (simple): Load all messages for the user, filter with `String.prototype.includes` or a regex. This works for small message histories (under 10,000 messages). - Option B (performant): Build a simple inverted index in a separate IndexedDB object store. On each message save, tokenize the plaintext, store `{token: string, message_id: string}` entries. Search queries look up tokens in the index. This scales to hundreds of thousands of messages. + Option B (performant): Build a simple inverted index in a separate IndexedDB object store. On each message save, tokenize the plaintext, store `{token: string, message_id: string}` entries. Search queries look up tokens in the index. This scales to hundreds of thousands of messages. 3. Build a search UI. Add a search input to the sidebar. Results should show message previews with the matching text highlighted. Clicking a result should navigate to that message in the conversation and scroll it into view. -4. Clear the search index on logout. Add this to whatever logout function already calls `clearAllKeys()` and `clearAllMessages()`. +4. Clear the search index on logout. Add this to whatever logout function already calls `clearAllKeys` and `clearAllMessages`. **Security considerations:** The local plaintext store is a security tradeoff. If the device is compromised, the search history exposes message content. Signal handles this by only searching messages from the current app session (not backed by a persistent index). Consider offering a setting: "Enable message search (stores decrypted messages locally)" vs. "Maximum security (no local message storage)". -Also consider: IndexedDB data survives browser cache clears in some browsers. Use `indexedDB.deleteDatabase()` on logout, not just clearing object stores. +Also consider: IndexedDB data survives browser cache clears in some browsers. Use `indexedDB.deleteDatabase` on logout, not just clearing object stores. **How to test:** - Send several messages containing known keywords between two users @@ -272,48 +272,48 @@ Also consider: IndexedDB data survives browser cache clears in some browsers. Us **Where to start reading:** -- `frontend/src/crypto/primitives.ts:224-259` is the `aesGcmEncrypt` function. It accepts `Uint8Array` for plaintext, which means binary data works directly with no conversion needed. -- `frontend/src/crypto/primitives.ts:261-292` is `aesGcmDecrypt` -- `backend/app/services/message_service.py:269-314` is `store_encrypted_message` which stores to SurrealDB -- `backend/app/config.py:44` defines `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` (this limits inline message content, not file uploads) +- `frontend/src/crypto/frontend/src/crypto/primitives.ts` is the `aesGcmEncrypt` function. It accepts `Uint8Array` for plaintext, which means binary data works directly with no conversion needed. +- `frontend/src/crypto/frontend/src/crypto/primitives.ts` is `aesGcmDecrypt` +- `backend/app/services/backend/app/services/message_service.py` is `store_encrypted_message` which stores to SurrealDB +- `backend/app/config.py` defines `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` (this limits inline message content, not file uploads) **Implementation approach:** 1. **Client: Read and encrypt the file** - - Use `FileReader.readAsArrayBuffer()` to get the raw bytes - - For small files (under 1MB): encrypt the entire file as one AES-256-GCM operation using a message key from the Double Ratchet - - For large files (over 1MB): chunk into 1MB segments. Encrypt each chunk with a derived key: `chunk_key = HKDF(message_key, salt=chunk_index)` - - Each chunk gets its own nonce (never reuse nonces) + - Use `FileReader.readAsArrayBuffer` to get the raw bytes + - For small files (under 1MB): encrypt the entire file as one AES-256-GCM operation using a message key from the Double Ratchet + - For large files (over 1MB): chunk into 1MB segments. Encrypt each chunk with a derived key: `chunk_key = HKDF(message_key, salt=chunk_index)` + - Each chunk gets its own nonce (never reuse nonces) 2. **Client: Upload encrypted blob** - - Create a new backend endpoint: `POST /api/messages/file` - - Send the encrypted blob as multipart form data - - Include metadata in the request: original filename, MIME type, file size, number of chunks, encryption header (same format as text message headers) + - Create a new backend endpoint: `POST /api/messages/file` + - Send the encrypted blob as multipart form data + - Include metadata in the request: original filename, MIME type, file size, number of chunks, encryption header (same format as text message headers) 3. **Backend: Store encrypted file** - - Store the encrypted blob on disk or in object storage (not in SurrealDB, which is not designed for large binary data) - - Store file metadata in SurrealDB: `file_id`, `sender_id`, `recipient_id`, `room_id`, `filename`, `mime_type`, `size`, `chunk_count`, `storage_path` - - Return a `file_id` to the client + - Store the encrypted blob on disk or in object storage (not in SurrealDB, which is not designed for large binary data) + - Store file metadata in SurrealDB: `file_id`, `sender_id`, `recipient_id`, `room_id`, `filename`, `mime_type`, `size`, `chunk_count`, `storage_path` + - Return a `file_id` to the client 4. **Client: Send file message** - - Send a regular encrypted message through the WebSocket, but the plaintext content is JSON metadata: - ```json - { - "type": "file", - "file_id": "abc123", - "filename": "document.pdf", - "mime_type": "application/pdf", - "size": 1048576, - "thumbnail": "" - } - ``` - - The recipient decrypts this metadata, then downloads the encrypted file by `file_id`, then decrypts the file client-side + - Send a regular encrypted message through the WebSocket, but the plaintext content is JSON metadata: + ```json + { + "type": "file", + "file_id": "abc123", + "filename": "document.pdf", + "mime_type": "application/pdf", + "size": 1048576, + "thumbnail": "" + } + ``` + - The recipient decrypts this metadata, then downloads the encrypted file by `file_id`, then decrypts the file client-side 5. **For images: generate encrypted thumbnails** - - Before uploading, use `` to resize the image to a thumbnail (200x200) - - Encrypt the thumbnail separately - - Include the encrypted thumbnail inline in the message (it is small enough) - - Render the thumbnail immediately, load the full image on click + - Before uploading, use `` to resize the image to a thumbnail (200x200) + - Encrypt the thumbnail separately + - Include the encrypted thumbnail inline in the message (it is small enough) + - Render the thumbnail immediately, load the full image on click **Security considerations:** - Validate file size limits on the backend to prevent DoS (suggest 100MB max) @@ -371,12 +371,12 @@ Each approach has different properties for security, UX, and complexity. This ch **Where to start reading:** -- `backend/app/core/websocket_manager.py:39` stores `active_connections: dict[UUID, list[WebSocket]]` which already supports multiple connections per user -- `backend/app/core/websocket_manager.py:52` enforces `WS_MAX_CONNECTIONS_PER_USER` (default 5, see `config.py:141`) -- `backend/app/core/websocket_manager.py:145-153` iterates over ALL connections for a user when sending a message -- `backend/app/services/prekey_service.py:152-219` initializes user keys with identity key, signed prekey, and one-time prekeys +- `backend/app/core/websocket_manager.py` stores `active_connections: dict[UUID, list[WebSocket]]` which already supports multiple connections per user +- `backend/app/core/websocket_manager.py` enforces `WS_MAX_CONNECTIONS_PER_USER` (default 5, see `config.py`) +- `backend/app/core/websocket_manager.py` iterates over ALL connections for a user when sending a message +- `backend/app/services/backend/app/services/prekey_service.py` initializes user keys with identity key, signed prekey, and one-time prekeys - `backend/app/models/IdentityKey.py` stores one identity key per user (this needs to change) -- `backend/app/services/message_service.py:48-166` `initialize_conversation` creates one ratchet per user pair (this also needs to change) +- `backend/app/services/backend/app/services/message_service.py` `initialize_conversation` creates one ratchet per user pair (this also needs to change) **Architecture change:** @@ -387,8 +387,8 @@ Alice ----[one ratchet session]----> Bob With multi-device, it becomes: ``` -Alice-Phone ----[ratchet A1-B1]----> Bob-Phone -Alice-Phone ----[ratchet A1-B2]----> Bob-Laptop +Alice-Phone ----[ratchet A1-B1]----> Bob-Phone +Alice-Phone ----[ratchet A1-B2]----> Bob-Laptop Alice-Laptop ----[ratchet A2-B1]----> Bob-Phone Alice-Laptop ----[ratchet A2-B2]----> Bob-Laptop ``` @@ -398,20 +398,19 @@ If Alice has 2 devices and Bob has 3 devices, there are 2 x 3 = 6 ratchet sessio **Implementation phases:** **Phase 1: Database schema changes** -- Add a `device_id` column to the `IdentityKey` model. Change the unique constraint from `(user_id)` to `(user_id, device_id)`. -- Add a `device_id` column to `RatchetState`. Change the unique constraint to `(user_id, peer_user_id, peer_device_id)`. +- Add a `device_id` column to the `IdentityKey`, `SignedPrekey`, and `OneTimePrekey` models. Change the unique constraint on `IdentityKey` from `(user_id)` to `(user_id, device_id)`. - Add a `devices` table: `device_id`, `user_id`, `device_name`, `created_at`, `last_active`. +- Client-side ratchet state in `frontend/src/crypto/key-store.ts` becomes keyed on `(peer_user_id, peer_device_id)` instead of just `peer_id`. **Phase 2: Device registration** - When a user registers a new device (via WebAuthn), generate a new identity key pair for that device. - Each device uploads its own prekey bundle (identity key, signed prekey, one-time prekeys). -- The prekey bundle endpoint (`prekey_service.py:293-361`) needs a `device_id` parameter. +- The prekey bundle endpoint (`backend/app/services/prekey_service.py`) needs a `device_id` parameter. **Phase 3: Message fan-out** -- When Alice sends to Bob, the backend looks up all of Bob's devices. -- For each device, it checks if a ratchet session exists. If not, it performs X3DH with that device's prekey bundle. -- Alice's client encrypts the message once per recipient device. -- The WebSocket forward logic at `websocket_service.py:87-179` sends the appropriate ciphertext to each device. +- Alice's client fetches all of Bob's per-device prekey bundles, then runs X3DH and encrypts once per device. +- The WebSocket payload from Alice to the server now carries `device_id` alongside `recipient_id`. +- The forwarding logic in `backend/app/services/websocket_service.py:handle_encrypted_message` routes each ciphertext to the recipient device's connection. **Phase 4: Device linking** - A new device should NOT just register independently. It should be "linked" by an existing device. @@ -422,13 +421,13 @@ If Alice has 2 devices and Bob has 3 devices, there are 2 x 3 = 6 ratchet sessio When Alice sends a message to Bob (who has 3 devices), the flow is: -1. Alice's client fetches prekey bundles for ALL of Bob's devices: `GET /api/encryption/prekeys/{bob_id}?all_devices=true` +1. Alice's client fetches prekey bundles for ALL of Bob's devices: `GET /encryption/prekey-bundle/{bob_id}?all_devices=true` 2. Alice performs X3DH separately with each device's prekey bundle (3 separate key exchanges) 3. Alice encrypts the message 3 times (once per ratchet session) 4. Alice sends 3 encrypted payloads to the server, each tagged with `(recipient_id, device_id)` 5. The server forwards each payload to the correct device -This means the `store_encrypted_message` function at `message_service.py:269-314` needs a `device_id` field. The SurrealDB message schema changes from `{sender_id, recipient_id, ciphertext}` to `{sender_id, sender_device_id, recipient_id, recipient_device_id, ciphertext}`. +This means the `store_encrypted_message` function at `backend/app/services/message_service.py` needs a `device_id` field. The SurrealDB message schema changes from `{sender_id, recipient_id, ciphertext}` to `{sender_id, sender_device_id, recipient_id, recipient_device_id, ciphertext}`. **Device verification:** @@ -437,9 +436,9 @@ How does Alice know she is encrypting to Bob's real devices and not a rogue devi Consider implementing a device list that shows all registered devices with their safety numbers: ``` Bob's Devices: - Phone (registered Jan 15) - Safety: 4a7b 2c3d ... - Laptop (registered Jan 20) - Safety: 9f1e 8d2c ... - Tablet (registered Feb 1) - Safety: 3b5a 7c4e ... + Phone (registered Jan 15) - Safety: 4a7b 2c3d ... + Laptop (registered Jan 20) - Safety: 9f1e 8d2c ... + Tablet (registered Feb 1) - Safety: 3b5a 7c4e ... ``` **How to test:** @@ -479,10 +478,10 @@ Signal uses Sender Keys for group chats. WhatsApp adopted the same approach. The **Where to start reading:** -- `backend/app/api/rooms.py:38-132` creates rooms with participants. This is where group support starts. -- `backend/app/core/encryption/double_ratchet.py:96-109` has the chain key derivation (`_kdf_ck`) which is the same pattern used in Sender Keys (single-direction chain) -- `backend/app/core/websocket_manager.py:155-175` `broadcast_to_room` sends to all room members -- `frontend/src/crypto/primitives.ts:224-259` AES-GCM encryption works for sender key messages too +- `backend/app/api/rooms.py` creates rooms with participants. This is where group support starts. +- `frontend/src/crypto/double-ratchet.ts` has the chain key derivation (`_kdf_ck`) which is the same pattern used in Sender Keys (single-direction chain) +- `backend/app/core/websocket_manager.py` `broadcast_to_room` sends to all room members +- `frontend/src/crypto/frontend/src/crypto/primitives.ts` AES-GCM encryption works for sender key messages too **Implementation phases:** @@ -498,52 +497,52 @@ Create these tables in SurrealDB: ``` sender_keys { - group_id: string, - user_id: string, - chain_key: bytes, - chain_iteration: int, - signing_public_key: bytes, - created_at: datetime + group_id: string, + user_id: string, + chain_key: bytes, + chain_iteration: int, + signing_public_key: bytes, + created_at: datetime } group_members { - group_id: string, - user_id: string, - role: string, // "admin" | "member" - joined_at: datetime + group_id: string, + user_id: string, + role: string, // "admin" | "member" + joined_at: datetime } ``` **Phase 3: Implementation** 1. **Group creation:** When Alice creates a group with Bob and Carol: - - Alice generates a sender key pair: `(chain_key_alice, signing_key_alice)` - - Alice sends her sender key to Bob via their existing 1:1 encrypted channel (Double Ratchet) - - Alice sends her sender key to Carol the same way - - Bob and Carol do the same (each distributes their sender key to all other members) - - Result: each member has N-1 sender keys (one per other member) + - Alice generates a sender key pair: `(chain_key_alice, signing_key_alice)` + - Alice sends her sender key to Bob via their existing 1:1 encrypted channel (Double Ratchet) + - Alice sends her sender key to Carol the same way + - Bob and Carol do the same (each distributes their sender key to all other members) + - Result: each member has N-1 sender keys (one per other member) 2. **Sending a message:** Alice encrypts the group message: - - Derive message key: `message_key = HMAC-SHA256(chain_key, 0x01)` - - Advance chain: `chain_key = HMAC-SHA256(chain_key, 0x02)` - - Encrypt: `ciphertext = AES-GCM(message_key, plaintext)` - - Sign: `signature = Ed25519.sign(signing_key, ciphertext)` - - Broadcast ciphertext + signature + chain_iteration to the group + - Derive message key: `message_key = HMAC-SHA256(chain_key, 0x01)` + - Advance chain: `chain_key = HMAC-SHA256(chain_key, 0x02)` + - Encrypt: `ciphertext = AES-GCM(message_key, plaintext)` + - Sign: `signature = Ed25519.sign(signing_key, ciphertext)` + - Broadcast ciphertext + signature + chain_iteration to the group 3. **Receiving a message:** Bob decrypts: - - Look up Alice's sender key by `(group_id, sender_id)` - - If `chain_iteration > local_iteration`: advance chain key to catch up - - Derive the message key, decrypt, verify signature + - Look up Alice's sender key by `(group_id, sender_id)` + - If `chain_iteration > local_iteration`: advance chain key to catch up + - Derive the message key, decrypt, verify signature 4. **Member removal:** When Carol is removed: - - Alice generates a NEW sender key pair - - Alice distributes the new sender key to Bob (but NOT Carol) - - Bob does the same - - Carol still has the old sender keys but they are no longer used + - Alice generates a NEW sender key pair + - Alice distributes the new sender key to Bob (but NOT Carol) + - Bob does the same + - Carol still has the old sender keys but they are no longer used 5. **Member addition:** When Dave joins: - - All existing members distribute their current sender keys to Dave via 1:1 channels - - Dave generates and distributes his sender key to all members + - All existing members distribute their current sender keys to Dave via 1:1 channels + - Dave generates and distributes his sender key to all members **Key rotation strategy:** @@ -592,22 +591,22 @@ The threat model that motivates this is "harvest now, decrypt later." An adversa **Where to start reading:** -- `backend/app/core/encryption/x3dh_manager.py:208-281` is `perform_x3dh_sender`. This is the exact function you will modify. -- `backend/app/core/encryption/x3dh_manager.py:241-255` is where the DH shared secrets are combined: - ```python - dh1 = alice_ik_private.exchange(bob_spk_public) # line 241 - dh2 = alice_ek_private.exchange(bob_ik_public) # line 242 - dh3 = alice_ek_private.exchange(bob_spk_public) # line 243 - # dh4 = alice_ek_private.exchange(bob_opk_public) # line 251 (optional) - key_material = dh1 + dh2 + dh3 [+ dh4] - ``` -- `backend/app/core/encryption/x3dh_manager.py:257-264` is where HKDF derives the shared key: - ```python - f = b'\xff' * X25519_KEY_SIZE - hkdf = HKDF(algorithm=hashes.SHA256(), length=X25519_KEY_SIZE, - salt=b'\x00' * X25519_KEY_SIZE, info=b'X3DH') - shared_key = hkdf.derive(f + key_material) - ``` +- `frontend/src/crypto/x3dh.ts` is `perform_x3dh_sender`. This is the exact function you will modify. +- `frontend/src/crypto/x3dh.ts` is where the DH shared secrets are combined: + ```python + dh1 = alice_ik_private.exchange(bob_spk_public) + dh2 = alice_ek_private.exchange(bob_ik_public) + dh3 = alice_ek_private.exchange(bob_spk_public) + # dh4 = alice_ek_private.exchange(bob_opk_public) (optional) + key_material = dh1 + dh2 + dh3 [+ dh4] + ``` +- `frontend/src/crypto/x3dh.ts` is where HKDF derives the shared key: + ```python + f = b'\xff' * X25519_KEY_SIZE + hkdf = HKDF(algorithm=hashes.SHA256, length=X25519_KEY_SIZE, + salt=b'\x00' * X25519_KEY_SIZE, info=b'X3DH') + shared_key = hkdf.derive(f + key_material) + ``` - `backend/pyproject.toml:30` already includes `liboqs-python>=0.14.1` in dependencies **The hybrid approach:** @@ -618,9 +617,9 @@ The key insight is that you combine BOTH classical and post-quantum shared secre - Both must be broken simultaneously to compromise the session ``` -Classical: SK_classical = HKDF(DH1 || DH2 || DH3 [|| DH4]) +Classical: SK_classical = HKDF(DH1 || DH2 || DH3 [|| DH4]) Post-quantum: SK_pq = Kyber.Decapsulate(ciphertext, secret_key) -Combined: SK = HKDF(SK_classical || SK_pq, info=b'PQXDH') +Combined: SK = HKDF(SK_classical || SK_pq, info=b'PQXDH') ``` **Implementation phases:** @@ -640,27 +639,27 @@ Understand the difference between a KEM (Key Encapsulation Mechanism) and a DH e **Phase 2: Prekey bundle extension** -1. Add a `kyber_public_key` field to the `PreKeyBundle` dataclass at `x3dh_manager.py:34-42` -2. When generating prekeys (`prekey_service.py:152-219`), also generate a Kyber keypair using `liboqs-python`: - ```python - import oqs - kem = oqs.KeyEncapsulation("Kyber768") - kyber_public = kem.generate_keypair() - kyber_private = kem.export_secret_key() - ``` +1. Add a `kyber_public_key` field to the `PreKeyBundle` dataclass at `frontend/src/crypto/x3dh.ts` +2. When generating prekeys (`backend/app/services/prekey_service.py`), also generate a Kyber keypair using `liboqs-python`: + ```python + import oqs + kem = oqs.KeyEncapsulation("Kyber768") + kyber_public = kem.generate_keypair + kyber_private = kem.export_secret_key + ``` 3. Store the Kyber public key in the prekey bundle, private key in the database **Phase 3: X3DH extension** -Modify `perform_x3dh_sender` at `x3dh_manager.py:208-281`: +Modify `perform_x3dh_sender` at `frontend/src/crypto/x3dh.ts`: -1. After computing the classical `key_material` (line 252-255), check if the bundle includes a Kyber key +1. After computing the classical `key_material` , check if the bundle includes a Kyber key 2. If yes: encapsulate against the Kyber public key to get `(pq_shared_secret, pq_ciphertext)` 3. Concatenate: `combined_material = key_material + pq_shared_secret` 4. Change the HKDF info parameter: `info=b'PQXDH'` (to distinguish from classical sessions) 5. Include `pq_ciphertext` in the `X3DHResult` so it can be sent to the recipient -Modify `perform_x3dh_receiver` at `x3dh_manager.py:283-350`: +Modify `perform_x3dh_receiver` at `frontend/src/crypto/x3dh.ts`: 1. After computing the classical key material, check if a PQ ciphertext was provided 2. If yes: decapsulate to recover `pq_shared_secret` @@ -668,7 +667,7 @@ Modify `perform_x3dh_receiver` at `x3dh_manager.py:283-350`: **Phase 4: Backwards compatibility** -If the peer does not have a Kyber public key in their prekey bundle (they have not upgraded), fall back to classical-only X3DH. The code path at `x3dh_manager.py:246-255` already handles the optional one-time prekey with an if/else. Use the same pattern for Kyber. +If the peer does not have a Kyber public key in their prekey bundle (they have not upgraded), fall back to classical-only X3DH. The code path at `frontend/src/crypto/x3dh.ts` already handles the optional one-time prekey with an if/else. Use the same pattern for Kyber. **Key sizes to be aware of:** - X25519 public key: 32 bytes @@ -702,43 +701,43 @@ This challenge implements true cryptographic deletion: the ciphertext remains in **Where to start reading:** -- `backend/app/core/encryption/double_ratchet.py:96-109` is `_kdf_ck` which derives message keys. Each message gets a unique key. -- `backend/app/core/encryption/double_ratchet.py:215-244` is `_store_skipped_message_keys` with eviction logic at line 232-234. This is the pattern you will extend for TTL-based eviction. -- `backend/app/core/encryption/double_ratchet.py:246-258` is `_evict_oldest_skipped_keys` which deletes keys. -- `frontend/src/crypto/key-store.ts:353-358` stores ratchet state in IndexedDB -- `frontend/src/crypto/primitives.ts:294-298` has `generateRandomBytes` using `crypto.getRandomValues` +- `frontend/src/crypto/double-ratchet.ts` is `_kdf_ck` which derives message keys. Each message gets a unique key. +- `frontend/src/crypto/double-ratchet.ts` is `_store_skipped_message_keys` with eviction logic -234. This is the pattern you will extend for TTL-based eviction. +- `frontend/src/crypto/double-ratchet.ts` is `_evict_oldest_skipped_keys` which deletes keys. +- `frontend/src/crypto/key-store.ts` stores ratchet state in IndexedDB +- `frontend/src/crypto/frontend/src/crypto/primitives.ts` has `generateRandomBytes` using `crypto.getRandomValues` **Implementation approach:** 1. **Conversation setting:** Add a per-conversation setting: `disappear_after_seconds`. Values: 0 (disabled), 30, 300, 3600, 86400, 604800. Store this in the room metadata in SurrealDB. 2. **Message key storage with TTL:** When a message is encrypted, the message key is normally discarded after encryption. For disappearing messages, the RECIPIENT stores the message key in IndexedDB with a TTL: - ``` - { - message_id: "msg_abc123", - message_key: <32 bytes>, - created_at: "2025-01-15T14:30:00Z", - expires_at: "2025-01-15T15:30:00Z" // created_at + disappear_after_seconds - } - ``` + ``` + { + message_id: "msg_abc123", + message_key: <32 bytes>, + created_at: "2025-01-15T14:30:00Z", + expires_at: "2025-01-15T15:30:00Z" // created_at + disappear_after_seconds + } + ``` 3. **Timer starts on read, not on send.** The recipient's timer starts when they first decrypt the message. If the recipient is offline for a week, they still get the full timer duration after reading. 4. **Key destruction:** When the timer expires: - - Frontend: overwrite the key material with random bytes before deleting from IndexedDB. In JavaScript: `crypto.getRandomValues(keyBuffer)` then delete. - - Backend: if you store message keys server-side (for the deprecated server-side encryption path), overwrite with `os.urandom(32)` then delete. + - Frontend: overwrite the key material with random bytes before deleting from IndexedDB. In JavaScript: `crypto.getRandomValues(keyBuffer)` then delete. + - Backend: just delete the SurrealDB row. The server holds no keys, so there's nothing to wipe — but the recipient's locally-cached plaintext (and message key, if it's still in the skipped-key cache) needs to be overwritten on the client. 5. **UI changes:** - - Show a countdown timer on disappearing messages - - After expiration, replace the message content with "[Message expired]" - - The ciphertext remains in SurrealDB but is now permanently unreadable + - Show a countdown timer on disappearing messages + - After expiration, replace the message content with "[Message expired]" + - The ciphertext remains in SurrealDB but is now permanently unreadable **The hard parts:** - What happens if the recipient never reads the message? The sender's copy should still eventually disappear. Implement a "maximum lifetime" that triggers regardless of read status. - What about screenshots? You cannot prevent them. This is a social problem, not a technical one. Signal shows a notification when a screenshot is taken (on mobile), but this is best-effort. - JavaScript garbage collection does not guarantee memory is wiped. When you overwrite a `Uint8Array`, the old values may still exist in memory until the GC reclaims the page. True secure deletion in JavaScript is not possible. Document this limitation. -- The skipped message key cache (`double_ratchet.py:47-49`) stores keys for out-of-order messages. These must also respect the TTL. +- The skipped message key cache (`frontend/src/crypto/double-ratchet.ts`) stores keys for out-of-order messages. These must also respect the TTL. **How to test:** - Enable disappearing messages (30 second timer) in a conversation @@ -759,9 +758,9 @@ This challenge implements true cryptographic deletion: the ciphertext remains in **What to build:** Implement deniable authentication so that neither Alice nor Bob can prove to a third party that the other sent a specific message. This is a privacy property of the Signal Protocol that most implementations skip or get wrong. -**Prerequisites:** Solid understanding of the X3DH implementation at `x3dh_manager.py:56-353` +**Prerequisites:** Solid understanding of the X3DH implementation at `frontend/src/crypto/x3dh.ts` -**Why it matters:** In the current implementation, if Alice has Bob's signed messages (ciphertext + authentication tags), she could potentially show them to a third party to prove Bob said something. This is because the associated data (`x3dh_manager.py:272`) binds both identity keys to the session: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes`. +**Why it matters:** In the current implementation, if Alice has Bob's signed messages (ciphertext + authentication tags), she could potentially show them to a third party to prove Bob said something. This is because the associated data (`frontend/src/crypto/x3dh.ts`) binds both identity keys to the session: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes`. Deniable authentication means that Alice could have forged the messages herself (because she also has the shared secret), so they are not valid cryptographic proof of Bob's authorship. This matters for journalists, activists, whistleblowers, and anyone who might be legally compelled to prove message authorship. @@ -773,9 +772,9 @@ In a deniable protocol, the authentication MAC can be computed by EITHER party, **Where to start reading:** -- `x3dh_manager.py:272` is the associated data computation: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` -- `double_ratchet.py:323-362` is `encrypt_message` where associated data is used in AES-GCM -- `double_ratchet.py:364-416` is `decrypt_message` which verifies the authentication tag +- `frontend/src/crypto/x3dh.ts` is the associated data computation: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` +- `frontend/src/crypto/double-ratchet.ts` is `encrypt_message` where associated data is used in AES-GCM +- `frontend/src/crypto/double-ratchet.ts` is `decrypt_message` which verifies the authentication tag **Implementation approach:** @@ -783,7 +782,7 @@ In a deniable protocol, the authentication MAC can be computed by EITHER party, 2. Alternatively, implement a "triple DH" variant where you add a third DH: `DH5 = IK_A x IK_B`. This creates a shared secret that only requires Alice and Bob's identity keys. Since both parties can compute it, a transcript of messages authenticated with this secret is not proof against either party. -3. The associated data modification at `x3dh_manager.py:272` and the corresponding line at `x3dh_manager.py:341` must change in tandem (sender and receiver must produce the same associated data). +3. The associated data modification at `frontend/src/crypto/x3dh.ts` and the corresponding line at `frontend/src/crypto/x3dh.ts` must change in tandem (sender and receiver must produce the same associated data). **How to verify deniability:** @@ -810,31 +809,31 @@ WhatsApp solved this with Google Drive/iCloud backups encrypted with a user pass **Planning questions you must answer before writing code:** 1. What key material needs to be backed up? - - Identity keys (X25519 + Ed25519 private keys) - - Active ratchet states for each conversation - - Signed prekey private keys - - Message history (optional, large) + - Identity keys (X25519 + Ed25519 private keys) + - Active ratchet states for each conversation + - Signed prekey private keys + - Message history (optional, large) 2. How is the backup encrypted? - - Option A: User PIN/password -> Argon2id -> AES-256-GCM key - - Option B: Random backup key displayed as a 24-word recovery phrase - - Option C: Server-side HSM with rate limiting (Signal's SVR approach, requires hardware) + - Option A: User PIN/password -> Argon2id -> AES-256-GCM key + - Option B: Random backup key displayed as a 24-word recovery phrase + - Option C: Server-side HSM with rate limiting (Signal's SVR approach, requires hardware) 3. Where is the backup stored? - - Server-side (encrypted blob in PostgreSQL or object storage) - - User-exported file (like Signal's plaintext export, but encrypted) + - Server-side (encrypted blob in PostgreSQL or object storage) + - User-exported file (like Signal's plaintext export, but encrypted) 4. How do you handle backup key loss? - - Unrecoverable by design (most secure) - - Social recovery (N-of-M trusted contacts must approve, like Shamir's Secret Sharing) - - Server-side recovery with identity verification (weakest, requires trusting the server) + - Unrecoverable by design (most secure) + - Social recovery (N-of-M trusted contacts must approve, like Shamir's Secret Sharing) + - Server-side recovery with identity verification (weakest, requires trusting the server) **Where to start reading:** -- `frontend/src/crypto/key-store.ts:85-99` saves identity keys to IndexedDB. This is the data that needs backing up. -- `frontend/src/crypto/key-store.ts:353-358` saves ratchet states -- `frontend/src/crypto/key-store.ts:392-412` `clearAllKeys()` deletes everything on logout -- `frontend/src/crypto/message-store.ts:61-67` saves decrypted messages +- `frontend/src/crypto/key-store.ts` saves identity keys to IndexedDB. This is the data that needs backing up. +- `frontend/src/crypto/key-store.ts` saves ratchet states +- `frontend/src/crypto/key-store.ts` `clearAllKeys` deletes everything on logout +- `frontend/src/crypto/message-store.ts` saves decrypted messages **Implementation phases:** @@ -843,28 +842,28 @@ WhatsApp solved this with Google Drive/iCloud backups encrypted with a user pass Define a JSON schema for the backup: ```json { - "version": 1, - "created_at": "2025-01-15T14:30:00Z", - "identity_keys": { - "x25519_private": "", - "x25519_public": "", - "ed25519_private": "", - "ed25519_public": "" - }, - "ratchet_states": [ - { - "peer_id": "bob-uuid", - "root_key": "", - "sending_chain_key": "", - "receiving_chain_key": "", - "dh_private_key": "", - "dh_peer_public_key": "", - "sending_message_number": 42, - "receiving_message_number": 37, - "previous_sending_chain_length": 12 - } - ], - "messages": [] + "version": 1, + "created_at": "2025-01-15T14:30:00Z", + "identity_keys": { + "x25519_private": "", + "x25519_public": "", + "ed25519_private": "", + "ed25519_public": "" + }, + "ratchet_states": [ + { + "peer_id": "bob-uuid", + "root_key": "", + "sending_chain_key": "", + "receiving_chain_key": "", + "dh_private_key": "", + "dh_peer_public_key": "", + "sending_message_number": 42, + "receiving_message_number": 37, + "previous_sending_chain_length": 12 + } + ], + "messages": [] } ``` @@ -935,7 +934,7 @@ Build a corporate messaging system with compliance features (backup and recovery ### Optimize Ratchet State Storage -The current implementation serializes the entire ratchet state to PostgreSQL on every single message. Look at `message_service.py:208-267` (`_save_ratchet_state_to_db`). Every call to `send_encrypted_message` or `decrypt_received_message` triggers a full ratchet state write with `await session.commit()` at line 258. +The current implementation serializes the entire ratchet state to PostgreSQL on every single message. Look at `backend/app/services/message_service.py` (`_save_ratchet_state_to_db`). Every call to `send_encrypted_message` or `decrypt_received_message` triggers a full ratchet state write with `await session.commit` . This is a bottleneck. For a conversation with rapid back-and-forth messaging, you are writing to PostgreSQL twice per message (once for encrypt, once for decrypt). @@ -945,7 +944,7 @@ This is a bottleneck. For a conversation with rapid back-and-forth messaging, yo 1. On first ratchet state load, cache it in Redis with a key like `ratchet:{user_id}:{peer_user_id}` 2. After each ratchet advance, write to Redis only (sub-millisecond) 3. Periodically (every 10 messages or every 30 seconds) flush to PostgreSQL (durable storage) -4. On WebSocket disconnect (`websocket_manager.py:97-133`), flush all cached states to PostgreSQL +4. On WebSocket disconnect (`websocket_manager.py`), flush all cached states to PostgreSQL 5. On startup, load from PostgreSQL (Redis is not durable across restarts) **Measurement:** Benchmark messages per second before and after with a simple load test. Target: greater than 100 messages per second per conversation. @@ -955,27 +954,27 @@ This is a bottleneck. For a conversation with rapid back-and-forth messaging, yo **Detailed profiling steps:** 1. Add timing instrumentation around `_save_ratchet_state_to_db`: - ```python - import time - start = time.perf_counter() - await session.commit() - elapsed_ms = (time.perf_counter() - start) * 1000 - logger.info("Ratchet state save: %.2fms", elapsed_ms) - ``` + ```python + import time + start = time.perf_counter + await session.commit + elapsed_ms = (time.perf_counter - start) * 1000 + logger.info("Ratchet state save: %.2fms", elapsed_ms) + ``` 2. Run a load test: two users exchanging 1000 messages as fast as possible. Measure: - - Average time per `_save_ratchet_state_to_db` call - - Average time per `_load_ratchet_state_from_db` call - - Total messages per second (end-to-end) - - PostgreSQL connection pool utilization (`DB_POOL_SIZE` at `config.py:118`) + - Average time per `_save_ratchet_state_to_db` call + - Average time per `_load_ratchet_state_from_db` call + - Total messages per second (end-to-end) + - PostgreSQL connection pool utilization (`DB_POOL_SIZE` at `config.py`) 3. Implement Redis caching, repeat the same load test, compare. **Redis cache schema:** ``` -Key: ratchet:{user_id}:{peer_user_id} +Key: ratchet:{user_id}:{peer_user_id} Value: JSON serialized ratchet state -TTL: 300 seconds (auto-expire as safety net) +TTL: 300 seconds (auto-expire as safety net) ``` On each ratchet advance, write to Redis. Every 10 messages OR every 30 seconds, flush to PostgreSQL. On disconnect, flush immediately. @@ -990,13 +989,13 @@ Use `locust` or `k6` to load test the WebSocket layer. Determine how many concur 1. Maximum concurrent WebSocket connections before memory exhaustion 2. Message throughput: messages per second at 100, 500, 1000, 5000 concurrent connections 3. Latency: p50, p95, p99 message delivery latency at various connection counts -4. Identify the bottleneck: Is it CPU (from encryption operations)? Memory (from connection pool at `websocket_manager.py:39`)? I/O (from SurrealDB writes)? Python GIL contention? +4. Identify the bottleneck: Is it CPU (from encryption operations)? Memory (from connection pool at `websocket_manager.py`)? I/O (from SurrealDB writes)? Python GIL contention? **Target:** Document the breaking point and propose a horizontal scaling strategy. **Scaling considerations:** -The current architecture has a single `ConnectionManager` instance at `websocket_manager.py:296` holding all WebSocket connections in memory (`self.active_connections: dict[UUID, list[WebSocket]]`). This does not scale horizontally because connections on Server A are invisible to Server B. +The current architecture has a single `ConnectionManager` instance at `websocket_manager.py` holding all WebSocket connections in memory (`self.active_connections: dict[UUID, list[WebSocket]]`). This does not scale horizontally because connections on Server A are invisible to Server B. To scale to multiple server instances: 1. Use Redis Pub/Sub for cross-instance message routing. When Server A needs to send a message to a user connected to Server B, it publishes to a Redis channel. Server B subscribes and forwards to the local WebSocket. @@ -1007,22 +1006,25 @@ To scale to multiple server instances: **Load test script outline (k6):** ```javascript import ws from 'k6/ws'; -export default function () { - const url = 'ws://localhost:8000/ws?user_id=...'; - const res = ws.connect(url, {}, function (socket) { - socket.on('open', () => { - socket.send(JSON.stringify({ - type: 'encrypted_message', - recipient_id: '...', - room_id: '...', - ciphertext: '...', - nonce: '...', - header: '...' - })); - }); - socket.on('message', (msg) => {}); - socket.setTimeout(() => socket.close(), 30000); - }); +export default function { + // The server expects a session cookie; load tests should sign in first + // and pass the cookie via params.headers.Cookie. + const url = 'ws://localhost:8000/ws'; + const params = { headers: { Cookie: 'chat_session=' } }; + const res = ws.connect(url, params, function (socket) { + socket.on('open', => { + socket.send(JSON.stringify({ + type: 'encrypted_message', + recipient_id: '...', + room_id: '...', + ciphertext: '...', + nonce: '...', + header: '...' + })); + }); + socket.on('message', (msg) => {}); + socket.setTimeout( => socket.close, 30000); + }); } ``` @@ -1038,7 +1040,7 @@ Run with: `k6 run --vus 100 --duration 60s load_test.js` Build a system where identity key changes are logged to a verifiable append-only log, similar to Certificate Transparency for TLS certificates. This prevents the server from silently swapping identity keys to perform a man-in-the-middle attack. -Currently, the server at `prekey_service.py:293-361` serves prekey bundles. A compromised server could serve a fake identity key (its own), perform X3DH with both parties, and relay decrypted messages. The users would not know. +Currently, the server at `backend/app/services/prekey_service.py` serves prekey bundles. A compromised server could serve a fake identity key (its own), perform X3DH with both parties, and relay decrypted messages. The users would not know. A key transparency log makes this detectable: every identity key is logged, and clients can audit the log to verify their key has not been replaced. @@ -1062,13 +1064,13 @@ Perform a manual security audit of the X3DH and Double Ratchet implementations. **Audit checklist:** -1. **HKDF parameter usage** (`x3dh_manager.py:258-264`): Is the salt correct? Is the info string distinct between different uses? Are the output lengths appropriate? +1. **HKDF parameter usage** (`frontend/src/crypto/x3dh.ts`): Is the salt correct? Is the info string distinct between different uses? Are the output lengths appropriate? -2. **Nonce generation** (`double_ratchet.py:122`): Is `os.urandom(AES_GCM_NONCE_SIZE)` called for every encryption? Is there any risk of nonce reuse? +2. **Nonce generation** (`frontend/src/crypto/double-ratchet.ts`): Is `os.urandom(AES_GCM_NONCE_SIZE)` called for every encryption? Is there any risk of nonce reuse? -3. **WebCrypto API usage** (`primitives.ts:224-259`): Is the `iv` parameter always fresh? Are `additionalData` bindings correct? Are key usages properly restricted? +3. **WebCrypto API usage** (`frontend/src/crypto/primitives.ts`): Is the `iv` parameter always fresh? Are `additionalData` bindings correct? Are key usages properly restricted? -4. **Timing side channels**: Look at `primitives.ts:388-397` (`constantTimeEqual`). Is this actually constant time in JavaScript? (Spoiler: the `if (a.length !== b.length) return false` on line 389 is an early return that leaks length information. Is this a problem in context?) +4. **Timing side channels**: Look at `frontend/src/crypto/primitives.ts` (`constantTimeEqual`). Is this actually constant time in JavaScript? (Spoiler: the `if (a.length !== b.length) return false` is an early return that leaks length information. Is this a problem in context?) 5. **Random number generation**: Verify that all randomness comes from `os.urandom` (Python) or `crypto.getRandomValues` (JavaScript). Search for any use of `random` or `Math.random`. @@ -1082,7 +1084,7 @@ Write a report documenting your findings. Build a system where the server does not know who sent a message, only who it is for. Signal implemented this as "Sealed Sender." -Currently, the WebSocket endpoint at `websocket.py:25-84` receives messages from authenticated users. The server knows `sender_id` (from the WebSocket connection) and `recipient_id` (from the message payload). It stores both in SurrealDB (`message_service.py:292-302`). +Currently, the WebSocket endpoint at `websocket.py` receives messages from authenticated users. The server knows `sender_id` (from the WebSocket connection) and `recipient_id` (from the message payload). It stores both in SurrealDB (`backend/app/services/message_service.py`). With Sealed Sender, the sender identity is encrypted inside the E2E encrypted payload. The server only sees the recipient (needed for routing). The sender is revealed only after the recipient decrypts. @@ -1097,22 +1099,22 @@ This requires the server to have its own keypair (separate from any user), and c 1. Generate a server X25519 keypair on startup. Publish the server's public key at a well-known endpoint: `GET /api/.well-known/server-key`. 2. Modify the client message sending flow: - - Construct the inner payload: the normal E2E encrypted message (ciphertext, nonce, header) PLUS `sender_id` in plaintext - - Encrypt the inner payload with the intended recipient's ratchet (as currently done) - - Construct the outer envelope: `{recipient_id: "bob-uuid", sealed_payload: ""}` - - Encrypt the outer envelope with the server's public key (simple X25519 + AES-GCM) - - Send only the outer ciphertext over the WebSocket + - Construct the inner payload: the normal E2E encrypted message (ciphertext, nonce, header) PLUS `sender_id` in plaintext + - Encrypt the inner payload with the intended recipient's ratchet (as currently done) + - Construct the outer envelope: `{recipient_id: "bob-uuid", sealed_payload: ""}` + - Encrypt the outer envelope with the server's public key (simple X25519 + AES-GCM) + - Send only the outer ciphertext over the WebSocket 3. Modify the server routing: - - Decrypt the outer envelope using the server's private key - - Read `recipient_id` from the decrypted envelope - - Forward `sealed_payload` to the recipient - - The server never sees `sender_id` + - Decrypt the outer envelope using the server's private key + - Read `recipient_id` from the decrypted envelope + - Forward `sealed_payload` to the recipient + - The server never sees `sender_id` 4. Modify the recipient decryption: - - Decrypt the sealed payload using the E2E ratchet - - Extract `sender_id` from the decrypted content - - Display the message with the correct sender attribution + - Decrypt the sealed payload using the E2E ratchet + - Extract `sender_id` from the decrypted content + - Display the message with the correct sender attribution **Limitation:** The server still knows the recipient (it needs to for routing). And it knows the sender's IP address and WebSocket connection. True sender anonymity requires additional measures like Tor or mix networks. Sealed Sender hides sender identity from the server's message logs, not from network-level observation. @@ -1133,17 +1135,17 @@ Use Hypothesis (Python) to write property-based tests for the cryptographic impl **Properties to verify:** -1. **X3DH symmetry:** For any two valid identity key pairs and any valid prekey bundle, `perform_x3dh_sender` and `perform_x3dh_receiver` produce the same `shared_key`. Reference: `x3dh_manager.py:208-281` and `x3dh_manager.py:283-350`. +1. **X3DH symmetry:** For any two valid identity key pairs and any valid prekey bundle, `perform_x3dh_sender` and `perform_x3dh_receiver` produce the same `shared_key`. Reference: `frontend/src/crypto/x3dh.ts` and `frontend/src/crypto/x3dh.ts`. -2. **Double Ratchet round-trip:** For any sequence of plaintext messages in any order, encrypting with `encrypt_message` and decrypting with `decrypt_message` always returns the original plaintext. Reference: `double_ratchet.py:323-362` and `double_ratchet.py:364-416`. +2. **Double Ratchet round-trip:** For any sequence of plaintext messages in any order, encrypting with `encrypt_message` and decrypting with `decrypt_message` always returns the original plaintext. Reference: `frontend/src/crypto/double-ratchet.ts` and `frontend/src/crypto/double-ratchet.ts`. -3. **AES-GCM round-trip:** For any plaintext and any valid key, `aesGcmEncrypt` followed by `aesGcmDecrypt` returns the original plaintext. Reference: `primitives.ts:224-292`. +3. **AES-GCM round-trip:** For any plaintext and any valid key, `aesGcmEncrypt` followed by `aesGcmDecrypt` returns the original plaintext. Reference: `frontend/src/crypto/primitives.ts`. 4. **No nonce reuse:** For N encryptions with the same key, all N nonces are distinct. (This should hold with overwhelming probability for random 12-byte nonces, but verify it empirically for N=10000.) 5. **Forward secrecy:** After advancing the ratchet, old message keys cannot be derived from the new state. Generate a ratchet state, encrypt a message, advance the ratchet 100 steps, then verify that the message key from step 0 cannot be recomputed from the state at step 100. -6. **Out-of-order decryption:** For any permutation of N messages, decrypting them in that permuted order produces the same set of plaintexts as decrypting in the original order. This tests the skipped message key logic at `double_ratchet.py:215-244`. +6. **Out-of-order decryption:** For any permutation of N messages, decrypting them in that permuted order produces the same set of plaintexts as decrypting in the original order. This tests the skipped message key logic at `frontend/src/crypto/double-ratchet.ts`. **Hypothesis strategy example:** @@ -1151,18 +1153,18 @@ Use Hypothesis (Python) to write property-based tests for the cryptographic impl from hypothesis import given, strategies as st @given( - message_count=st.integers(min_value=1, max_value=50), - delivery_order=st.permutations(range(50)) # random permutation + message_count=st.integers(min_value=1, max_value=50), + delivery_order=st.permutations(range(50)) # random permutation ) def test_out_of_order_delivery(message_count, delivery_order): - """All messages decrypt correctly regardless of delivery order.""" - # Generate N messages - # Encrypt them in order (advancing the sending ratchet) - # Decrypt them in the shuffled order - # Assert all plaintexts match the originals + """All messages decrypt correctly regardless of delivery order.""" + # Generate N messages + # Encrypt them in order (advancing the sending ratchet) + # Decrypt them in the shuffled order + # Assert all plaintexts match the originals ``` -**Where to put the tests:** Create a `tests/` directory in the backend and structure tests as `tests/test_x3dh.py`, `tests/test_double_ratchet.py`, `tests/test_crypto_properties.py`. +**Where to put the tests:** The crypto property tests belong in the frontend Vitest suite next to `x3dh.test.ts` and `double-ratchet.test.ts`. Add a new file like `frontend/src/crypto/properties.test.ts` for randomized round-trip and tamper checks. Reserve `backend/tests/` for backend integration tests (auth, prekey storage, session enforcement). ---