diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md new file mode 100644 index 00000000..d416b275 --- /dev/null +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/00-OVERVIEW.md @@ -0,0 +1,428 @@ +# Encrypted P2P Chat + +## What This Is + +This is a production-grade end-to-end encrypted peer-to-peer messaging application built on the Signal Protocol. It implements X3DH (Extended Triple Diffie-Hellman) for asynchronous key exchange, the Double Ratchet algorithm for forward-secret message encryption, and WebAuthn/Passkeys for phishing-resistant authentication. The server orchestrates message delivery and stores encrypted blobs, but it never sees plaintext messages or private keys. If someone compromises the server, they get ciphertext and public keys. That is it. + +## Why This Matters + +On January 22, 2020, the United Nations confirmed that Jeff Bezos's phone had been compromised through a WhatsApp message. The forensic analysis by FTI Consulting traced the infection to a video file sent from the personal WhatsApp account of Saudi Crown Prince Mohammed bin Salman. Within hours of receiving that message, data exfiltration from the phone increased by approximately 29,000 percent. The attackers used a zero-day exploit in the WhatsApp video rendering stack to deliver spyware, likely NSO Group's Pegasus. What made this attack especially instructive is what it was not: it was not a failure of end-to-end encryption. The E2E encryption in WhatsApp (which uses the Signal Protocol) worked correctly. The message content was encrypted in transit. But the exploit happened on the device itself, after decryption. This distinction matters because it shows exactly where E2E encryption draws its line. It protects data in transit and at rest on the server. It does not protect against compromised endpoints. Understanding this boundary is critical to building and reasoning about encrypted messaging systems. + +Facebook Messenger, by contrast, showed what happens when encryption is not the default. In September 2019, a server containing phone numbers linked to hundreds of millions of Facebook accounts was found exposed online without password protection. The dataset included 419 million records spanning users in the US, UK, and Vietnam. While this particular leak was about phone numbers rather than message content, the core problem was structural: Facebook Messenger did not enable end-to-end encryption by default. "Secret Conversations" existed as an opt-in feature, but the vast majority of messages flowed through Facebook's servers in a form the company could read. This meant that any server compromise, government subpoena, or rogue employee could access message content directly. Facebook did not roll out default E2E encryption for Messenger until late 2023. For years, billions of messages were protected only by transport-layer encryption (TLS), which encrypts data between the client and the server but leaves the server itself as a trusted party with full access to plaintext. + +The 2020 Zoom encryption controversy demonstrated how the gap between marketing claims and cryptographic reality can erode user trust. Zoom's marketing materials and security white paper stated that the platform supported "end-to-end encryption." The Citizen Lab at the University of Toronto investigated and found that Zoom was actually using TLS for transport encryption, with AES-128 keys generated by Zoom's own servers. This meant Zoom had the ability to decrypt any meeting. The keys were sometimes routed through servers in China, even for meetings where all participants were in North America. Zoom eventually settled an FTC complaint and committed to implementing actual E2E encryption, but the incident highlighted a critical point: the phrase "end-to-end encrypted" is meaningless without verifiable implementation. In a real E2E system, key generation happens on client devices, the server relays ciphertext, and at no point does the server have access to decryption keys. That is exactly what this project implements. + +The Cambridge Analytica scandal of 2018, while primarily about data harvesting through Facebook's Graph API rather than message interception, drives home a broader principle: any data a server can access, a server can leak, sell, or be compelled to hand over. Cambridge Analytica harvested personal data from approximately 87 million Facebook profiles through a personality quiz app that exploited Facebook's permissive data-sharing policies. The data was used for political advertising targeting during the 2016 US presidential election and the Brexit referendum. The lesson for messaging applications is straightforward. If your server can read messages, your server is a liability. Every employee with database access, every government with a court order, every attacker who breaches your infrastructure becomes a threat to user privacy. A zero-knowledge server design, where the server handles routing and storage of ciphertext but never possesses decryption keys, eliminates this entire category of risk. That is the architecture this project uses. + +The Signal Protocol is now the gold standard for encrypted messaging. Signal (the application) pioneered it. WhatsApp adopted it in 2016, bringing end-to-end encryption to over 2 billion users. Google Messages uses it for RCS chats. Meta's Messenger migration to default E2E encryption in December 2023 also drew on Signal Protocol concepts. The protocol's core innovations, X3DH for asynchronous key establishment and the Double Ratchet for per-message forward secrecy, solve problems that earlier encrypted messaging systems (like PGP-encrypted email) left open for decades. This project implements both protocols from scratch, giving you direct experience with the same cryptographic machinery that protects billions of real conversations. And by pairing it with WebAuthn/Passkey authentication, it eliminates the number one attack vector for account compromise: password phishing. Phishing accounts for approximately 36% of all data breaches according to the 2023 Verizon Data Breach Investigations Report. WebAuthn makes phishing structurally impossible because the authenticator is bound to the origin domain. A fake login page on a different domain simply cannot trigger the correct credential. + +## What You'll Learn + +### Cryptographic Protocols + +- **X3DH key exchange for asynchronous session establishment.** The fundamental problem with Diffie-Hellman key exchange is that both parties need to be online at the same time. If Alice wants to send a message to Bob, but Bob is offline, classic DH cannot work. X3DH (Extended Triple Diffie-Hellman) solves this by having Bob pre-publish a bundle of public keys to the server: a long-term identity key, a medium-term signed prekey, and a set of single-use one-time prekeys. When Alice wants to start a conversation, she downloads Bob's bundle and performs three (or four, if a one-time prekey is available) DH computations to derive a shared secret. Bob can complete his half of the exchange when he comes back online. This project implements X3DH in `backend/app/core/encryption/x3dh_manager.py` on the server side and `frontend/src/crypto/x3dh.ts` on the client side, using X25519 for Diffie-Hellman and Ed25519 for prekey signatures. + +- **Double Ratchet algorithm for forward secrecy and post-compromise security.** Once X3DH establishes an initial shared secret, the Double Ratchet takes over for ongoing message encryption. It combines two ratcheting mechanisms: a symmetric-key ratchet (using HMAC-based key derivation) that advances with every message, and a Diffie-Hellman ratchet that advances every time the conversation "turn" changes. The symmetric ratchet provides forward secrecy: if an attacker steals your current keys, they cannot decrypt past messages because each message key is derived from and then discarded after the previous chain key. The DH ratchet provides post-compromise security (also called "future secrecy" or "self-healing"): even if an attacker compromises your current state completely, once you send a new message with a fresh DH key pair, the attacker loses access because they do not know the new private key. The implementation lives in `backend/app/core/encryption/double_ratchet.py` and `frontend/src/crypto/double-ratchet.ts`. + +- **AES-256-GCM authenticated encryption.** Each message is encrypted using AES-256-GCM (Galois/Counter Mode), which provides both confidentiality and integrity. GCM is an AEAD (Authenticated Encryption with Associated Data) cipher, meaning it produces a ciphertext and an authentication tag. If anyone tampers with the ciphertext, the tag verification fails and decryption is rejected. The "associated data" in this project is the concatenation of both users' identity public keys, binding each encrypted message to the specific conversation participants. The implementation uses 12-byte random nonces and 32-byte keys derived from the Double Ratchet chain. You will see this in the `_encrypt_with_message_key` and `_decrypt_with_message_key` methods of the `DoubleRatchet` class, and in the `aesGcmEncrypt`/`aesGcmDecrypt` functions in `frontend/src/crypto/primitives.ts`. + +- **HKDF-SHA256 key derivation.** Raw Diffie-Hellman outputs are not suitable for direct use as encryption keys. They have non-uniform distribution and may contain structural biases. HKDF (HMAC-based Key Derivation Function) extracts entropy from DH outputs and expands it into keys of the required length. This project uses HKDF-SHA256 throughout: in X3DH to derive the initial shared secret from concatenated DH outputs, and in the Double Ratchet to derive new root keys and chain keys from each DH ratchet step. The X3DH derivation uses a salt of all-zero bytes and an info string of `b'X3DH'`, prepending a block of `0xFF` bytes as specified by the Signal Protocol. You will see this in both the Python `cryptography` library's HKDF and the browser's WebCrypto `deriveBits` API with HKDF. + +- **Ed25519 digital signatures and X25519 Diffie-Hellman.** The project uses two related but distinct elliptic curve operations on Curve25519. X25519 is used for Diffie-Hellman key exchange: two parties each generate a key pair, exchange public keys, and compute a shared secret. Ed25519 is used for digital signatures: the identity key owner signs their signed prekey so that other users can verify it was genuinely published by that identity, not substituted by a malicious server. The distinction matters because the mathematical operations are different (X25519 operates on the Montgomery form of the curve, Ed25519 on the Edwards form), but they share the same underlying curve. This project maintains separate X25519 and Ed25519 key pairs for each user identity, as you can see in the `generate_identity_keypair_x25519` and `generate_identity_keypair_ed25519` methods. + +### Authentication and Identity + +- **WebAuthn/FIDO2 passkey registration and authentication.** WebAuthn is a W3C standard that allows users to authenticate using public key cryptography instead of passwords. During registration, the browser creates a new key pair on a hardware or platform authenticator (a YubiKey, a fingerprint reader, a phone's secure enclave). The public key is sent to the server. The private key never leaves the authenticator. During login, the server sends a random challenge, the authenticator signs it with the private key, and the server verifies the signature against the stored public key. No shared secret ever crosses the network. The implementation uses the `py_webauthn` library on the server side (`backend/app/core/passkey/passkey_manager.py`) and the browser's native `navigator.credentials` API on the client. Registration options include resident key requirement (enabling usernameless login) and user verification preference. + +- **Public key cryptography for identity verification.** Each user in the system has two types of public keys: WebAuthn credentials (for authentication with the server) and Signal Protocol identity keys (for end-to-end encryption with other users). The WebAuthn credential proves to the server that you are who you claim to be. The Signal Protocol identity key lets other users verify your encryption keys. These are separate key pairs serving separate purposes. The identity key is an Ed25519 key pair that signs the user's medium-term signed prekeys, letting recipients verify that the prekey bundle they downloaded was genuinely published by that user and not substituted by a compromised server. + +- **Authenticator clone detection via signature counters.** Every WebAuthn authenticator maintains a signature counter that increments with each authentication ceremony. The server stores the last-seen counter value. If the counter ever goes backward or stays the same (when both the server and authenticator track non-zero counters), it indicates that the authenticator may have been cloned. This is a known FIDO2 security mechanism. The `verify_authentication` method in `passkey_manager.py` explicitly checks for this: if `new_sign_count <= credential_current_sign_count`, it raises a `ValueError` with a message about potential cloned authenticator detection. This catches scenarios where an attacker has physically duplicated a hardware security key. + +### Fullstack Architecture + +- **FastAPI async backend with SQLModel ORM.** The backend is built with FastAPI, using its native async support with `asyncio` and `async/await` throughout. Database models use SQLModel, which merges SQLAlchemy's ORM capabilities with Pydantic's data validation. The app factory pattern in `factory.py` creates the FastAPI instance with a lifespan context manager that handles startup (initializing PostgreSQL, connecting to Redis, connecting to SurrealDB) and shutdown (disconnecting cleanly). API routes are organized into four routers: `auth` (WebAuthn registration and login), `encryption` (prekey bundle management), `rooms` (chat room CRUD), and `websocket` (real-time message transport). The backend uses `ORJSONResponse` as the default response class for faster JSON serialization. + +- **SolidJS reactive frontend with TypeScript.** The frontend uses SolidJS, a reactive UI framework with fine-grained reactivity (no virtual DOM). State management uses Nanostores with the `@nanostores/solid` binding and `@nanostores/persistent` for localStorage persistence. Routing uses `@solidjs/router`. Server state management uses `@tanstack/solid-query` for data fetching, caching, and synchronization. The component structure is organized into domain folders: `Auth` (AuthCard, AuthForm, PasskeyButton), `Chat` (ChatInput, ChatHeader, ConversationList, MessageBubble, MessageList, EncryptionBadge, OnlineStatus, TypingIndicator, UserSearch), `Layout` (AppShell, Header, Sidebar, ProtectedRoute), and `UI` (Avatar, Badge, Button, Dropdown, Input, Modal, Spinner, Toast, Tooltip). Styling uses Tailwind CSS v4 via the Vite plugin. + +- **WebSocket real-time messaging.** Messages are delivered in real time over WebSocket connections. The backend's `websocket_manager.py` maintains a connection pool and handles broadcasting. The frontend's `websocket/websocket-manager.ts` manages connection lifecycle, reconnection, and heartbeats. Message types include encrypted messages, typing indicators, presence updates, read receipts, and error notifications. The WebSocket protocol carries ciphertext between clients; the server routes messages without decrypting them. Heartbeat interval is configurable (default 30 seconds), and each user can have up to 5 concurrent connections (for multiple devices or tabs). + +- **Multi-database architecture: PostgreSQL, SurrealDB, Redis.** The project uses three databases, each chosen for a specific access pattern. PostgreSQL (via `asyncpg` and SQLModel) stores authentication data, user profiles, WebAuthn credentials, and encryption key metadata (identity keys, signed prekeys, one-time prekeys, ratchet state). This data is relational, requires ACID transactions, and benefits from SQL's querying power. SurrealDB stores real-time chat data: messages, room membership, and presence information. Its live query feature enables real-time push notifications without polling. Redis stores ephemeral state: WebAuthn challenge tokens (with TTL-based expiration), rate limiting counters, and session caches. Each database has its own manager class (`Base.py` for PostgreSQL, `surreal_manager.py` for SurrealDB, `redis_manager.py` for Redis) handling connection lifecycle and operations. + +- **Docker containerization with Nginx reverse proxy.** The entire application runs in Docker containers orchestrated by Docker Compose. The development compose file (`dev.compose.yml`) defines six services: PostgreSQL 16 Alpine, SurrealDB (latest), Redis 8 Alpine, the FastAPI backend, the Vite dev server for the frontend, and Nginx Alpine as a reverse proxy. Each database has health checks to ensure the backend only starts after its dependencies are ready. The Nginx configuration routes HTTP traffic and proxies WebSocket connections. There are separate Dockerfiles for development (with hot-reload via volume mounts) and production (with optimized builds) in `conf/docker/dev/` and `conf/docker/prod/`. + +### Security Engineering + +- **Zero-knowledge server design.** The server in this application is designed so that it cannot read messages even if it wanted to. Private keys are generated on client devices and stored in the browser's IndexedDB. The server stores only public keys and encrypted message blobs. When Alice sends a message to Bob, the client-side crypto layer encrypts the plaintext using the Double Ratchet, sends the ciphertext over the WebSocket, and the server forwards it to Bob. At no point does the server possess any decryption key. The associated data binding (concatenation of both users' identity public keys) ensures that even if the server tried to relay a message to the wrong recipient, decryption would fail because the associated data would not match. This is a meaningful defense against a compromised server attempting message misdirection. + +- **Key rotation and lifecycle management.** Cryptographic keys in this system have defined lifecycles. One-time prekeys are single-use: once consumed during an X3DH exchange, they are deleted from the server. The system maintains a pool of 100 one-time prekeys and replenishes them when the count drops. Signed prekeys rotate every 48 hours, with a 7-day retention period for old keys to handle in-flight messages. Identity keys are long-term but can be rotated (configurable, default 90 days). Ratchet state is per-conversation and advances with every message. Skipped message keys (for out-of-order delivery) are cached up to a configurable limit (default 2000) with the oldest keys evicted first. These lifecycle parameters are centralized in `config.py` as named constants. + +- **Threat modeling for messaging applications.** Building this project exposes you to the specific threat model that the Signal Protocol is designed to address. The primary threats include: a compromised server attempting to read or modify messages (mitigated by E2E encryption and associated data binding), an attacker who steals a user's current session keys (mitigated by forward secrecy from the symmetric ratchet), an attacker who fully compromises a device and then loses access (mitigated by post-compromise security from the DH ratchet), a man-in-the-middle attempting to substitute public keys (mitigated by identity key signing and optional safety number verification), and an attacker who intercepts one-time prekeys (mitigated by the three base DH computations that provide security even without the one-time prekey). Understanding which threats each mechanism addresses is the core of cryptographic protocol design. + +- **Client-side cryptography with WebCrypto API and IndexedDB key storage.** All encryption and decryption happens in the browser using the WebCrypto API (`window.crypto.subtle`). The `primitives.ts` file wraps the WebCrypto API into functions for X25519 key generation and DH, Ed25519 signing and verification, HKDF key derivation, AES-256-GCM encryption/decryption, HMAC-SHA256, and utility functions like constant-time comparison. Private keys are stored in IndexedDB (`key-store.ts`) in a database called `encrypted-chat-keys` with separate object stores for identity keys, signed prekeys, one-time prekeys, and ratchet states. IndexedDB was chosen over localStorage because it can store structured data (including `CryptoKey` objects in browsers that support it) and has no size limits comparable to localStorage's typical 5-10MB cap. + +## Prerequisites + +### Required Knowledge + +- **Python async programming (asyncio, async/await).** The entire backend is async. Route handlers are `async def`, database queries use `await`, and the application lifespan is managed by an `asynccontextmanager`. You need to understand coroutines, event loops, and how `await` yields control. +- **TypeScript and reactive UI frameworks.** The frontend is written in TypeScript with SolidJS. You should be comfortable with TypeScript's type system (generics, discriminated unions, type guards) and the concept of reactive state (signals, effects, derived state). Prior experience with React or Vue is transferable. +- **Basic understanding of public key cryptography.** You need to know what a key pair is (private key stays secret, public key is shared), what signing and verification mean, and the general concept of Diffie-Hellman (two parties derive a shared secret from their public/private key pairs). You do not need to know the math behind elliptic curves. +- **Docker and Docker Compose.** The development environment runs six containers. You need to know how to read a compose file, run `docker compose up`, check logs, and debug container startup issues. +- **SQL and database fundamentals.** The PostgreSQL models use SQLModel (SQLAlchemy under the hood). You should be able to read an ORM model definition and understand concepts like foreign keys, indexes, and migrations. + +### Required Tools + +- **Docker and Docker Compose.** For running the multi-service development environment. Docker Desktop or Docker Engine with the Compose plugin. +- **Python 3.13+.** The project requires Python 3.13 or higher as specified in `pyproject.toml` with `requires-python = ">=3.13"`. The type annotations and language features used throughout the codebase depend on this version. +- **Node.js 20+.** For the SolidJS/Vite frontend build toolchain. +- **uv (Python package manager).** This project uses `uv` for Python dependency management. Never use pip directly. Install with `curl -LsSf https://astral.sh/uv/install.sh | sh` or `brew install uv`. +- **pnpm (Node package manager).** This project uses `pnpm` for Node dependency management. Never use npm directly. Install with `corepack enable && corepack prepare pnpm@latest --activate` or `npm install -g pnpm`. +- **just (command runner).** The project uses a `justfile` for build automation, linting, testing, database migrations, and Docker orchestration. Install with `cargo install just`, `brew install just`, or your system's package manager. + +### Helpful But Optional + +- **Prior exposure to WebSocket programming.** The real-time messaging layer uses WebSockets. Familiarity with the WebSocket lifecycle (connect, send, receive, close) and concepts like heartbeats and reconnection will help, but the code is self-documenting. +- **Understanding of Diffie-Hellman key exchange.** Knowing the basic DH protocol (Alice and Bob each generate key pairs, exchange public keys, compute the same shared secret) will make the X3DH implementation much easier to follow. The X3DH protocol is essentially four DH computations combined. +- **Familiarity with FastAPI or Flask.** The backend uses FastAPI patterns: route decorators, dependency injection, Pydantic schemas for request/response validation, and middleware. If you have used any Python web framework, the structure will be familiar. +- **Knowledge of browser IndexedDB API.** The client-side key storage uses IndexedDB directly (no wrapper library). IndexedDB is an asynchronous, transactional, key-value database built into every modern browser. Understanding its callback-based API (which this project wraps in Promises) will help you follow the `key-store.ts` implementation. + +## Quick Start + +### Docker (Recommended) + +```bash +# Clone and enter the project +cd PROJECTS/advanced/encrypted-p2p-chat + +# Copy environment file and update passwords +cp .env.example .env + +# Start all services (PostgreSQL, SurrealDB, Redis, backend, frontend, nginx) +docker compose -f dev.compose.yml up + +# Backend API: http://localhost:8000 +# Frontend dev: http://localhost:5173 +# API docs: http://localhost:8000/docs +# Nginx proxy: http://localhost:80 +``` + +### Manual Setup (Without Docker) + +```bash +# Backend +cd backend +uv sync +uv run uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 + +# Frontend (separate terminal) +cd frontend +pnpm install +pnpm dev +``` + +For manual setup, you need PostgreSQL, SurrealDB, and Redis running locally. Update the connection strings in `.env` to point to `localhost` instead of the Docker service names. + +### Using the justfile + +```bash +# See all available commands +just + +# Full setup (backend deps, frontend deps, env file) +just setup + +# Start dev environment +just dev-up + +# Run linters +just lint + +# Run tests +just test + +# Database migrations +just migrate-local head +``` + +## Project Structure + +``` +encrypted-p2p-chat/ +| +|-- backend/ +| |-- app/ +| | |-- api/ # HTTP and WebSocket route handlers +| | | |-- auth.py # WebAuthn registration and login endpoints +| | | |-- encryption.py # X3DH prekey bundle upload and retrieval +| | | |-- rooms.py # Chat room creation, listing, membership +| | | +-- websocket.py # WebSocket connection and message routing +| | | +| | |-- core/ +| | | |-- encryption/ +| | | | |-- x3dh_manager.py # X3DH protocol: key generation, exchange, verification +| | | | +-- double_ratchet.py # Double Ratchet: symmetric + DH ratchet, encrypt/decrypt +| | | |-- passkey/ +| | | | +-- passkey_manager.py # WebAuthn credential registration and verification +| | | |-- websocket_manager.py # Connection pool, broadcasting, heartbeat management +| | | |-- surreal_manager.py # SurrealDB client: connect, query, live subscriptions +| | | |-- redis_manager.py # Redis client: challenges, rate limits, ephemeral cache +| | | |-- enums.py # Shared enumerations (message types, status codes) +| | | |-- exceptions.py # Domain-specific exception classes +| | | +-- exception_handlers.py # FastAPI exception handler registration +| | | +| | |-- models/ # SQLModel database models (PostgreSQL) +| | | |-- Base.py # SQLAlchemy engine, session factory, init_db +| | | |-- User.py # User account with username and display name +| | | |-- Credential.py # WebAuthn credential storage (public key, counter) +| | | |-- IdentityKey.py # Long-term X25519 + Ed25519 identity key pairs +| | | |-- SignedPrekey.py # Medium-term signed prekeys (rotate every 48h) +| | | |-- OneTimePrekey.py # Single-use prekeys for X3DH (pool of 100) +| | | |-- RatchetState.py # Per-conversation Double Ratchet state blob +| | | +-- SkippedMessageKey.py # Cached keys for out-of-order message delivery +| | | +| | |-- services/ # Business logic layer +| | | |-- auth_service.py # Registration and authentication orchestration +| | | |-- prekey_service.py # Key bundle management and rotation scheduling +| | | |-- message_service.py # Encryption/decryption coordination +| | | |-- presence_service.py # Online/offline status tracking via SurrealDB +| | | +-- websocket_service.py # WebSocket message routing and type dispatch +| | | +| | |-- schemas/ # Pydantic request/response models +| | | |-- auth.py # Registration/login options and verification schemas +| | | |-- rooms.py # Room creation and listing schemas +| | | |-- websocket.py # WebSocket message payload schemas +| | | |-- surreal.py # SurrealDB record schemas +| | | +-- common.py # Shared schemas (health, root, pagination) +| | | +| | |-- config.py # Centralized settings, constants, environment loading +| | |-- factory.py # FastAPI app factory with lifespan context manager +| | +-- main.py # Uvicorn entry point +| | +| |-- tests/ # pytest test suite +| | |-- conftest.py # Shared fixtures (async engine, test client) +| | |-- test_auth_service.py # Authentication flow tests +| | |-- test_encryption.py # Double Ratchet encrypt/decrypt round-trip tests +| | |-- test_message_service.py # Message service integration tests +| | +-- test_x3dh.py # X3DH key exchange protocol tests +| | +| |-- alembic/ # Database migration scripts +| | +-- env.py # Alembic environment configuration +| +-- pyproject.toml # uv dependencies and tool configuration +| +|-- frontend/ +| |-- src/ +| | |-- crypto/ # Client-side Signal Protocol implementation +| | | |-- primitives.ts # WebCrypto API wrappers (X25519, Ed25519, AES-GCM, HKDF) +| | | |-- x3dh.ts # X3DH key exchange (client-side sender/receiver) +| | | |-- double-ratchet.ts # Double Ratchet state machine (client-side) +| | | |-- crypto-service.ts # Encryption orchestrator (ties X3DH + Ratchet together) +| | | |-- key-store.ts # IndexedDB private key storage (identity, prekeys, ratchet) +| | | +-- message-store.ts # Encrypted message serialization and deserialization +| | | +| | |-- pages/ # Route-level page components +| | | |-- Home.tsx # Landing page +| | | |-- Login.tsx # WebAuthn login flow +| | | |-- Register.tsx # WebAuthn registration flow +| | | |-- Chat.tsx # Main chat interface +| | | +-- NotFound.tsx # 404 page +| | | +| | |-- components/ +| | | |-- Auth/ # Authentication components +| | | | |-- AuthCard.tsx # Card container for auth forms +| | | | |-- AuthForm.tsx # Username/display name input form +| | | | +-- PasskeyButton.tsx # WebAuthn credential creation trigger +| | | |-- Chat/ # Chat interface components +| | | | |-- ChatHeader.tsx # Conversation header with user info +| | | | |-- ChatInput.tsx # Message composition and send +| | | | |-- ConversationList.tsx # Sidebar list of active conversations +| | | | |-- ConversationItem.tsx # Single conversation preview +| | | | |-- MessageBubble.tsx # Individual message display +| | | | |-- MessageList.tsx # Scrollable message history +| | | | |-- EncryptionBadge.tsx # Visual indicator of encryption status +| | | | |-- OnlineStatus.tsx # User online/offline indicator +| | | | |-- TypingIndicator.tsx # "User is typing..." display +| | | | |-- UserSearch.tsx # Search for users to start conversations +| | | | +-- NewConversation.tsx # New conversation creation flow +| | | |-- Layout/ # Page layout components +| | | | |-- AppShell.tsx # Main app layout wrapper +| | | | |-- Header.tsx # Top navigation bar +| | | | |-- Sidebar.tsx # Side navigation panel +| | | | +-- ProtectedRoute.tsx # Auth-gated route wrapper +| | | +-- UI/ # Reusable UI primitives +| | | |-- Avatar.tsx # User avatar display +| | | |-- Badge.tsx # Status badge component +| | | |-- Button.tsx # Button with variants +| | | |-- Dropdown.tsx # Dropdown menu +| | | |-- IconButton.tsx # Icon-only button +| | | |-- Input.tsx # Text input field +| | | |-- Modal.tsx # Dialog overlay +| | | |-- Skeleton.tsx # Loading placeholder +| | | |-- Spinner.tsx # Loading spinner +| | | |-- TextArea.tsx # Multiline text input +| | | |-- Toast.tsx # Notification toast +| | | +-- Tooltip.tsx # Hover tooltip +| | | +| | |-- services/ # API client layer +| | | |-- auth.service.ts # Authentication API calls +| | | +-- room.service.ts # Room and message API calls +| | | +| | |-- stores/ # Nanostores state management +| | | |-- auth.store.ts # Authentication state (user, token) +| | | |-- session.store.ts # Session lifecycle management +| | | |-- rooms.store.ts # Room list and active room state +| | | |-- messages.store.ts # Message history per conversation +| | | |-- presence.store.ts # User online status tracking +| | | |-- typing.store.ts # Typing indicator state +| | | |-- settings.store.ts # User preferences +| | | +-- ui.store.ts # UI state (modals, sidebars, loading) +| | | +| | |-- websocket/ # WebSocket client +| | | |-- websocket-manager.ts # Connection lifecycle and reconnection +| | | +-- message-handlers.ts # Incoming message type dispatch +| | | +| | |-- lib/ # Shared utilities +| | | |-- api-client.ts # HTTP client wrapper (fetch-based) +| | | |-- base64.ts # Base64/Base64URL encoding helpers +| | | |-- validators.ts # Input validation functions +| | | +-- date.ts # Date formatting utilities +| | | +| | |-- types/ # TypeScript type definitions +| | | |-- encryption.ts # Crypto-related types and interfaces +| | | |-- auth.ts # Authentication types +| | | |-- chat.ts # Chat message and room types +| | | |-- websocket.ts # WebSocket message types +| | | |-- api.ts # API response types +| | | |-- components.ts # Component prop types +| | | +-- guards.ts # Type guard functions +| | | +| | |-- App.tsx # Root component with router +| | |-- config.ts # Environment configuration +| | |-- index.tsx # Application entry point +| | +-- index.css # Global styles and Tailwind imports +| | +| |-- vite.config.ts # Vite build configuration +| |-- tsconfig.json # TypeScript compiler options +| |-- eslint.config.js # ESLint configuration +| +-- package.json # pnpm dependencies +| +|-- conf/ +| |-- docker/ +| | |-- dev/ +| | | |-- fastapi.docker # Dev Dockerfile for backend (hot reload) +| | | +-- vite.docker # Dev Dockerfile for frontend (HMR) +| | +-- prod/ +| | |-- fastapi.docker # Production Dockerfile for backend +| | +-- vite.docker # Production Dockerfile for frontend (static build) +| +-- nginx/ +| |-- dev.nginx # Development Nginx config +| |-- prod.nginx # Production Nginx config +| +-- http.conf # Shared HTTP settings +| +|-- learn/ # Educational documentation (you are here) +|-- .env.example # Template environment variables +|-- compose.yml # Production Docker Compose +|-- dev.compose.yml # Development Docker Compose +|-- justfile # Command runner recipes ++-- LICENSE +``` + +## Architecture Overview + +The following diagram shows how data flows through the system during an encrypted message exchange: + +``` ++------------------+ +------------------+ +| Alice's | | Bob's | +| Browser | | Browser | +| | | | +| +------------+ | | +------------+ | +| | IndexedDB | | | | IndexedDB | | +| | (private | | | | (private | | +| | keys) | | | | keys) | | +| +-----+------+ | | +-----+------+ | +| | | | | | +| +-----v------+ | | +-----v------+ | +| | WebCrypto | | | | WebCrypto | | +| | X3DH + | | | | X3DH + | | +| | Ratchet | | | | Ratchet | | +| +-----+------+ | | +-----+------+ | +| | | | ^ | +| plaintext | | plaintext | +| encrypted to | | decrypted from | +| ciphertext | | ciphertext | +| | | | | | +| +-----v------+ | +----------------+ | +-----+------+ | +| | WebSocket +--+-------->| Server +-------->+--+ WebSocket | | +| | Client | | | | | | Client | | +| +------------+ | | +------------+ | | +------------+ | +| | | | FastAPI | | | | ++------------------+ | | WebSocket | | +------------------+ + | | Router | | + | +-----+------+ | + | | | + | +-----v------+ | + | | SurrealDB | | + | | (encrypted | | + | | blobs) | | + | +------------+ | + | | + | +------------+ | + | | PostgreSQL | | + | | (auth, | | + | | keys) | | + | +------------+ | + | | + | +------------+ | + | | Redis | | + | | (challenges| | + | | sessions) | | + | +------------+ | + +----------------+ +``` + +Key points: + +1. **Plaintext never leaves the browser.** Encryption happens in the WebCrypto API before the message hits the WebSocket. +2. **The server sees only ciphertext.** It stores and forwards encrypted blobs. It cannot decrypt them. +3. **Private keys live in IndexedDB.** They are generated by the WebCrypto API and stored client-side. The server stores only public keys. +4. **Three databases serve distinct purposes.** PostgreSQL handles relational auth data, SurrealDB handles real-time message storage, and Redis handles ephemeral state like challenge tokens. + +## Next Steps + +The remaining documents in this learn folder walk through the project layer by layer: + +- **01-CONCEPTS.md** -- Signal Protocol theory, X3DH mathematics, Double Ratchet mechanics, WebAuthn internals, and the real-world breaches that motivated these protocols. This is where you build the mental model for why each cryptographic decision was made. + +- **02-ARCHITECTURE.md** -- System design, database schemas, data flow diagrams, security architecture, and the reasoning behind each technology choice. Covers the three-database split, the WebSocket message protocol, and the zero-knowledge server design. + +- **03-IMPLEMENTATION.md** -- Line-by-line walkthrough of the cryptographic implementations, authentication flows, WebSocket messaging, and key lifecycle management. Follows the code through a complete message exchange from registration through key upload, session establishment, message encryption, delivery, and decryption. + +- **04-CHALLENGES.md** -- Extension projects ranging from adding read receipts and typing indicators to implementing group encryption (Sender Keys), multi-device support, disappearing messages, and post-quantum key exchange using the `liboqs-python` library already included in the project dependencies. + +## Common Issues + +**Docker services will not start:** +Check that ports 5432, 8001, 6379, 8000, 5173, and 80 are not already in use. Note that SurrealDB maps to host port 8001 (not 8000, which is used by the FastAPI backend). Run `docker compose -f dev.compose.yml down -v` to remove stale containers and volumes, then restart. If a specific service fails, check its logs with `just dev-logs postgres` (or whichever service name). + +**WebAuthn registration fails in development:** +WebAuthn requires either `localhost` or a valid HTTPS origin. The default configuration uses `RP_ID=localhost` and `RP_ORIGIN=http://localhost` in `.env.example`. Make sure your `.env` file has these values set correctly. If you are accessing the frontend at `http://localhost:5173`, then `RP_ORIGIN` must match that exact origin (scheme + host + port). If you are testing from a LAN IP address or a non-localhost hostname, you will need a TLS certificate because WebAuthn will refuse to create credentials on non-secure origins other than localhost. + +**"No identity key" error when starting a chat:** +After WebAuthn registration, the client must generate and upload encryption keys (identity key pair, signed prekey, one-time prekeys) to the server. This happens in `crypto-service.ts`. If this step fails silently, the user will have an account but no encryption keys, and other users will not be able to initiate encrypted sessions with them. Check the browser console for errors during the key generation step. Common causes include IndexedDB access being blocked (private browsing mode in some browsers restricts IndexedDB), WebCrypto API not being available (requires a secure context), or network errors during the key upload POST request to the `/api/encryption/keys` endpoint. + +**Database migration errors:** +If you see schema mismatch errors on startup, run migrations first. For Docker: `just migrate head`. For local development: `just migrate-local head`. If you need to start fresh, `just dev-down` followed by removing the Docker volumes will give you a clean database. + +**SurrealDB connection refused:** +SurrealDB runs on port 8000 inside the container but is mapped to port 8001 on the host. The backend connects to `ws://surrealdb:8000` (the internal Docker network address). If you are running the backend outside Docker, update `SURREAL_URL` in `.env` to `ws://localhost:8001`. + +**Frontend WebSocket disconnects:** +The WebSocket client has automatic reconnection built in, but if the backend is not running or the connection URL is wrong, you will see repeated disconnection/reconnection attempts in the browser console. Verify that `VITE_WS_URL` in the frontend `.env` matches the actual backend WebSocket endpoint. The default is `ws://localhost:8000`. + +## Related Projects + +- **API Rate Limiter** (advanced) -- Implements Redis-based sliding window and token bucket rate limiting. The same Redis patterns (TTL-based key expiration, atomic increment operations) are used in this project for WebAuthn challenge storage and message rate limiting at 60 messages per minute per user. + +- **Bug Bounty Platform** (advanced) -- Another fullstack project combining authentication, real-time features, and multi-database architecture. Shares architectural patterns like the FastAPI app factory, service layer abstraction, and Docker Compose orchestration. + +- **Simple Port Scanner** (beginner) -- Introduces TCP networking fundamentals including socket programming, port states, and connection handshakes. These network-layer concepts underpin the WebSocket transport used for real-time message delivery in this project. diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md new file mode 100644 index 00000000..78647dda --- /dev/null +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/01-CONCEPTS.md @@ -0,0 +1,1467 @@ +# Security Concepts + +This document covers the cryptographic and authentication foundations of +the encrypted P2P chat application. Every concept here maps directly to +code in the repository. If you can read and understand this document, you +will know exactly what happens to a message from the moment a user types +it to the moment it appears on the recipient's screen, and why every +step exists. + + +## End-to-End Encryption + +### What It Is + +End-to-end encryption (E2EE) is a communication model where only the +two endpoints of a conversation can read the messages exchanged between +them. The defining property is that the server acts as a blind relay: it +stores and forwards encrypted blobs, but it has no key material that +would allow it to decrypt those blobs. The encryption happens on the +sender's device, the decryption happens on the recipient's device, and +at no point between those two devices does the plaintext exist. + +This is different from transport encryption (TLS/HTTPS), which encrypts +the link between your device and the server, and between the server and +the other device, but allows the server itself to read everything. With +transport encryption, the server is a trusted intermediary. With E2EE, +the server is an untrusted courier. + +The distinction matters because trust is a vulnerability. If the server +can read your messages, then anyone who compromises the server can also +read your messages. That includes attackers who breach the server, rogue +employees, and government agencies with legal authority to compel the +server operator to produce data. + +### Why It Matters + +The history of messaging security is a history of servers being +compromised, coerced, or caught lying about their access to user data. + +**2013: Edward Snowden and the NSA PRISM program.** Snowden leaked +classified NSA documents showing that major technology companies +including Google, Microsoft, Yahoo, Facebook, Apple, and others had +provided the NSA with direct server-side access to user communications +under the PRISM surveillance program. Because these services used +transport encryption rather than E2E encryption, the companies held +decryption keys on their servers and could comply with government +requests (or be compelled to comply through FISA court orders). The +server had access, so the government got access. This was the single +largest catalyst for the adoption of end-to-end encryption in consumer +products. + +**2019: Jeff Bezos WhatsApp hack.** In January 2020, forensic analysis +by FTI Consulting concluded that Amazon CEO Jeff Bezos's iPhone was +compromised via a malicious video file sent through WhatsApp from Saudi +Crown Prince Mohammed bin Salman's account. While WhatsApp's Signal +Protocol E2E encryption protected the message content in transit, the +attack targeted the endpoint device itself with spyware (attributed to +NSO Group's Pegasus). This case is instructive because it shows both +the strength and the boundary of E2E encryption: it protects messages +between devices, but if the device itself is compromised, the attacker +reads the plaintext after decryption. E2E encryption defends against +network and server compromise, not endpoint compromise. + +**2020: Zoom E2E encryption scandal.** Zoom Video Communications +marketed their product as providing "end-to-end encryption" for video +calls. Investigation by The Intercept (March 2020) and the Citizen Lab +at University of Toronto revealed that Zoom held the encryption keys on +their servers. Calls were encrypted with AES-128 in ECB mode (a weak +cipher mode that leaks patterns) between the client and Zoom's server, +but Zoom's infrastructure could decrypt all call content. The FTC +settlement in November 2020 required Zoom to implement actual security +measures and prohibited them from misrepresenting their encryption +practices. This case demonstrates that the word "encrypted" without the +qualifier "end-to-end" is meaningless for privacy. + +**2021: ProtonMail logging controversy.** ProtonMail, a Swiss encrypted +email provider that markets itself on privacy, was compelled by a Swiss +court order (requested via Europol on behalf of French authorities) to +log the IP address and browser fingerprint of a French climate activist. +While ProtonMail's E2E encryption meant they could not read email +content, they were compelled to collect metadata. The server could not +read messages, but it could identify who was sending them. This case +shows that E2E encryption solves the content problem but does not +automatically solve the metadata problem. + +The pattern is consistent: any server that CAN read your messages WILL +eventually be compelled to, whether by government subpoena, by +attackers who breach the infrastructure, or by insiders who abuse their +access. The only reliable defense is to make it architecturally +impossible for the server to read the data in the first place. + +### How It Works (in this project) + +``` +Alice's Device Server Bob's Device ++----------------+ +----------------+ +----------------+ +| Plaintext | | | | | +| "Hello Bob" | | Encrypted | | Plaintext | +| | | ---> | blob only | ---> | "Hello Bob" | +| v | | No keys | | ^ | +| AES-256-GCM | | No access | | AES-256-GCM | +| encrypt | | No decrypt | | decrypt | ++----------------+ +----------------+ +----------------+ + +Keys derived from Server stores Keys derived from +Double Ratchet on ciphertext, nonce, Double Ratchet on +Alice's device and header verbatim Bob's device +``` + +On the server side, `message_service.py:269-314` implements the +`store_encrypted_message` method. Look at the docstring on line 280: +`"Stores client-encrypted message in SurrealDB (pass-through, no server encryption)"`. +The function receives `ciphertext`, `nonce`, and `header` as strings +from the client, and stores them directly in SurrealDB at lines 292-302 +without any decryption or re-encryption step. The server is a +passthrough. It writes what it receives and reads what it wrote. At no +point does it import any cryptographic key or call any decryption +function on these message parameters. + +On the client side, `crypto-service.ts:217-249` implements the +`encrypt` method. The client calls `encryptMessage` from +`double-ratchet.ts` at line 229, receives ciphertext and nonce, then +sends these as base64-encoded strings to the server. The plaintext +never leaves the client's process. The actual symmetric encryption +happens in `primitives.ts:224-259` using the WebCrypto API's AES-GCM +implementation, which runs in the browser's native cryptographic module +rather than in JavaScript. This means the plaintext is never even +accessible to JavaScript debugging tools during the encryption +operation itself. + + +## The Signal Protocol + +### What It Is + +The Signal Protocol is a cryptographic ratcheting protocol originally +developed by Trevor Perrin and Moxie Marlinspike at Open Whisper Systems +(now the Signal Foundation). It was designed to provide end-to-end +encryption for instant messaging with strong forward secrecy and +post-compromise security properties. + +The protocol was first deployed in the TextSecure application (the +predecessor to Signal) and was formally described in a series of +technical specifications published at signal.org/docs. It was +independently analyzed and formally verified in academic papers, +including "A Formal Security Analysis of the Signal Messaging Protocol" +by Cohn-Gordon et al. at IEEE EuroS&P 2017, which proved that the +protocol meets its claimed security properties under standard +cryptographic assumptions. + +The Signal Protocol is now deployed at massive scale. WhatsApp completed +its rollout of Signal Protocol encryption to all users in April 2016, +covering over 1 billion users at the time (now over 2 billion). Google +Messages adopted the Signal Protocol for RCS messaging. Facebook +Messenger offered it as an optional "Secret Conversations" mode. Skype +implemented it as "Private Conversations." The protocol's design has +been influential enough that it is effectively the industry standard for +secure messaging. + +### The Two Core Components + +The Signal Protocol combines two distinct cryptographic mechanisms, each +solving a different problem: + +``` +Signal Protocol = X3DH (initial handshake) + Double Ratchet (ongoing encryption) + +X3DH: "How do Alice and Bob agree on a shared secret + when Bob might be offline?" + + Solves the ASYNCHRONOUS key agreement problem. + Bob uploads prekey bundles to the server ahead of time. + Alice can start a conversation using Bob's prekeys + without Bob being online. + +Double Ratchet: "Once they share a secret, how do they + encrypt each message with a UNIQUE key + that can never be recovered?" + + Solves the FORWARD SECRECY and POST-COMPROMISE + SECURITY problem. Every message gets its own + ephemeral encryption key. Compromising one key + does not reveal past or future messages. +``` + +X3DH runs once at the start of a conversation. The Double Ratchet runs +continuously for every message after that. The output of X3DH (a shared +secret) is the input to the Double Ratchet (the initial root key). + + +## X3DH (Extended Triple Diffie-Hellman) + +### What It Is + +Standard Diffie-Hellman key exchange requires both parties to be online +at the same time. Alice generates a value, sends it to Bob, Bob +generates a value, sends it back to Alice, and they both compute the +shared secret. This works fine for a phone call or a live connection, +but it does not work for asynchronous messaging. If Alice wants to send +Bob a message at 3 AM while Bob's phone is off, standard DH cannot +proceed because Bob is not there to generate and send his half. + +X3DH solves this by having Bob pre-generate a set of key material and +upload it to the server before going offline. This pre-generated material +is called a "prekey bundle." When Alice wants to start a conversation, +she downloads Bob's prekey bundle from the server and uses it to compute +a shared secret without Bob's participation. When Bob comes back online, +he can compute the same shared secret from the information Alice sends +him, because the mathematics of Diffie-Hellman allow both parties to +independently arrive at the same result. + +The "Extended Triple" in X3DH refers to the fact that the protocol +performs three or four separate Diffie-Hellman operations (not just one) +to achieve stronger security properties than a single DH exchange would +provide. + +### Key Types + +X3DH uses four types of keys. Each has a different lifetime and purpose. +Understanding why four types exist (instead of just one) is essential to +understanding the security model. + +**Identity Key (IK) -- Long-term, generated once per user** + +The identity key is a permanent keypair that represents the user's +cryptographic identity. It is generated once and kept for the lifetime +of the account. This project generates two identity keypairs per user: + +- An X25519 keypair for Diffie-Hellman operations + (ref: `x3dh_manager.py:60-86`) +- An Ed25519 keypair for digital signatures + (ref: `x3dh_manager.py:88-114`) + +The X25519 keypair participates directly in the DH calculations. The +Ed25519 keypair signs the signed prekey to prove it belongs to the same +identity. These are separate curves because X25519 is a Diffie-Hellman +function (it computes shared secrets) and Ed25519 is a signature scheme +(it signs and verifies data). They cannot be interchanged. The private +keys are stored in the database (server-side) or in IndexedDB +(client-side), and they never change unless the user explicitly resets +their identity. + +**Signed Prekey (SPK) -- Medium-term, rotated every 48 hours** + +The signed prekey is an X25519 keypair that rotates periodically. In +this project, rotation happens every 48 hours as configured at +`config.py:76` (`SIGNED_PREKEY_ROTATION_HOURS=48`). + +When a new SPK is generated (`x3dh_manager.py:116-152`), the public +key is signed using the Ed25519 identity key at line 141: +`signature = identity_private.sign(spk_public_bytes)`. This signature +proves that the SPK was created by the holder of the identity key. When +Alice downloads Bob's prekey bundle, she verifies this signature before +using the SPK, which prevents a man-in-the-middle from substituting +their own SPK. + +The rotation period is a tradeoff. Shorter rotation provides better +forward secrecy (because old SPKs are deleted, and any DH secrets +computed with them become unrecoverable). Longer rotation means fewer +key management operations and less complexity. The 48-hour window used +here is consistent with Signal's recommendation. + +Old signed prekeys are retained for 7 days after deactivation +(`config.py:77`, `SIGNED_PREKEY_RETENTION_DAYS=7`) to allow +messages-in-flight that were encrypted against the old SPK to still be +decrypted. After that retention period, +`prekey_service.py:428-465` (`cleanup_old_signed_prekeys`) deletes them +permanently. + +**One-Time Prekey (OPK) -- Single use, consumed and deleted** + +One-time prekeys are X25519 keypairs that are used exactly once and then +discarded. They are generated in batches +(ref: `x3dh_manager.py:154-174` for generation, +`prekey_service.py:363-407` for batch replenishment) and uploaded to the +server. + +When Alice initiates a conversation with Bob, the server gives Alice one +of Bob's unused OPKs and marks it as consumed +(`prekey_service.py:331-338`). This OPK participates in the fourth DH +operation (DH4) of the X3DH handshake. Because the OPK is used only +once and then deleted, it provides an additional layer of forward +secrecy specifically for the initial message of a conversation. + +If Bob has no unused OPKs remaining (they have all been consumed by +other users initiating conversations), X3DH falls back to three DH +operations instead of four. The protocol still works, but the initial +message has slightly weaker forward secrecy because the fourth DH +operation is skipped. The system generates 100 OPKs initially +(`config.py:75`, `DEFAULT_ONE_TIME_PREKEY_COUNT=100`) and replenishes +them when the supply drops below half. + +**Ephemeral Key (EK) -- Generated per session, never stored** + +The ephemeral key is a fresh X25519 keypair generated by Alice (the +sender) at the moment she initiates a conversation. It is used in DH2, +DH3, and DH4 of the X3DH handshake. It is never stored on disk; it +exists only in memory for the duration of the key exchange computation. + +In the code, it is generated at `x3dh_manager.py:227-228`: + +```python +alice_ek_private = X25519PrivateKey.generate() +alice_ek_public = alice_ek_private.public_key() +``` + +After the shared secret is computed, Alice sends the ephemeral public +key to Bob (so Bob can perform the same DH operations on his side), and +the ephemeral private key is discarded. Because the private component is +never persisted, even if Alice's device is later compromised, the +attacker cannot recover the ephemeral private key and therefore cannot +recompute the initial shared secret. + +### The Math + +X3DH performs four Diffie-Hellman operations between different +combinations of keys. Each operation produces a 32-byte shared secret. +The four secrets are concatenated and fed into HKDF to produce the final +shared key. + +``` +Alice (sender) has: Bob (receiver) has: + IK_A (identity private key) IK_B (identity public key) + EK_A (ephemeral, just generated) SPK_B (signed prekey public) + OPK_B (one-time prekey public) + +DH Operations (each produces 32 bytes): + + DH1 = X25519(IK_A_private, SPK_B_public) + Alice's identity x Bob's signed prekey + + DH2 = X25519(EK_A_private, IK_B_public) + Alice's ephemeral x Bob's identity + + DH3 = X25519(EK_A_private, SPK_B_public) + Alice's ephemeral x Bob's signed prekey + + DH4 = X25519(EK_A_private, OPK_B_public) [optional] + Alice's ephemeral x Bob's one-time prekey + +Key Material Derivation: + + input = 0xFF * 32 || DH1 || DH2 || DH3 || DH4 + salt = 0x00 * 32 + info = "X3DH" + SK = HKDF-SHA256(salt, input, info, length=32) +``` + +In the codebase, the sender side is at `x3dh_manager.py:241-264`: + +- Line 241: `dh1 = alice_ik_private.exchange(bob_spk_public)` -- DH1 +- Line 242: `dh2 = alice_ek_private.exchange(bob_ik_public)` -- DH2 +- Line 243: `dh3 = alice_ek_private.exchange(bob_spk_public)` -- DH3 +- Line 251: `dh4 = alice_ek_private.exchange(bob_opk_public)` -- DH4 (if OPK available) +- Line 252: `key_material = dh1 + dh2 + dh3 + dh4` -- concatenation +- Lines 257-264: HKDF derivation with `0xFF * 32` prefix and `b'X3DH'` info string + +The receiver side at `x3dh_manager.py:308-333` performs the same +operations but with the roles reversed. DH1 becomes +`bob_spk_private.exchange(alice_ik_public)` (line 308), because Bob has +the SPK private key and Alice's IK public key. The property of +Diffie-Hellman guarantees that `X25519(a_priv, B_pub)` produces the +same result as `X25519(b_priv, A_pub)`, so both sides compute identical +shared secrets. + +The `0xFF * 32` prefix prepended at line 257 (`f = b'\xff' * X25519_KEY_SIZE`) +is a fixed padding specified by the X3DH standard. It ensures the HKDF +input is at least 32 bytes long even in edge cases and provides domain +separation from other uses of the same keys. + +### Why Four DH Operations? + +Each DH operation provides a specific security property. If any single +operation were removed, a specific class of attack would become possible. + +**DH1: IK_A x SPK_B -- Authenticates Alice to Bob** + +This operation uses Alice's long-term identity key. Only Alice (the +holder of IK_A_private) could have produced this DH output with +SPK_B_public. When Bob computes the same value using SPK_B_private and +IK_A_public, he has cryptographic proof that the message came from +Alice. Without DH1, anyone who knows Bob's public SPK could impersonate +any sender. + +**DH2: EK_A x IK_B -- Authenticates Bob to Alice** + +This operation uses Bob's long-term identity key. Only Bob (the holder +of IK_B_private) could reproduce this DH output. This ensures Alice is +actually talking to Bob, not to an impersonator who uploaded their own +prekey bundle to the server. Without DH2, a malicious server could +substitute its own keys for Bob's and perform a man-in-the-middle attack. + +**DH3: EK_A x SPK_B -- Provides forward secrecy** + +This operation uses Alice's ephemeral key (generated fresh, never stored) +and Bob's signed prekey (rotated every 48 hours). Because EK_A_private +is discarded immediately and SPK_B_private is eventually deleted during +rotation, this DH output becomes unrecoverable after both keys are gone. +Even if both Alice and Bob's identity keys are later compromised, past +session keys derived partly from DH3 cannot be recomputed. This is the +core forward secrecy guarantee. + +**DH4: EK_A x OPK_B -- Additional forward secrecy for initial messages** + +This operation uses Bob's one-time prekey, which is consumed and deleted +immediately after use. It provides forward secrecy specifically for the +first message in a conversation. Without DH4, if an attacker compromised +Bob's SPK_private (which exists for up to 48 hours), they could +retroactively decrypt initial messages sent during that window. DH4 +ensures that even a compromised SPK is insufficient, because OPK_private +was deleted the moment it was used. + +DH4 also prevents replay attacks on the initial handshake. Because the +OPK is single-use, an attacker who records Alice's initial message +cannot replay it later: Bob has already consumed the OPK, so the server +will not provide the same one again, and Bob's side will not have the +OPK private key available for a replayed handshake. + +### Prekey Bundle Verification + +Before performing any DH operations, Alice must verify that Bob's signed +prekey actually belongs to Bob. A malicious server could substitute its +own SPK and intercept communications. + +The verification happens at `x3dh_manager.py:217-220` inside +`perform_x3dh_sender`: + +```python +if not self.verify_signed_prekey(bob_bundle.signed_prekey, + bob_bundle.signed_prekey_signature, + bob_identity_public_ed25519): + raise ValueError("Invalid signed prekey signature") +``` + +The `verify_signed_prekey` method at `x3dh_manager.py:176-206` uses +Ed25519 signature verification. It decodes the SPK public key bytes, the +signature bytes, and the Ed25519 identity public key bytes, then calls +`identity_public.verify(signature_bytes, spk_public_bytes)` at line 196. +Ed25519 verification either succeeds or raises `InvalidSignature`. If +verification fails, the entire X3DH handshake is aborted. + +This verification is critical. Without it, a server-side attacker could +replace Bob's SPK with one they control, perform DH operations using +their own private key, and transparently proxy messages between Alice +and Bob while reading everything. The Ed25519 signature binds the SPK +to Bob's identity key, making substitution detectable. + +Note that this only works if Alice has Bob's authentic identity public +key. In practice, identity key verification is done through "safety +numbers" or "key verification" -- a separate out-of-band process where +Alice and Bob compare fingerprints of each other's identity keys in +person or through a trusted secondary channel. + + +## Double Ratchet Algorithm + +### What It Is + +The Double Ratchet is an algorithm for managing encryption keys in an +ongoing conversation. It was developed by Trevor Perrin and Moxie +Marlinspike as part of the Signal Protocol, building on earlier work +from the Off-the-Record (OTR) messaging protocol. + +The Double Ratchet provides two critical security properties that go +beyond what a static shared key could provide: + +1. **Forward secrecy** -- Compromising a current key does not expose + past messages. Even if an attacker steals the current encryption key, + they cannot derive previous keys and therefore cannot decrypt earlier + messages. + +2. **Post-compromise security (break-in recovery)** -- After a key + compromise, future messages become secure again once a new DH ratchet + step occurs. If an attacker temporarily gains access to key material, + they lose access to the conversation as soon as the keys advance + through a new Diffie-Hellman exchange. + +The name "Double Ratchet" refers to the fact that it combines two +ratcheting mechanisms: a **DH ratchet** (Diffie-Hellman ratchet) that +advances when the conversation's turn changes, and a **symmetric +ratchet** (hash ratchet) that advances with every single message. + +### The Three Chains + +The Double Ratchet maintains three linked KDF chains: the root chain, +the sending chain, and the receiving chain. + +``` +Root Chain (KDF_RK) + | + |-- [DH ratchet step] --> new root key + new chain key + | + +-- Sending Chain (KDF_CK) + | |-- advance --> Message Key 1 --> Encrypt msg 1 + | |-- advance --> Message Key 2 --> Encrypt msg 2 + | +-- advance --> Message Key 3 --> Encrypt msg 3 + | + +-- Receiving Chain (KDF_CK) + |-- advance --> Message Key 1 --> Decrypt msg 1 + |-- advance --> Message Key 2 --> Decrypt msg 2 + +-- advance --> Message Key 3 --> Decrypt msg 3 +``` + +Each chain is a sequence of key derivation operations. The root chain +produces new sending and receiving chain keys through the DH ratchet. +The sending and receiving chains produce individual message keys through +the symmetric ratchet. Every message key is used exactly once and then +discarded. + +### KDF Chain Operations + +**KDF_RK: Root Key Derivation** + +The root chain advances during a DH ratchet step. It takes the current +root key and a fresh DH output (from a new DH key exchange) and +produces a new root key and a new chain key. + +Reference: `double_ratchet.py:79-94` + +```python +def _kdf_rk(self, root_key: bytes, dh_output: bytes) -> tuple[bytes, bytes]: + hkdf = HKDF( + algorithm = hashes.SHA256(), + length = HKDF_OUTPUT_SIZE * 2, # 64 bytes total + salt = root_key, # current root key as salt + info = b'', + ) + output = hkdf.derive(dh_output) + new_root_key = output[: HKDF_OUTPUT_SIZE] # first 32 bytes + new_chain_key = output[HKDF_OUTPUT_SIZE :] # last 32 bytes + return new_root_key, new_chain_key +``` + +HKDF-SHA256 is used with the current root key as the salt and the DH +output as the input key material. The output is 64 bytes, split in half: +the first 32 bytes become the new root key, the last 32 bytes become the +new chain key. This split ensures that knowing the chain key does not +reveal the root key, maintaining the separation between the root chain +and the message chains. + +**KDF_CK: Chain Key Derivation** + +The symmetric chains advance with every message. Each step takes the +current chain key and produces the next chain key and a message key. + +Reference: `double_ratchet.py:96-109` + +```python +def _kdf_ck(self, chain_key: bytes) -> tuple[bytes, bytes]: + h_chain = hmac.HMAC(chain_key, hashes.SHA256()) + h_chain.update(b'\x01') + next_chain_key = h_chain.finalize() + + h_message = hmac.HMAC(chain_key, hashes.SHA256()) + h_message.update(b'\x02') + message_key = h_message.finalize() + + return next_chain_key, message_key +``` + +Two separate HMAC-SHA256 computations are performed using the same chain +key but with different constants: + +``` +chain_key --+-- HMAC(chain_key, 0x01) --> next_chain_key (kept for future) + | + +-- HMAC(chain_key, 0x02) --> message_key (used once, discarded) +``` + +The use of different constants (0x01 and 0x02) is essential. If the same +constant were used, the chain key and message key would be identical, +which would mean that learning the message key (perhaps through a chosen +plaintext attack) would reveal the chain key and allow derivation of all +future keys. By using different HMAC inputs, the message key and the +next chain key are cryptographically independent: knowing one does not +reveal the other. + +The message key is used exactly once to encrypt or decrypt a single +message, then discarded. The next chain key replaces the current chain +key and is used to derive the next message key. This one-way chain is +what provides forward secrecy within a single DH ratchet epoch. + +### The DH Ratchet Step + +The symmetric ratchet handles the simple case: sequential messages from +the same sender. But it cannot, on its own, provide post-compromise +security. If an attacker compromises a chain key, they can derive all +future message keys from that chain. The DH ratchet solves this. + +A DH ratchet step occurs whenever the conversation's direction changes. +When Alice receives a message from Bob that includes a new DH public +key (one she has not seen before), she performs a DH ratchet step: she +generates a new DH keypair, performs a DH exchange with Bob's new public +key, and uses the output to derive new root and chain keys through +KDF_RK. + +``` +Message flow and DH ratchet steps: + +Alice sends msgs 1,2,3 using DH keypair A1: + A1 --> msg1(mk1), msg2(mk2), msg3(mk3) + [symmetric ratchet advances 3 times, same DH key] + +Bob receives, generates new DH keypair B1, sends reply: + DH ratchet: root_key' = KDF_RK(root_key, DH(B1_priv, A1_pub)) + B1 --> msg4(mk1'), msg5(mk2') + [new chain, new keys, A1 compromise no longer helps] + +Alice receives, generates new DH keypair A2, sends reply: + DH ratchet: root_key'' = KDF_RK(root_key', DH(A2_priv, B1_pub)) + A2 --> msg6(mk1''), msg7(mk2'') + [new chain again, B1 compromise no longer helps] +``` + +Each DH ratchet step introduces fresh random entropy (from the newly +generated DH keypair) into the key derivation chain. This means that +even if an attacker had compromised all previous key material, the new +DH exchange produces a shared secret they cannot predict, and all +subsequent keys are secure again. + +The implementation spans `double_ratchet.py:155-213`: + +- `_dh_ratchet_send` (lines 155-176): Called when the sender needs to + advance the ratchet. Generates a new DH keypair at line 159, performs + DH with the peer's public key at line 169, and derives new root and + sending chain keys at lines 171-174. + +- `_dh_ratchet_receive` (lines 178-213): Called when a received message + contains a new DH public key. Updates the peer public key at line 189, + performs DH with the existing private key to derive a new receiving + chain key at lines 195-198, then generates a new private key at line + 200 and performs another DH to derive a new sending chain key at lines + 208-211. This double DH step on the receiver side ensures both + receiving and sending chains are updated. + +### Out-of-Order Message Handling + +Internet messages can arrive out of order. If Alice sends messages 1, 2, +3 and Bob receives 1, 3 (message 2 is delayed), Bob needs to: + +1. Process message 1 normally (derive mk1, decrypt) +2. When processing message 3, recognize that message 2 was skipped +3. Derive and cache mk2 (so it can be used later when message 2 arrives) +4. Derive mk3 and decrypt message 3 + +The skipped message key mechanism handles this. Reference: +`double_ratchet.py:215-277`. + +`_store_skipped_message_keys` (lines 215-244) is called when the +received message number is greater than the expected message number. It +iterates through the gap, deriving and caching each skipped message key: + +```python +chain_key = state.receiving_chain_key +for msg_num in range(state.receiving_message_number, until_message_number): + chain_key, message_key = self._kdf_ck(chain_key) + state.skipped_message_keys[(dh_public_key, msg_num)] = message_key +state.receiving_chain_key = chain_key +``` + +The skipped keys are stored in a dictionary keyed by `(dh_public_key, +message_number)`. This tuple key is necessary because message numbers +reset with each DH ratchet step: message 0 under DH key A1 is different +from message 0 under DH key A2. + +`_try_skipped_message_key` (lines 260-277) checks whether a received +message matches a previously cached skipped key. If it does, the cached +key is used for decryption and then removed from the cache (it is +consumed by `dict.pop()` at line 270). + +Security limits prevent abuse. An attacker who sends messages with +enormous message numbers could force the ratchet to derive and store +millions of keys, exhausting memory. Two limits are enforced: + +- `MAX_SKIP_MESSAGE_KEYS = 1000` (`config.py:73`): No more than 1000 + message keys can be skipped in a single gap. If a message arrives + claiming to be message number 5000 when we expect message 0, the + decryption is rejected at lines 226-230. + +- `MAX_CACHED_MESSAGE_KEYS = 2000` (`config.py:74`): The total number + of cached skipped keys across all ratchet epochs. If the cache is + full, the oldest keys are evicted at lines 232-234 via + `_evict_oldest_skipped_keys`. + +### Forward Secrecy Proof + +Here is a step-by-step walkthrough of why compromising key material at +time T does not expose messages before T. + +Assume at message N, an attacker steals the current chain_key_N. + +``` +What the attacker CAN compute (forward direction): + chain_key_N -----> HMAC(chain_key_N, 0x01) = chain_key_N+1 + chain_key_N+1 ---> HMAC(chain_key_N+1, 0x01) = chain_key_N+2 + ... and so on for all future chain keys + +What the attacker CANNOT compute (backward direction): + chain_key_N <-/-- chain_key_N-1 + + Why? Because HMAC is a one-way function. + + chain_key_N = HMAC(chain_key_N-1, 0x01) + + Given chain_key_N, you cannot solve for chain_key_N-1. + This would require inverting HMAC-SHA256, which is + computationally infeasible (preimage resistance). + +Therefore: + message_key_N-1 = HMAC(chain_key_N-1, 0x02) <-- UNREACHABLE + message_key_N-2 = HMAC(chain_key_N-2, 0x02) <-- UNREACHABLE + message_key_1 = HMAC(chain_key_1, 0x02) <-- UNREACHABLE +``` + +The attacker can decrypt messages N+1, N+2, N+3, and so on (until the +next DH ratchet step introduces new entropy). But they cannot decrypt +any message before N. All past message keys are derived from chain keys +that are computationally inaccessible given only chain_key_N. + +Now consider what happens at the next DH ratchet step. Bob sends a +message with a new DH public key B2. Alice generates a new keypair A3 +and performs DH(A3_priv, B2_pub). This produces a fresh DH output that +the attacker cannot predict (because they do not know A3_priv, which +was just generated from secure random data). The new root key and chain +key are derived from this fresh DH output through KDF_RK. The +attacker's knowledge of the old chain key becomes useless. This is +post-compromise security: the system self-heals. + + +## AES-256-GCM Encryption + +### What It Is + +AES-256-GCM is the symmetric cipher used to encrypt each individual +message. It is an AEAD (Authenticated Encryption with Associated Data) +cipher, meaning it provides both confidentiality (nobody can read the +message without the key) and integrity (nobody can modify the message +without detection) in a single operation. + +AES-256-GCM combines the AES block cipher in Counter mode (CTR) for +encryption with GHASH for authentication. The "256" refers to the key +size (256 bits / 32 bytes). The "GCM" stands for Galois/Counter Mode. + +Each message key derived from the Double Ratchet's symmetric chain is +used as the AES-256-GCM key. A fresh random nonce (also called IV - +initialization vector) is generated for every message. The ciphertext +includes a 128-bit authentication tag that detects any tampering. + +### How It's Used + +The encryption flow is: + +1. The Double Ratchet derives a message key (32 bytes) via KDF_CK +2. A random 12-byte nonce is generated using `os.urandom` (backend) or + `crypto.getRandomValues` (frontend) +3. AES-256-GCM encrypts the plaintext using the message key and nonce +4. Associated data (sender and recipient identifiers) is authenticated + but not encrypted +5. The output is ciphertext + a 16-byte authentication tag (GCM appends + the tag to the ciphertext) + +Backend implementation at `double_ratchet.py:111-130`: + +```python +def _encrypt_with_message_key(self, message_key, plaintext, associated_data): + aesgcm = AESGCM(message_key) + nonce = os.urandom(AES_GCM_NONCE_SIZE) # 12 bytes from config.py:69 + ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data) + return nonce, ciphertext +``` + +Backend decryption at `double_ratchet.py:132-153` catches `InvalidTag` +exceptions (line 151), which indicate that the ciphertext was tampered +with, the wrong key was used, or the associated data does not match. +The error is re-raised as `ValueError("Message tampered or corrupted")` +at line 153. + +Frontend implementation at `primitives.ts:224-259` uses the WebCrypto +API: + +```typescript +const nonce = generateRandomBytes(AES_GCM_NONCE_SIZE) +const ciphertext = await subtle.encrypt( + { + name: "AES-GCM", + iv: nonce.buffer, + additionalData: associatedData?.buffer, + }, + cryptoKey, + plaintext.buffer +) +``` + +Frontend decryption at `primitives.ts:261-292` mirrors this with +`subtle.decrypt`. The WebCrypto API throws a `DOMException` if +authentication fails, which is functionally equivalent to the Python +`InvalidTag` exception. + +### Why Not Just AES-CBC? + +AES-CBC (Cipher Block Chaining) is the other commonly seen AES mode. It +provides confidentiality but not integrity. A CBC-encrypted message can +be modified by an attacker (bit-flipping attacks) without the recipient +detecting the modification. To add integrity, you need a separate HMAC +computation (Encrypt-then-MAC or MAC-then-Encrypt), which adds +complexity and opportunities for implementation errors. + +GCM handles both in a single operation. It also has practical +performance advantages: the CTR-mode encryption in GCM is parallelizable +across CPU cores and benefits from AES-NI hardware instructions, while +CBC is inherently sequential (each block depends on the previous +ciphertext block). + +AES-GCM is the NIST-recommended mode for new applications (NIST SP +800-38D). It is the mandatory cipher suite in TLS 1.3. There is no +security reason to prefer CBC over GCM for new implementations. + +### Parameters + +As defined in `config.py:68-70`: + +``` +AES_GCM_KEY_SIZE = 32 (256 bits) line 68 +AES_GCM_NONCE_SIZE = 12 (96 bits) line 69 +HKDF_OUTPUT_SIZE = 32 (256 bits) line 70 +``` + +The 12-byte (96-bit) nonce is the recommended size for GCM. Longer +nonces are allowed by the specification but require additional internal +processing. The authentication tag is 128 bits (16 bytes), which is the +full-length tag and the default for both the Python `cryptography` +library and WebCrypto. + +With a 256-bit key, AES-256-GCM provides 128-bit security against key +recovery attacks (Grover's algorithm would reduce AES-256 to 128-bit +security on a quantum computer, but AES-128 would drop to 64-bit, which +is why 256-bit keys are the forward-looking choice). + +The nonce must be unique per key. Because each message key from the +Double Ratchet is used exactly once, the nonce uniqueness requirement is +automatically satisfied even if the random number generator produced a +collision: the same nonce with a different key is not a problem. This is +a subtle but important point. The Double Ratchet's key-per-message +design means that nonce reuse (which would be catastrophic with a static +key) is not a realistic concern here. + + +## WebAuthn / Passkeys + +### What It Is + +WebAuthn (Web Authentication) is a W3C standard (first published in +March 2019, Level 2 in April 2021, Level 3 in progress) for +passwordless authentication using public key cryptography. Instead of +passwords, users authenticate using asymmetric key pairs managed by an +authenticator: a hardware security key (YubiKey, SoloKey), a platform +authenticator (Touch ID, Face ID, Windows Hello, Android biometrics), or +a cross-platform authenticator accessed through a phone. + +The term "Passkey" refers to a discoverable credential (also called a +"resident key") that is synced across devices through a platform +credential manager (iCloud Keychain, Google Password Manager, 1Password, +etc.). Passkeys are the consumer-friendly branding for WebAuthn +discoverable credentials. + +The key insight is that with WebAuthn, the private key never leaves the +authenticator. The server stores only the public key and a credential +ID. Authentication is a challenge-response protocol: the server sends a +random challenge, the authenticator signs it with the private key, and +the server verifies the signature with the stored public key. No secret +is transmitted, no secret is stored on the server, and there is nothing +for an attacker to steal from the server that would allow them to +impersonate the user. + +### Why Not Passwords? + +Passwords are the primary attack vector for authentication compromise. +Here is why, specifically in the context of an encrypted messaging +application: + +**Phishing.** An attacker creates a convincing replica of the login page +and tricks the user into entering their password. With WebAuthn, the +authenticator cryptographically binds the credential to the origin +(domain name) of the website. If the user visits `evil-chat.com` +instead of `real-chat.com`, the authenticator will not use the credential +for `real-chat.com` because the origin does not match. The user cannot +be tricked into authenticating to the wrong site because the +authenticator will simply not respond to the challenge. This is +automatic and requires no user awareness of the attack. + +**Credential stuffing.** Users reuse passwords across services. When one +service is breached (and breaches of password databases happen +constantly -- Collection #1 in 2019 exposed 773 million email/password +pairs), attackers try those passwords against other services. WebAuthn +credentials are unique per relying party (website). There is no password +to reuse. + +**Keyloggers.** Malware that captures keystrokes can record passwords as +users type them. WebAuthn authentication uses biometric verification +(fingerprint, face) or PIN entry on the authenticator device, not the +keyboard. Even if a keylogger captured a PIN, the PIN alone is useless +without physical possession of the authenticator device. + +**Server compromise.** If a server stores password hashes and the +database is stolen, attackers can attempt offline cracking. The 2023 +LastPass breach exposed encrypted password vaults for 25+ million users; +users with weak master passwords had their vaults cracked. With +WebAuthn, the server stores only public keys. Stealing public keys gives +the attacker nothing: you cannot derive a private key from a public key, +and you cannot forge a signature without the private key. + +### Registration Flow + +``` +Step 1: Client requests registration options + Client ----> POST /auth/register/begin ----> Server + + Server: + - Generates 32-byte random challenge (secrets.token_bytes) + - Stores challenge in Redis with 10-minute TTL + - Returns WebAuthn PublicKeyCredentialCreationOptions + +Step 2: Browser creates credential + Browser ----> navigator.credentials.create(options) ----> Authenticator + + Authenticator: + - Prompts user for biometric/PIN verification + - Generates new ECDSA or EdDSA keypair + - Stores private key internally (NEVER exported) + - Returns attestation object (signed credential public key) + +Step 3: Client sends attestation for verification + Client ----> POST /auth/register/complete ----> Server + + Server: + - Verifies attestation signature + - Verifies challenge matches stored value + - Extracts credential public key and credential ID + - Stores PUBLIC KEY + credential ID in PostgreSQL + - Deletes challenge from Redis + - Private key STAYS on authenticator -- server never sees it +``` + +Implementation reference: `passkey_manager.py:55-94` +(`generate_registration_options`). At line 65, a 32-byte challenge is +generated: `challenge = secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)`. +At lines 74-87, the WebAuthn options are constructed with RP +configuration, user information, and authenticator requirements. The +authenticator selection at lines 82-85 specifies +`ResidentKeyRequirement.REQUIRED`, which forces creation of a +discoverable credential (passkey). + +Registration verification at `passkey_manager.py:96-130` +(`verify_registration`) calls `verify_registration_response` at lines +105-110, which validates the attestation object, checks the challenge, +verifies the RP ID, and confirms the origin. + +### Authentication Flow + +``` +Step 1: Client requests authentication options + Client ----> POST /auth/authenticate/begin ----> Server + + Server: + - Generates new 32-byte challenge + - Stores challenge in Redis with 10-minute TTL + - Returns WebAuthn PublicKeyCredentialRequestOptions + +Step 2: Browser signs challenge + Browser ----> navigator.credentials.get(options) ----> Authenticator + + Authenticator: + - Prompts user for biometric/PIN + - Signs challenge with stored private key + - Increments signature counter + - Returns assertion (signed challenge + counter) + +Step 3: Client sends assertion for verification + Client ----> POST /auth/authenticate/complete ----> Server + + Server: + - Retrieves stored public key from PostgreSQL + - Verifies signature using stored public key + - Verifies challenge matches stored value + - Checks signature counter INCREASED (clone detection!) + - Updates stored counter value + - Returns authenticated session +``` + +Implementation reference: `passkey_manager.py:132-160` +(`generate_authentication_options`). At line 139, a fresh challenge is +generated. At lines 148-153, WebAuthn authentication options are +constructed. + +Authentication verification at `passkey_manager.py:162-207` +(`verify_authentication`). At lines 173-180, the assertion is verified +against the expected challenge, RP ID, origin, and stored credential +public key. The critical clone detection check follows. + +### Clone Detection + +Hardware authenticators (YubiKeys, Titan keys, etc.) maintain an +internal signature counter that increments every time the authenticator +is used. This counter is included in the signed assertion data. The +server stores the latest counter value and checks that each new +authentication presents a higher counter. + +If the server receives an assertion with a counter value that has not +increased (or has decreased), it indicates one of two things: + +1. The authenticator hardware was cloned (its key material was + extracted and loaded onto a second device) +2. A replay attack is being attempted + +Both scenarios are security incidents that warrant blocking +authentication and alerting the user. + +Reference: `passkey_manager.py:184-193`: + +```python +if (credential_current_sign_count != 0 and new_sign_count != 0 + and new_sign_count <= credential_current_sign_count): + logger.error( + "Signature counter did not increase: current=%s, new=%s. " + "Possible cloned authenticator detected!", + credential_current_sign_count, + new_sign_count + ) + raise ValueError( + "Signature counter anomaly detected - potential cloned authenticator" + ) +``` + +The conditions `credential_current_sign_count != 0` and +`new_sign_count != 0` are defensive: some authenticators (particularly +platform authenticators and passkeys) always report a counter of 0, +indicating that they do not implement counter-based clone detection. +For those authenticators, the clone detection check is skipped because +it would always trigger a false positive. This is consistent with the +WebAuthn specification's guidance on handling authenticators that do not +support signature counters. + + +## Constant-Time Comparison + +A detail worth calling out: `primitives.ts:388-397` implements a +constant-time byte array comparison: + +```typescript +export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean { + if (a.length !== b.length) return false + let result = 0 + for (let i = 0; i < a.length; i++) { + result |= a[i] ^ b[i] + } + return result === 0 +} +``` + +This function compares two byte arrays in constant time, meaning the +execution time does not depend on where the first difference occurs. +A naive comparison (`a[i] !== b[i]` with an early return) leaks +information through timing: if the first byte differs, the function +returns immediately, and an attacker measuring response time can deduce +that the first byte was wrong. By iterating through all bytes and +OR-ing the XOR results, the function always takes the same amount of +time regardless of whether the arrays match at byte 0 or byte 31. + +Timing side-channel attacks are not theoretical. In 2009, Nate Lawson +and Taylor Nelson demonstrated practical timing attacks against HMAC +verification in a web application framework, recovering the correct HMAC +one byte at a time. The constant-time comparison eliminates this attack +vector entirely. + + +## How These Concepts Relate + +The following diagram shows how the five major concepts connect to form +the full security architecture of the application: + +``` ++-----------------------------------------------------------+ +| AUTHENTICATION LAYER | +| | +| WebAuthn/Passkeys | +| | | +| +--> User identity established | +| +--> No password to steal, phish, or brute force | +| +--> Clone detection via signature counter | +| | | +| v | +| +----------------------------------------------------+ | +| | KEY AGREEMENT LAYER | | +| | | | +| | X3DH Key Exchange | | +| | +--> Asynchronous (works when peer is offline) | | +| | +--> 4 DH operations for mutual authentication | | +| | +--> Produces initial shared secret (32 bytes) | | +| | | | | +| | v | | +| | Double Ratchet Initialization | | +| | +--> Shared secret becomes root key | | +| | +--> Sending and receiving chains created | | +| +----------------------------------------------------+ | +| | | +| v | +| +----------------------------------------------------+ | +| | MESSAGE ENCRYPTION LAYER | | +| | | | +| | Double Ratchet (ongoing) | | +| | +--> KDF_CK derives per-message keys | | +| | +--> DH ratchet provides post-compromise security| | +| | +--> Skipped key cache handles out-of-order msgs | | +| | | | | +| | v | | +| | AES-256-GCM | | +| | +--> Encrypts plaintext with message key | | +| | +--> Random 12-byte nonce per message | | +| | +--> Authentication tag detects tampering | | +| +----------------------------------------------------+ | +| | | +| v | +| +----------------------------------------------------+ | +| | TRANSPORT LAYER | | +| | | | +| | WebSocket (real-time delivery) | | +| | +--> Carries encrypted blobs between clients | | +| | +--> Server sees only ciphertext + metadata | | +| | | | +| | SurrealDB (persistence) | | +| | +--> Stores encrypted messages at rest | | +| | +--> No decryption capability on server | | +| +----------------------------------------------------+ | ++-----------------------------------------------------------+ +``` + +The layers interact in a strict top-down sequence for new conversations: + +1. The user authenticates with WebAuthn (proving their identity without + a password) +2. X3DH establishes a shared secret with the peer (even if the peer is + offline) +3. The shared secret initializes the Double Ratchet + (`double_ratchet.py:279-302`) +4. Each message is encrypted with a unique AES-256-GCM key derived from + the ratchet (`double_ratchet.py:323-362`) +5. The encrypted message is transmitted via WebSocket and stored in + SurrealDB (`message_service.py:269-314`) + +For ongoing conversations, only steps 4 and 5 repeat. The X3DH +handshake happens once per conversation. The Double Ratchet then runs +autonomously, deriving fresh keys for every message without any further +server interaction for key management. + + +## Industry Standards + +This section maps the project's security measures to specific industry +frameworks. These mappings are useful for compliance discussions, +security audits, and threat modeling. + +### OWASP Top 10 (2021) + +**A02: Cryptographic Failures** -- Formerly "Sensitive Data Exposure," +this category covers failures in cryptographic implementation. This +project addresses it through: +- AES-256-GCM with HKDF-SHA256 key derivation (no weak algorithms) +- 256-bit keys meeting NIST minimum requirements for post-2030 use +- Per-message unique keys (no key reuse) +- Proper nonce generation via CSPRNG (`os.urandom`, `crypto.getRandomValues`) +- No storage of plaintext on the server + +**A04: Insecure Design** -- Covers architecture-level security +weaknesses. The zero-knowledge architecture prevents entire classes of +server-side attacks: +- Server compromise does not reveal message content +- No server-side decryption keys to steal +- Prekey bundles contain only public key material +- Store-and-forward model treats all messages as opaque blobs + +**A07: Identification and Authentication Failures** -- Covers broken +authentication. WebAuthn eliminates the most common authentication +attacks: +- No passwords means no credential stuffing (CWE-521) +- Origin binding prevents phishing (CWE-352) +- Signature counters detect cloned authenticators +- Challenge-response prevents replay attacks + +### MITRE ATT&CK + +**T1557: Adversary-in-the-Middle** -- An attacker intercepts +communications between two parties. E2E encryption with X3DH mutual +authentication prevents meaningful MITM attacks. Even if an attacker +controls the network path, they see only AES-256-GCM ciphertext. The +X3DH signed prekey verification (using Ed25519 signatures) prevents the +attacker from substituting their own keys. + +**T1528: Steal Application Access Token** -- An attacker steals an +authentication token to impersonate a user. WebAuthn credentials are +cryptographically bound to the relying party origin. A token stolen from +one site cannot be used on another. The private key never leaves the +authenticator, so there is no token to steal from the server or from +network traffic. + +**T1110: Brute Force** -- An attacker attempts to guess credentials +through exhaustive trial. With WebAuthn, there are no passwords to brute +force. Authentication requires physical possession of the authenticator +and biometric verification (or PIN), making remote brute force +impossible. The challenge changes with every authentication attempt, +preventing replay of captured assertions. + +### CWE References + +**CWE-327: Use of a Broken or Risky Cryptographic Algorithm** -- This +project uses exclusively NIST-approved and widely-vetted algorithms: +X25519 (Curve25519 ECDH), Ed25519 (EdDSA), AES-256-GCM, HMAC-SHA256, +HKDF-SHA256. No custom cryptographic primitives are implemented. Both +the Python `cryptography` library and the browser WebCrypto API provide +well-tested implementations. + +**CWE-326: Inadequate Encryption Strength** -- 256-bit symmetric keys +(AES-256) and 256-bit elliptic curve keys (X25519, Ed25519) provide a +128-bit security level, which exceeds NIST's minimum recommendation of +112 bits for use through 2030+ (NIST SP 800-57 Part 1). + +**CWE-330: Use of Insufficiently Random Values** -- Randomness comes +from two sources: `os.urandom()` on the backend (which reads from the +operating system's CSPRNG -- `/dev/urandom` on Linux) and +`crypto.getRandomValues()` on the frontend (which uses the browser's +CSPRNG). Both are cryptographically secure. Nonce generation at +`double_ratchet.py:122` and `primitives.ts:294-297` use these +exclusively. + +**CWE-311: Missing Encryption of Sensitive Data** -- All message content +is encrypted at rest (SurrealDB stores only ciphertext) and in transit +(WebSocket over TLS carries AES-256-GCM ciphertext). The Double Ratchet +state itself is serialized and stored, but this state does not contain +any plaintext; it contains key material for future messages. + + +## Real-World Case Studies + +### Case Study 1: The 2020 Zoom E2E Encryption Controversy + +**Timeline.** In March 2020, The Intercept published an investigation +revealing that Zoom's claims of "end-to-end encryption" were false. The +Citizen Lab at the University of Toronto published a follow-up report in +April 2020, identifying that Zoom used AES-128 in ECB mode and that +encryption keys were generated by Zoom's servers and transmitted to +participants through Zoom's infrastructure. In November 2020, the FTC +issued a complaint, and Zoom agreed to a settlement requiring them to +implement a comprehensive security program and cease misrepresenting +their encryption. + +**What failed architecturally.** Zoom's design placed the encryption +keys on the server. The connection between the client and the server was +encrypted (transport encryption via TLS), and the media streams between +the server and the participants were encrypted with AES-128, but the +server generated and held all key material. This meant Zoom's servers +could decrypt every call. The use of ECB mode (Electronic Codebook) was +an additional failure: ECB encrypts each block independently, meaning +identical plaintext blocks produce identical ciphertext blocks, which +leaks structural patterns in the data. ECB has been considered insecure +for decades and is explicitly warned against in every modern +cryptography textbook. + +**How this project prevents the same failure.** In this project, the +server never generates or holds encryption keys for message content. +Key generation happens in two places: + +1. On the backend, `x3dh_manager.py:60-86` generates X25519 identity + keypairs using `X25519PrivateKey.generate()`, which calls into + OpenSSL's random number generator. These keys are for the X3DH + protocol, and the private keys are stored in the database for the + server-side key exchange path. + +2. On the frontend, `primitives.ts:15-24` generates X25519 keypairs + using the WebCrypto API (`subtle.generateKey`), which uses the + browser's hardware-backed CSPRNG. In the client-side encryption + model, these keys never leave the browser. + +The `store_encrypted_message` function (`message_service.py:269-314`) +receives pre-encrypted ciphertext from the client and stores it directly +in SurrealDB at lines 292-302 without any server-side decryption. The +server's role is explicitly that of a blind relay. Even if the entire +server infrastructure were compromised, the attacker would obtain only +encrypted blobs with no corresponding decryption keys. + +### Case Study 2: Signal Protocol Adoption by WhatsApp (2016) + +**Background.** In November 2014, Open Whisper Systems announced a +partnership with WhatsApp to integrate the Signal Protocol into the +WhatsApp messaging client. The rollout happened incrementally: +TextSecure's Axolotl protocol (later renamed to the Signal Protocol) +was first deployed for Android-to-Android messages, then extended to +group messages, media, and voice calls on all platforms. Full deployment +was announced in April 2016, making WhatsApp the largest deployment of +E2E encryption in history, covering over 1 billion users at the time. + +**Technical details.** WhatsApp implemented the same X3DH + Double +Ratchet combination used in this project. Each message gets a unique +AES key through the ratchet mechanism, providing forward secrecy across +billions of daily messages. WhatsApp's implementation stores prekey +bundles on their servers (just as this project does via +`prekey_service.py:293-361`), allowing asynchronous session +establishment. The X3DH handshake is performed when a user initiates a +new conversation, and the Double Ratchet runs continuously thereafter. + +**Impact.** When the Brazilian government ordered WhatsApp to provide +message content in 2016, WhatsApp demonstrated that they architecturally +could not comply: they did not possess the decryption keys. This was not +a policy decision or a promise; it was a mathematical fact. The protocol +makes it provably impossible for the server to decrypt messages. The +same situation occurred with FBI requests in the United States and +government demands in India and the UK. + +**Connection to this project.** The X3DH implementation at +`x3dh_manager.py:208-281` and the Double Ratchet at +`double_ratchet.py:279-416` implement the same cryptographic operations +described in the Signal Protocol specification. The same four DH +operations (lines 241-252), the same HKDF derivation (lines 257-264), +the same KDF chain operations (lines 79-109 of double_ratchet.py), and +the same skipped message key mechanism (lines 215-277). The protocol +specifications are public, the formal security proofs are published, and +the implementation follows them directly. + +### Case Study 3: The 2023 LastPass Breach + +**Timeline.** In August 2022, an attacker compromised a LastPass +developer's workstation through a vulnerable third-party media software +package. Using the developer's credentials, the attacker accessed +LastPass's development environment and stole source code and technical +information. In a second incident, the attacker used information from +the first breach to target a DevOps engineer's home computer, installing +a keylogger that captured the engineer's master password for a LastPass +corporate vault. With this access, the attacker exfiltrated encrypted +customer password vaults and backup data from LastPass's cloud storage. + +**What was exposed.** The encrypted vaults for approximately 25.6 +million users were stolen. While the vaults were encrypted with AES-256 +using each user's master password as the key derivation input, the +security of the vaults depended entirely on the strength of the user's +master password. Users with short, common, or previously-breached master +passwords had their vaults cracked through offline brute-force attacks. +The breach also exposed unencrypted metadata including website URLs, +which revealed which services each user had accounts with. + +**Connection to this project.** The LastPass breach demonstrates +precisely why WebAuthn/Passkeys are superior to password-based +authentication, even when the passwords are used to derive encryption +keys. + +In this project's WebAuthn implementation +(`passkey_manager.py:55-94`), there is no password. The user +authenticates with a biometric or PIN on their authenticator device. The +authenticator holds an ECDSA or EdDSA private key that is hardware-bound +and never exported. If the server database is fully compromised, the +attacker obtains only credential public keys and credential IDs. Public +keys cannot be reversed to obtain private keys (this would require +solving the elliptic curve discrete logarithm problem, which is +computationally infeasible). There is no master password to brute force, +no password hash to crack, and no password-equivalent secret stored on +the server. + +The contrast is stark: LastPass stored user secrets protected by a +user-chosen password. This project stores user secrets protected by a +hardware-bound private key that the user cannot choose, cannot weaken, +and cannot accidentally reuse on another site. + + +## Testing Your Understanding + +Before moving on to the architecture document, you should be able to +answer these questions. If you cannot answer one confidently, re-read the +relevant section. + +1. **Why does X3DH need four separate DH operations instead of just + one?** What specific security property does each operation provide? + What attack becomes possible if any single operation is removed? + +2. **If an attacker compromises a Double Ratchet chain key at message N, + which messages can they decrypt?** Which messages remain protected? + When does the attacker lose access? Trace through the KDF_CK function + to prove your answer. + +3. **Why is WebAuthn resistant to phishing attacks, while traditional + passwords are not?** What property of the WebAuthn protocol prevents + a fake login page from capturing usable credentials? Hint: think + about what the authenticator checks before signing the challenge. + +4. **What happens if Alice sends Bob messages 1, 2, 3, 4, 5 and Bob + receives them in order 1, 3, 5, 2, 4?** Walk through the skipped + message key mechanism for each received message. How many keys are + cached after processing message 3? After processing message 5? + +5. **Why does the Double Ratchet use two different HMAC constants (0x01 + and 0x02) in KDF_CK?** What would go wrong if both the chain key and + the message key were derived with the same constant? + + +## Further Reading + +### Essential Specifications + +- The Signal Protocol specifications: https://signal.org/docs/ +- X3DH specification: https://signal.org/docs/specifications/x3dh/ +- Double Ratchet specification: https://signal.org/docs/specifications/doubleratchet/ +- WebAuthn Level 3 specification: https://www.w3.org/TR/webauthn-3/ + +### Academic Analysis + +- "A Formal Security Analysis of the Signal Messaging Protocol" -- + Cohn-Gordon, Cremers, Dowling, Garratt, Stebila. IEEE EuroS&P 2017. + Formal proof that the Signal Protocol meets its claimed security + properties (authenticated key exchange, forward secrecy, post-compromise + security) under the Gap-DH assumption. + +- "The Signal Protocol: A Cryptographic Analysis" -- Cohn-Gordon et al. + (2017). Extended version with proofs covering the X3DH and Double + Ratchet components individually and composed. + +- "On Ends-to-Ends Encryption: Asynchronous Group Messaging with Strong + Security Guarantees" -- Cohn-Gordon et al. IEEE S&P 2018. Extends the + analysis to group messaging scenarios. + +### FIDO and WebAuthn + +- FIDO2 technical overview: https://fidoalliance.org/fido2/ +- FIDO Alliance whitepaper on passkeys: + https://fidoalliance.org/passkeys/ +- NIST SP 800-63B Digital Identity Guidelines (authenticator types and + assurance levels) + +### Cryptographic Primitives + +- Daniel J. Bernstein, "Curve25519: new Diffie-Hellman speed records" + (2006) -- The paper introducing the X25519 function used for DH in + this project. + +- Daniel J. Bernstein, Niels Duif, Tanja Lange, Peter Schwabe, Bo-Yin + Yang, "High-speed high-security signatures" (2012) -- The paper + introducing Ed25519 used for signing prekeys. + +- NIST SP 800-38D, "Recommendation for Block Cipher Modes of Operation: + Galois/Counter Mode (GCM) and GMAC" -- The specification for the + AES-GCM mode used for message encryption. + +- Hugo Krawczyk, "Cryptographic Extraction and Key Derivation: The HKDF + Scheme" (2010) -- The paper behind HKDF, the key derivation function + used throughout the Double Ratchet. + +### Historical Context + +- Nikita Borisov, Ian Goldberg, Eric Brewer, "Off-the-Record + Communication, or, Why Not To Use PGP" (2004) -- The OTR protocol + that introduced the concept of deniable, forward-secret messaging and + directly inspired the Signal Protocol's design. + +- Whitfield Diffie and Martin Hellman, "New Directions in Cryptography" + (1976) -- The original paper introducing public key cryptography and + the Diffie-Hellman key exchange that underlies X3DH. + +- Phil Zimmermann and PGP: The "Crypto Wars" of the 1990s, where the + US government attempted to restrict the export of strong cryptography. + Zimmermann published PGP's source code in a printed book to circumvent + export controls under First Amendment protection. This history is why + cryptographic software can be freely distributed today. diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md new file mode 100644 index 00000000..d87f8d60 --- /dev/null +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/02-ARCHITECTURE.md @@ -0,0 +1,1432 @@ +# System Architecture + +This document describes the full architecture of the encrypted P2P chat application. It implements a subset of the Signal Protocol (X3DH + Double Ratchet) for end-to-end encryption, with WebAuthn/Passkeys for passwordless authentication. The server is a relay that stores ciphertext blobs and never sees plaintext message content. + +Everything that follows is grounded in the actual source code. File references use the format `filename.py:line-range` relative to the backend `app/` or frontend `src/` directory. + +--- + +## High-Level Architecture + +``` + HTTPS (TLS) + ┌────────────────────────────────────────────────────────────┐ + │ Client Browser │ + │ ┌──────────────────────────────────────────────────────┐ │ + │ │ SolidJS 1.9 + TypeScript 5.9 │ │ + │ │ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │ │ + │ │ │ WebCrypto │ │ IndexedDB │ │ nanostores │ │ │ + │ │ │ X25519 │ │ Private │ │ Reactive │ │ │ + │ │ │ Ed25519 │ │ Keys │ │ State │ │ │ + │ │ │ AES-256-GCM │ │ Ratchet │ │ │ │ │ + │ │ │ HKDF-SHA256 │ │ States │ │ │ │ │ + │ │ └──────────────┘ └──────────────┘ └────────────┘ │ │ + │ └──────────────────────────┬───────────────────────────┘ │ + │ │ │ + └──────────────────────────────┼──────────────────────────────┘ + │ + HTTPS + WSS │ + │ + ┌──────────────────────────────┼──────────────────────────────┐ + │ Nginx Reverse Proxy │ │ + │ ┌───────────────────────────┴──────────────────────────┐ │ + │ │ /api/* ──────► HTTP ──────► FastAPI :8000 │ │ + │ │ /ws ──────► WS ──────► FastAPI :8000 │ │ + │ │ /* ──────► Static files (SolidJS build) │ │ + │ └──────────────────────────────────────────────────────┘ │ + └──────────────────────────────┬──────────────────────────────┘ + │ + ┌──────────────────────────────┼──────────────────────────────┐ + │ FastAPI Backend │ │ + │ (Python 3.13, ORJSONResponse, async/await throughout) │ + │ ┌──────────────────────────────────────────────────────┐ │ + │ │ Routers: auth, rooms, encryption, websocket │ │ + │ │ Services: auth, prekey, message, presence, websocket│ │ + │ │ Core: x3dh, double_ratchet, passkey, surreal, redis │ │ + │ └────────┬──────────────┬──────────────┬───────────────┘ │ + │ │ │ │ │ + └───────────┼──────────────┼──────────────┼───────────────────┘ + │ │ │ + asyncpg │ WebSocket │ redis. │ + (TCP 5432) │ (WS 8000) │ asyncio │ + │ │ (TCP 6379) │ + ┌───────────┴──┐ ┌───────┴──────┐ ┌───┴────────────┐ + │ PostgreSQL │ │ SurrealDB │ │ Redis 8 │ + │ 16-alpine │ │ (latest) │ │ (alpine) │ + │ │ │ │ │ │ + │ Auth data │ │ Messages │ │ Challenges │ + │ Credentials │ │ Presence │ │ Rate limits │ + │ X3DH keys │ │ Rooms │ │ │ + │ Ratchet │ │ Live │ │ TTL-based │ + │ states │ │ queries │ │ auto-expiry │ + └──────────────┘ └──────────────┘ └────────────────┘ +``` + +A few things worth noting about this diagram: + +1. All three databases serve distinct purposes. PostgreSQL handles relational data that requires ACID transactions (users, credentials, cryptographic keys). SurrealDB handles real-time data that benefits from live query push notifications (messages, presence). Redis handles ephemeral data that needs automatic expiry (WebAuthn challenges, rate limit counters). + +2. The WebSocket connection between client and FastAPI is the primary channel for real-time messaging. HTTP is used only for auth flows and key management. + +3. The client does all encryption and decryption. The server receives ciphertext and stores it as-is. This is the central design constraint that everything else follows from. + +--- + +## Component Breakdown + +### FastAPI Backend + +**Purpose:** API server that handles authentication, key management, message relay, and WebSocket connections. + +**Key File:** `factory.py` + +The application uses the factory pattern. `create_app()` at lines 63-115 builds the FastAPI instance: + +``` +create_app() (factory.py:63-115) + ├── FastAPI( + │ title = settings.APP_NAME, + │ default_response_class = ORJSONResponse, + │ lifespan = lifespan, + │ ) + ├── CORSMiddleware (factory.py:78-85) + ├── GZipMiddleware, minimum_size=1000 (factory.py:87) + ├── register_exception_handlers (factory.py:89) + ├── Root endpoint "/" (factory.py:91-101) + ├── Health endpoint "/health" (factory.py:103-108) + └── Routers: + ├── auth_router → /auth/* + ├── rooms_router → /rooms/* + ├── encryption_router → /encryption/* + └── websocket_router → /ws +``` + +The lifespan manager at lines 39-61 runs database connections on startup and disconnections on shutdown: + +``` +lifespan(app) (factory.py:39-61) + Startup: + 1. await init_db() → PostgreSQL tables via SQLModel + 2. await redis_manager.connect() → Redis connection pool + 3. await surreal_db.connect() → SurrealDB WebSocket connection + Shutdown: + 1. await redis_manager.disconnect() + 2. await surreal_db.disconnect() +``` + +The startup order matters. PostgreSQL is initialized first because the auth tables must exist before any request processing. Redis and SurrealDB follow because they depend on having a running application context. If any of these fail, the application does not start. + +**Interfaces:** + +| Route Prefix | Purpose | Key Endpoints | +|---|---|---| +| `/auth` | WebAuthn registration + login | `register/begin`, `register/complete`, `authenticate/begin`, `authenticate/complete`, `users/search` | +| `/encryption` | X3DH key management | `prekey-bundle/{id}`, `upload-keys/{id}`, `initialize-keys/{id}`, `rotate-signed-prekey/{id}`, `opk-count/{id}` | +| `/rooms` | Chat room CRUD | Room creation, listing, participant management | +| `/ws` | WebSocket endpoint | Real-time messaging, typing, presence, receipts | + +--- + +### WebSocket Layer + +**Purpose:** Real-time bidirectional communication for messaging, typing indicators, presence updates, and read receipts. + +**Key Files:** `core/websocket_manager.py`, `api/websocket.py`, `services/websocket_service.py` + +The `ConnectionManager` class (`websocket_manager.py:31-296`) is the center of the real-time system. It maintains three dictionaries: + +```python +self.active_connections: dict[UUID, list[WebSocket]] = {} # line 39 +self.live_query_ids: dict[UUID, str] = {} # line 40 +self.heartbeat_tasks: dict[UUID, asyncio.Task] = {} # line 41 +``` + +The first maps user IDs to lists of WebSocket connections. Each user can have up to 5 simultaneous connections (`WS_MAX_CONNECTIONS_PER_USER = 5`, `config.py:141`). This supports multi-device usage without exhausting server resources. When a sixth connection attempt arrives, the manager sends an error and closes the socket (`websocket_manager.py:52-64`). + +**Connection lifecycle:** + +``` +connect(websocket, user_id) (websocket_manager.py:43-95) + 1. Accept WebSocket + 2. Check connection limit (max 5 per user) + 3. Add to active_connections pool + 4. Set user online via presence_service + 5. Start heartbeat loop (30s interval) + 6. Subscribe to SurrealDB live query for user's messages + 7. Return True + +disconnect(websocket, user_id) (websocket_manager.py:97-133) + 1. Remove WebSocket from user's connection list + 2. If last connection: + a. Set user offline via presence_service + b. Kill SurrealDB live query + c. Cancel heartbeat task +``` + +**Heartbeat:** The `_heartbeat_loop` method (`websocket_manager.py:177-201`) sends a ping every `WS_HEARTBEAT_INTERVAL` seconds (30s, from `config.py:140`). If the send fails, the connection is marked dead and disconnected. This catches silently dropped connections that TCP keepalive alone would miss. + +**Live query subscription:** When a user connects, `_subscribe_to_messages` (`websocket_manager.py:203-223`) registers a SurrealDB live query that watches for new messages where the user is the recipient. SurrealDB pushes new messages to the server in real-time through this subscription. The callback wraps updates and dispatches them through `_handle_live_message` (`websocket_manager.py:225-251`), which forwards the encrypted payload to all of the user's active WebSocket connections. + +**Message routing:** The WebSocket endpoint (`websocket.py:25-84`) is thin. It accepts connections, delegates to `connection_manager.connect()`, and then sits in a loop reading JSON messages. Each message gets routed by `websocket_service.route_message()`, which dispatches based on the `type` field: + +``` +route_message(websocket, user_id, message) (websocket_service.py:40-85) + ├── "encrypted_message" → handle_encrypted_message() + ├── "typing" → handle_typing_indicator() + ├── "presence" → handle_presence_update() + ├── "receipt" → handle_read_receipt() + ├── "heartbeat" → handle_heartbeat() + └── unknown → send error response +``` + +**Dead connection cleanup:** When `send_message()` (`websocket_manager.py:135-153`) fails to send to a connection, it collects the dead connection and calls `disconnect()` on it. This prevents stale connections from accumulating. + +--- + +### Encryption Engine + +**Purpose:** Implements the Signal Protocol cryptographic primitives for end-to-end encryption. + +**Key Files:** `core/encryption/x3dh_manager.py`, `core/encryption/double_ratchet.py`, `frontend/src/crypto/primitives.ts` + +The encryption system has two layers: + +1. **X3DH (Extended Triple Diffie-Hellman):** Establishes a shared secret between two users who have never communicated before, even if one of them is offline. + +2. **Double Ratchet:** Uses the X3DH shared secret to derive per-message encryption keys with forward secrecy. Every message gets a unique key, and compromising one key does not reveal past or future messages. + +**X3DHManager** (`x3dh_manager.py:56-353`) handles: +- Key generation for identity keys (X25519 for DH, Ed25519 for signing) +- Signed prekey generation with Ed25519 signatures (`x3dh_manager.py:116-152`) +- One-time prekey generation (`x3dh_manager.py:154-174`) +- Signed prekey verification (`x3dh_manager.py:176-206`) +- Sender-side X3DH exchange (`x3dh_manager.py:208-281`) +- Receiver-side X3DH exchange (`x3dh_manager.py:283-350`) + +The X3DH exchange on the sender side (`perform_x3dh_sender`, lines 208-281) works like this: + +``` +Alice wants to message Bob (who might be offline): + +1. Alice fetches Bob's prekey bundle from server: + - Bob's identity key (IK_B) + - Bob's signed prekey (SPK_B) + signature + - Bob's one-time prekey (OPK_B), if available + +2. Alice verifies SPK_B signature using Bob's Ed25519 identity key + +3. Alice generates ephemeral keypair (EK_A) + +4. Four DH operations: + DH1 = X25519(IK_A_private, SPK_B) # Alice identity x Bob signed prekey + DH2 = X25519(EK_A_private, IK_B) # Alice ephemeral x Bob identity + DH3 = X25519(EK_A_private, SPK_B) # Alice ephemeral x Bob signed prekey + DH4 = X25519(EK_A_private, OPK_B) # Alice ephemeral x Bob one-time prekey + # (only if OPK available) + +5. Concatenate: key_material = DH1 || DH2 || DH3 [|| DH4] + +6. Derive shared key via HKDF-SHA256: + shared_key = HKDF( + salt = 0x00 * 32, + ikm = 0xFF * 32 || key_material, + info = "X3DH", + length = 32 + ) + +7. Return: shared_key, associated_data (IK_A_pub || IK_B_pub), EK_A_pub +``` + +The `0xFF * 32` prefix before the key material is a domain separator. This matches the Signal specification and prevents potential cross-protocol attacks. + +**DoubleRatchet** (`double_ratchet.py:64-419`) handles: +- Sender initialization from X3DH shared secret (`double_ratchet.py:279-302`) +- Receiver initialization (`double_ratchet.py:304-321`) +- Message encryption with chain key advancement (`double_ratchet.py:323-362`) +- Message decryption with out-of-order support (`double_ratchet.py:364-416`) +- DH ratchet steps for forward secrecy (`double_ratchet.py:155-213`) +- Skipped message key storage for out-of-order delivery (`double_ratchet.py:215-258`) + +The ratchet uses two KDF chains: + +``` +Root Key Chain: + _kdf_rk(root_key, dh_output) → (new_root_key, new_chain_key) + Uses HKDF-SHA256 with root_key as salt (double_ratchet.py:79-94) + +Symmetric Key Chain: + _kdf_ck(chain_key) → (next_chain_key, message_key) + Uses HMAC-SHA256 with 0x01 for chain key, 0x02 for message key + (double_ratchet.py:96-109) +``` + +Each message is encrypted with AES-256-GCM using a 12-byte random nonce (`double_ratchet.py:111-130`). The message key is derived from the sending chain and used exactly once. After encryption, the sending chain advances, and the old message key is discarded. + +**Security limits from `config.py`:** +- `MAX_SKIP_MESSAGE_KEYS = 1000` (line 73): Maximum messages that can arrive out of order before the protocol rejects them +- `MAX_CACHED_MESSAGE_KEYS = 2000` (line 74): Maximum stored skipped keys before eviction +- `AES_GCM_NONCE_SIZE = 12` (line 69): 96-bit nonces for AES-GCM +- `HKDF_OUTPUT_SIZE = 32` (line 70): 256-bit derived keys + +**Client-side crypto:** The frontend `primitives.ts` (lines 1-397) mirrors the backend crypto using the WebCrypto API. It provides: +- `generateX25519KeyPair()` (line 15) +- `x25519DeriveSharedSecret()` (line 26) +- `generateEd25519KeyPair()` (line 89) +- `ed25519Sign()` / `ed25519Verify()` (lines 99, 113) +- `hkdfDerive()` / `hkdfDeriveKey()` (lines 166, 194) +- `aesGcmEncrypt()` / `aesGcmDecrypt()` (lines 224, 261) +- `hmacSha256()` / `hmacSha256Verify()` (lines 310, 329) +- `constantTimeEqual()` (line 388): Constant-time comparison to prevent timing side channels + +The client also has `crypto-service.ts`, `double-ratchet.ts`, `x3dh.ts`, `key-store.ts`, and `message-store.ts` which orchestrate these primitives into the full protocol flows. + +**Server-side vs client-side encryption paths:** + +The `MessageService` (`message_service.py`) has two paths: +- `store_encrypted_message()` (lines 269-314): Client-side passthrough. The server receives ciphertext, nonce, and header as strings and stores them as-is in SurrealDB. No decryption or re-encryption on the server. +- `send_encrypted_message()` (lines 316-402): Server-side encryption. Marked `[DEPRECATED]` in the docstring (line 325). This path loads the ratchet state from PostgreSQL, encrypts on the server, and stores in SurrealDB. It exists for backwards compatibility during migration to full client-side encryption. + +The client-side path is the intended production path. In this path, the server literally cannot read messages because it never has the keys. + +--- + +### Authentication System + +**Purpose:** Passwordless authentication using WebAuthn/FIDO2 passkeys. + +**Key Files:** `core/passkey/passkey_manager.py`, `api/auth.py`, `services/auth_service.py`, `core/redis_manager.py` + +The `PasskeyManager` (`passkey_manager.py:43-210`) wraps the `py_webauthn` library. It is configured with the Relying Party (RP) settings from `config.py`: + +```python +self.rp_id = settings.RP_ID # e.g., "localhost" or "chat.example.com" +self.rp_name = settings.RP_NAME # "Encrypted P2P Chat" +self.rp_origin = settings.RP_ORIGIN # "https://chat.example.com" +``` + +**Registration flow:** + +``` +Client Server Redis + │ │ │ + │ POST /auth/register/begin│ │ + │ { username, display_name }│ │ + │ ─────────────────────────►│ │ + │ │ │ + │ PasskeyManager.generate_registration_options() + │ (passkey_manager.py:55-94) │ + │ │ │ + │ │ SET webauthn:reg_challenge:{username} + │ │ challenge_bytes, TTL=600s│ + │ │ ──────────────────────────► + │ │ │ + │ ◄─── registration options (publicKey config) ────────│ + │ │ │ + │ Browser WebAuthn API │ │ + │ navigator.credentials │ │ + │ .create(options) │ │ + │ User touches │ │ + │ authenticator │ │ + │ │ │ + │ POST /auth/register/complete │ + │ { credential, username } │ │ + │ ─────────────────────────►│ │ + │ │ │ + │ │ GET+DEL webauthn:reg_challenge:{username} + │ │ ──────────────────────────► + │ │ ◄── challenge_bytes ──────│ + │ │ │ + │ PasskeyManager.verify_registration() │ + │ (passkey_manager.py:96-130) │ + │ │ │ + │ Create User in PostgreSQL │ + │ Store Credential in PostgreSQL │ + │ Initialize X3DH keys │ + │ (prekey_service.py:152-219) │ + │ │ │ + │ ◄─── UserResponse (id, username, etc.) ──────────────│ +``` + +**Authentication flow:** + +The authentication flow is similar but uses `generate_authentication_options` and `verify_authentication`. A critical detail is the signature counter check (`passkey_manager.py:184-193`): if the new counter is not greater than the stored counter, it raises a `ValueError` indicating a potentially cloned authenticator. This is the WebAuthn clone detection mechanism. + +**Challenge storage:** Redis stores challenges with a 600-second TTL (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600`, `config.py:84`). The `get_registration_challenge` and `get_authentication_challenge` methods use Redis pipelines to atomically GET and DELETE the challenge (`redis_manager.py:86-95`). This ensures each challenge is used exactly once. + +The challenge itself is 32 random bytes (`WEBAUTHN_CHALLENGE_BYTES = 32`, `config.py:85`), stored as hex in Redis. + +**Auth endpoints** (`auth.py:31-103`): + +| Endpoint | Method | Status | Description | +|---|---|---|---| +| `/auth/register/begin` | POST | 200 | Generate WebAuthn registration options | +| `/auth/register/complete` | POST | 201 | Verify credential, create user + keys | +| `/auth/authenticate/begin` | POST | 200 | Generate WebAuthn authentication options | +| `/auth/authenticate/complete` | POST | 200 | Verify credential, update counter | +| `/auth/users/search` | POST | 200 | Search users by username/display name | + +--- + +### Database Layer + +**Purpose:** Persistent storage split across three purpose-built databases. + +**PostgreSQL via SQLModel/SQLAlchemy async:** Handles all relational data with ACID guarantees. + +The engine is configured in `models/Base.py:38-44`: + +```python +engine = create_async_engine( + str(settings.DATABASE_URL), # postgresql+asyncpg://... + echo = settings.DEBUG, # SQL logging in development + pool_size = settings.DB_POOL_SIZE, # 20 (config.py:118) + max_overflow = settings.DB_MAX_OVERFLOW, # 40 (config.py:119) + pool_pre_ping = True, # Detect stale connections +) +``` + +The session factory (`Base.py:47-51`) creates `AsyncSession` instances with `expire_on_commit=False` so that objects remain usable after commit without requiring a refresh. + +**SurrealDB via AsyncSurreal:** Handles real-time messaging data. The `SurrealDBManager` (`surreal_manager.py:28-428`) connects over WebSocket to SurrealDB and provides methods for message CRUD, room management, presence tracking, and live query subscriptions. The live query feature is the primary reason SurrealDB was chosen: it pushes new records to subscribers in real-time, eliminating the need for polling. + +**Redis via redis.asyncio:** Handles ephemeral data. The `RedisManager` (`redis_manager.py:19-174`) uses a connection pool of 50 connections (`redis_manager.py:39`) and stores challenges as hex-encoded bytes with TTL-based expiry. + +--- + +## Data Flow + +### Primary Flow: Sending an Encrypted Message + +This is the most important flow in the system. Here is what happens step by step when Alice sends a message to Bob, assuming they already have an established Double Ratchet session: + +``` +Alice's Browser Server Bob's Browser + │ │ │ + ┌────┴────────────────┐ │ │ + │ 1. User types msg │ │ │ + │ in ChatInput.tsx │ │ │ + └────┬────────────────┘ │ │ + │ │ │ + ┌────┴────────────────┐ │ │ + │ 2. crypto-service.ts│ │ │ + │ encrypts: │ │ │ + │ a. Advance send │ │ │ + │ chain via │ │ │ + │ HMAC-SHA256 │ │ │ + │ b. Derive unique │ │ │ + │ message key │ │ │ + │ c. AES-256-GCM │ │ │ + │ encrypt with │ │ │ + │ random nonce │ │ │ + │ d. Build header: │ │ │ + │ {dh_pub_key, │ │ │ + │ msg_number, │ │ │ + │ prev_chain} │ │ │ + └────┬────────────────┘ │ │ + │ │ │ + │ 3. WebSocket send: │ │ + │ { │ │ + │ type: "encrypted_message", │ + │ recipient_id: bob_uuid, │ │ + │ room_id: room_uuid, │ │ + │ ciphertext: "base64...",│ │ + │ nonce: "base64...", │ │ + │ header: "{...json...}", │ │ + │ temp_id: "client_123" │ │ + │ } │ │ + │ ───────────────────────────► │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 4. websocket.py:46-54 │ │ + │ │ receives JSON, parses │ │ + │ │ routes via │ │ + │ │ websocket_service │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 5. websocket_service.py: │ │ + │ │ handle_encrypted_message│ │ + │ │ (lines 87-179) │ │ + │ │ Extracts fields, │ │ + │ │ opens DB session │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 6. message_service.py: │ │ + │ │ store_encrypted_message │ │ + │ │ (lines 269-314) │ │ + │ │ Looks up sender user │ │ + │ │ for username. │ │ + │ │ Stores ciphertext, │ │ + │ │ nonce, header AS-IS in │ │ + │ │ SurrealDB. │ │ + │ │ NO DECRYPTION. │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 7. SurrealDB live query │ │ + │ │ fires for Bob because │ │ + │ │ recipient_id matches │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 8. websocket_manager.py: │ │ + │ │ _handle_live_message │ │ + │ │ (lines 225-251) │ │ + │ │ Wraps as EncryptedMsgWS │ │ + │ └────────────┬───────────────┘ │ + │ │ │ + │ ┌────────────┴───────────────┐ │ + │ │ 9. send_message(bob_uuid) │ │ + │ │ (lines 135-153) │ │ + │ │ Sends to ALL of Bob's │ │ + │ │ active WebSockets │ │ + │ └────────────┬───────────────┘ │ + │ │ ─────────────────────────────► + │ │ │ + │ │ ┌────────────────┴──┐ + │ │ │ 10. Bob's crypto- │ + │ │ │ service.ts │ + │ │ │ decrypts: │ + │ │ │ a. Check for │ + │ │ │ skipped │ + │ │ │ msg keys │ + │ │ │ b. If new DH │ + │ │ │ pub key, │ + │ │ │ DH ratchet │ + │ │ │ step │ + │ │ │ c. Advance │ + │ │ │ recv chain │ + │ │ │ d. Derive msg │ + │ │ │ key │ + │ │ │ e. AES-256-GCM│ + │ │ │ decrypt │ + │ │ └────────┬─────────┘ + │ │ │ + │ │ ┌────────┴─────────┐ + │ │ │ 11. Plaintext │ + │ │ │ rendered in │ + │ │ │ MessageList │ + │ │ └──────────────────┘ + │ │ │ + │ ◄── confirmation ─────────│ │ + │ { type: "message_sent", │ │ + │ temp_id: "client_123", │ │ + │ status: "sent" } │ │ +``` + +The server never sees the plaintext. It acts purely as a relay that stores and forwards ciphertext blobs. The `store_encrypted_message` method (`message_service.py:269-314`) literally just wraps the received fields into a dict and calls `surreal_db.create_message()`. + +The confirmation message sent back to Alice (`websocket_service.py:148-159`) includes the `temp_id` so the client can match it to the optimistically rendered message in the UI. + +--- + +### Secondary Flow: New User Registration + Key Setup + +This flow happens once per user. + +``` +Step 1: POST /auth/register/begin + auth_service.py:331-374 + → PasskeyManager generates challenge + → Challenge stored in Redis with 600s TTL + → Registration options returned to client + +Step 2: Browser WebAuthn API (navigator.credentials.create) + → User interacts with authenticator (Touch ID, YubiKey, etc.) + → Browser creates credential bound to RP origin + +Step 3: POST /auth/register/complete + auth_service.py:376-437 + → Challenge retrieved from Redis (GET+DEL atomic) + → PasskeyManager verifies credential against challenge + → User record created in PostgreSQL + → Credential record created in PostgreSQL + → prekey_service.initialize_user_keys() called + (prekey_service.py:152-219) + +Step 4: Server-side key initialization (prekey_service.py:152-219) + → Generate X25519 identity keypair (IK) + → Generate Ed25519 signing keypair + → Store both public+private in PostgreSQL identity_keys table + → Generate signed prekey (SPK) signed with Ed25519 IK + → Generate 100 one-time prekeys (OPKs) + → Store all in PostgreSQL + +Step 5: (Client-side, post-registration) + Client generates its own X3DH keys using WebCrypto: + → X25519 identity keypair + → Ed25519 signing keypair + → Signed prekey with signature + → 100 one-time prekeys + +Step 6: POST /encryption/upload-keys/{user_id} + encryption.py:72-95 + → prekey_service.store_client_keys() + (prekey_service.py:45-150) + → Only PUBLIC keys stored on server + → Private keys remain in browser IndexedDB +``` + +Note there is a dual path here. Step 4 is the server-side key generation (used as a fallback and for backwards compatibility). Step 5-6 is the client-side key generation path (the preferred production path). In the client-side path, the server never sees private keys. + +--- + +### Secondary Flow: Establishing a New Chat Session (X3DH) + +When Alice wants to message Bob for the first time: + +``` +Step 1: Alice's client requests Bob's prekey bundle + GET /encryption/prekey-bundle/{bob_id} + encryption.py:32-51 + +Step 2: prekey_service.get_prekey_bundle(session, bob_id) + prekey_service.py:293-361 + → Fetch Bob's identity key (IK) + → Fetch Bob's active signed prekey (SPK) + → If no active SPK, auto-rotate (prekey_service.py:321) + → Fetch one unused one-time prekey (OPK) + → Mark OPK as used (single-use guarantee, line 332) + → If unused OPK count < 20, auto-replenish 100 more + (encryption.py:47-49) + → Return PreKeyBundle{IK, IK_ed25519, SPK, SPK_sig, OPK} + +Step 3: Alice performs X3DH sender-side locally + Using the prekey bundle and her own identity key: + a. Verify SPK signature with Bob's Ed25519 IK + b. Generate ephemeral keypair (EK) + c. Compute DH1..DH4 + d. Derive shared_key via HKDF + +Step 4: Initialize Double Ratchet with shared_key + Alice's client calls double_ratchet.initialize_sender() + with the shared key and Bob's SPK as the initial peer key + +Step 5: First message includes X3DH header + The header contains Alice's ephemeral public key and + identity key reference so Bob can derive the same + shared secret when he comes online + +Step 6: Bob receives the message + Bob's client uses the X3DH header + his own private keys + to perform the receiver-side X3DH exchange + (x3dh_manager.py:283-350) + Both parties now share the same root key for the + Double Ratchet +``` + +The OPK single-use guarantee is enforced at the database level. When `get_prekey_bundle` fetches an OPK, it immediately marks `is_used = True` and commits (`prekey_service.py:332-345`). If someone else requests the same OPK concurrently, they will get a different one or none. + +--- + +## Design Patterns + +### Application Factory Pattern + +**Where:** `factory.py:63-115` + +**Why:** The `create_app()` function builds and returns a fully configured FastAPI instance. This separates app creation from app execution (`main.py` just calls `create_app()`). The practical benefit is testability: you can call `create_app()` with different configurations in tests without starting a server. The lifespan manager (`factory.py:39-61`) ensures all databases are connected before the first request and properly disconnected on shutdown. + +### Service Layer Pattern + +**Where:** `services/` directory (auth_service.py, prekey_service.py, message_service.py, presence_service.py, websocket_service.py) + +**Why:** Business logic lives in service classes, not in API endpoint functions. The API layer (`api/`) is thin: it handles request validation and response formatting, then delegates to services. Services are stateless singletons instantiated at module level: + +```python +message_service = MessageService() # message_service.py:469 +prekey_service = PrekeyService() # prekey_service.py:468 +auth_service = AuthService() # auth_service.py:601 +websocket_service = WebSocketService() # websocket_service.py:324 +``` + +This means any endpoint can import and call any service without worrying about instantiation or dependency injection. The tradeoff is that services cannot be easily swapped at runtime, but for this application that is not needed. + +### Connection Pool Pattern + +**Where:** `websocket_manager.py:31-42` + +**Why:** The `ConnectionManager` maps user IDs to lists of WebSocket connections. This supports multiple simultaneous devices per user (up to 5, enforced at `websocket_manager.py:52`). When a message needs to be delivered, `send_message()` iterates over all connections for that user. Dead connections are detected during send attempts and cleaned up immediately. + +### Observer Pattern (Live Queries) + +**Where:** `websocket_manager.py:203-224`, `surreal_manager.py:341-359` + +**Why:** SurrealDB live queries implement a push-based notification system. When a new message is created in SurrealDB with `recipient_id = bob`, SurrealDB pushes that record to the server through the live query callback. The server then forwards it to Bob's WebSocket connections. This eliminates polling entirely. The alternative would be the server polling SurrealDB for new messages, which would add latency proportional to the polling interval and waste resources when no messages are pending. + +The subscription is per-user, not per-room. `live_messages_for_user()` (`surreal_manager.py:341-359`) watches `WHERE recipient_id = '{user_id}'`, so the server receives all messages destined for that user regardless of room. + +### Singleton Pattern + +**Where:** Module-level instances at the bottom of each manager file: + +```python +x3dh_manager = X3DHManager() # x3dh_manager.py:353 +double_ratchet = DoubleRatchet() # double_ratchet.py:419 +passkey_manager = PasskeyManager() # passkey_manager.py:210 +connection_manager = ConnectionManager() # websocket_manager.py:296 +surreal_db = SurrealDBManager() # surreal_manager.py:428 +redis_manager = RedisManager() # redis_manager.py:174 +``` + +**Why:** These managers hold no per-request state. `X3DHManager` and `DoubleRatchet` are pure functions wrapped in a class (they take state as arguments and return results). `PasskeyManager` holds only the RP configuration. The connection managers (`surreal_db`, `redis_manager`, `connection_manager`) hold shared connection pools. A single instance per process is the correct model. + +--- + +## Layer Separation + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ API Layer (api/) │ +│ ┌───────────┐ ┌──────────────┐ ┌─────────┐ ┌──────────────┐ │ +│ │ auth.py │ │ encryption.py│ │ rooms.py│ │ websocket.py │ │ +│ │ lines │ │ lines 1-127 │ │ │ │ lines 1-85 │ │ +│ │ 1-104 │ │ │ │ │ │ │ │ +│ └───────────┘ └──────────────┘ └─────────┘ └──────────────┘ │ +│ HTTP/WS endpoints. Request validation. Response formatting. │ +│ Thin wrappers that delegate to services. │ +├─────────────────────────────────────────────────────────────────┤ +│ Service Layer (services/) │ +│ ┌──────────────┐ ┌───────────────┐ ┌─────────────────────┐ │ +│ │ auth_service │ │ prekey_service│ │ message_service │ │ +│ │ lines 1-601 │ │ lines 1-468 │ │ lines 1-469 │ │ +│ └──────────────┘ └───────────────┘ └─────────────────────┘ │ +│ ┌──────────────────┐ ┌──────────────────┐ │ +│ │ presence_service │ │ websocket_service│ │ +│ └──────────────────┘ └──────────────────┘ │ +│ Business logic. Orchestration. Error handling. │ +│ Stateless singletons. Import Core + Models. │ +├─────────────────────────────────────────────────────────────────┤ +│ Core Layer (core/) │ +│ ┌──────────────┐ ┌───────────────┐ ┌─────────────────────┐ │ +│ │ x3dh_manager │ │ double_ratchet│ │ passkey_manager │ │ +│ │ lines 1-353 │ │ lines 1-419 │ │ lines 1-210 │ │ +│ └──────────────┘ └───────────────┘ └─────────────────────┘ │ +│ ┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ +│ │ websocket_manager│ │ surreal_mgr │ │ redis_manager │ │ +│ │ lines 1-296 │ │ lines 1-428 │ │ lines 1-174 │ │ +│ └──────────────────┘ └──────────────┘ └──────────────────┘ │ +│ ┌──────────────────┐ ┌──────────────────────────────────┐ │ +│ │ exceptions.py │ │ exception_handlers.py │ │ +│ └──────────────────┘ └──────────────────────────────────┘ │ +│ Protocol implementations. Database clients. WebSocket pool. │ +│ No imports from API or Services. │ +├─────────────────────────────────────────────────────────────────┤ +│ Model Layer (models/) │ +│ ┌──────┐ ┌────────────┐ ┌─────────────┐ ┌──────────────┐ │ +│ │ User │ │ Credential │ │ IdentityKey │ │ SignedPrekey │ │ +│ └──────┘ └────────────┘ └─────────────┘ └──────────────┘ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ +│ │ OneTimePrekey│ │ RatchetState │ │ SkippedMessageKey │ │ +│ └──────────────┘ └──────────────┘ └──────────────────────┘ │ +│ SQLModel ORM classes. Data structures. Validation. │ +│ Import only config constants and Base. │ +├─────────────────────────────────────────────────────────────────┤ +│ Schema Layer (schemas/) │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ +│ │ auth.py │ │ common.py│ │ rooms.py │ │ websocket.py │ │ +│ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │ +│ ┌──────────────┐ │ +│ │ surreal.py │ │ +│ └──────────────┘ │ +│ Pydantic request/response models. API contracts. │ +│ Serialization. No business logic. │ +└─────────────────────────────────────────────────────────────────┘ +``` + +**Import rules:** API imports Services. Services import Core + Models. Core imports nothing from API or Services. Models import nothing except config constants and the Base class. Schemas import nothing except config constants. + +The only exception is `websocket_service.py`, which imports `connection_manager` from Core and `message_service` from Services. This is acceptable because WebSocket message handling straddles both layers. + +--- + +## Data Models + +### PostgreSQL Schema (Auth + Keys) + +All PostgreSQL models inherit from `BaseDBModel` (`models/Base.py:20-34`), which provides `created_at` and `updated_at` timestamp fields with timezone awareness. + +**users** (`models/User.py:24-68`) + +``` +┌────────────────────────────────────────────────────────────┐ +│ users │ +├──────────────────────┬──────────────────┬──────────────────┤ +│ Column │ Type │ Constraints │ +├──────────────────────┼──────────────────┼──────────────────┤ +│ id │ UUID │ PK, default uuid4│ +│ username │ VARCHAR(50) │ UNIQUE, INDEX │ +│ display_name │ VARCHAR(100) │ NOT NULL │ +│ is_active │ BOOLEAN │ default True │ +│ is_verified │ BOOLEAN │ default False │ +│ identity_key │ VARCHAR(500) │ nullable │ +│ signed_prekey │ VARCHAR(500) │ nullable │ +│ signed_prekey_sig │ VARCHAR(500) │ nullable │ +│ one_time_prekeys │ TEXT │ nullable │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +├──────────────────────┴──────────────────┴──────────────────┤ +│ Relationships: credentials (1:many → Credential) │ +└────────────────────────────────────────────────────────────┘ +``` + +**credentials** (`models/Credential.py:27-78`) + +``` +┌────────────────────────────────────────────────────────────┐ +│ credentials │ +├──────────────────────┬──────────────────┬──────────────────┤ +│ Column │ Type │ Constraints │ +├──────────────────────┼──────────────────┼──────────────────┤ +│ id │ INTEGER │ PK, autoincrement│ +│ credential_id │ VARCHAR(512) │ UNIQUE, INDEX │ +│ public_key │ VARCHAR(1024) │ NOT NULL │ +│ sign_count │ INTEGER │ default 0 │ +│ aaguid │ VARCHAR(64) │ nullable │ +│ backup_eligible │ BOOLEAN │ default False │ +│ backup_state │ BOOLEAN │ default False │ +│ attestation_type │ VARCHAR(50) │ nullable │ +│ transports │ VARCHAR(200) │ nullable │ +│ user_id │ UUID │ FK → users.id │ +│ device_name │ VARCHAR(100) │ nullable │ +│ last_used_at │ TIMESTAMP(tz) │ nullable │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +├──────────────────────┴──────────────────┴──────────────────┤ +│ Relationships: user (many:1 → User) │ +└────────────────────────────────────────────────────────────┘ +``` + +**identity_keys** (`models/IdentityKey.py:18-48`) + +``` +┌────────────────────────────────────────────────────────────┐ +│ identity_keys │ +├──────────────────────┬──────────────────┬──────────────────┤ +│ Column │ Type │ Constraints │ +├──────────────────────┼──────────────────┼──────────────────┤ +│ id │ INTEGER │ PK, autoincrement│ +│ user_id │ UUID │ FK → users.id │ +│ │ │ UNIQUE, INDEX │ +│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ +│ private_key │ VARCHAR(64) │ NOT NULL (X25519)│ +│ public_key_ed25519 │ VARCHAR(64) │ NOT NULL │ +│ private_key_ed25519 │ VARCHAR(64) │ NOT NULL │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +└────────────────────────────────────────────────────────────┘ +``` + +Note: When client-side key generation is used (`store_client_keys`, `prekey_service.py:45-150`), the `private_key` and `private_key_ed25519` fields are stored as empty strings. Only the public keys are actually stored on the server. The private key fields remain in the schema for backward compatibility with the server-side key generation path. + +**signed_prekeys** (`models/SignedPrekey.py:20-50`) + +``` +┌────────────────────────────────────────────────────────────┐ +│ signed_prekeys │ +├──────────────────────┬──────────────────┬──────────────────┤ +│ Column │ Type │ Constraints │ +├──────────────────────┼──────────────────┼──────────────────┤ +│ id │ INTEGER │ PK, autoincrement│ +│ user_id │ UUID │ FK → users.id │ +│ key_id │ INTEGER │ INDEX │ +│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ +│ private_key │ VARCHAR(64) │ NOT NULL │ +│ signature │ VARCHAR(128) │ NOT NULL (Ed2551)│ +│ is_active │ BOOLEAN │ default True │ +│ expires_at │ TIMESTAMP(tz) │ nullable │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +└────────────────────────────────────────────────────────────┘ +``` + +Rotation: New signed prekey every 48 hours (`SIGNED_PREKEY_ROTATION_HOURS = 48`, `config.py:76`). Old inactive prekeys retained for 7 days (`SIGNED_PREKEY_RETENTION_DAYS = 7`, `config.py:77`) then cleaned up by `cleanup_old_signed_prekeys()` (`prekey_service.py:428-465`). + +**one_time_prekeys** (`models/OneTimePrekey.py:18-45`) + +``` +┌────────────────────────────────────────────────────────────┐ +│ one_time_prekeys │ +├──────────────────────┬──────────────────┬──────────────────┤ +│ Column │ Type │ Constraints │ +├──────────────────────┼──────────────────┼──────────────────┤ +│ id │ INTEGER │ PK, autoincrement│ +│ user_id │ UUID │ FK → users.id │ +│ key_id │ INTEGER │ INDEX │ +│ public_key │ VARCHAR(64) │ NOT NULL (X25519)│ +│ private_key │ VARCHAR(64) │ NOT NULL │ +│ is_used │ BOOLEAN │ default False │ +│ │ │ INDEX │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +└────────────────────────────────────────────────────────────┘ +``` + +Each user starts with 100 OPKs (`DEFAULT_ONE_TIME_PREKEY_COUNT = 100`, `config.py:75`). When the count drops below 20, the `get_prekey_bundle` endpoint auto-replenishes (`encryption.py:47-49`). + +**ratchet_states** (`models/RatchetState.py:18-68`) + +``` +┌────────────────────────────────────────────────────────────┐ +│ ratchet_states │ +├────────────────────────────┬──────────────┬────────────────┤ +│ Column │ Type │ Constraints │ +├────────────────────────────┼──────────────┼────────────────┤ +│ id │ INTEGER │ PK │ +│ user_id │ UUID │ FK → users.id │ +│ peer_user_id │ UUID │ FK → users.id │ +│ dh_private_key │ VARCHAR(100K)│ nullable │ +│ dh_public_key │ VARCHAR(100K)│ nullable │ +│ dh_peer_public_key │ VARCHAR(100K)│ nullable │ +│ root_key │ VARCHAR(100K)│ NOT NULL │ +│ sending_chain_key │ VARCHAR(100K)│ NOT NULL │ +│ receiving_chain_key │ VARCHAR(100K)│ NOT NULL │ +│ sending_message_number │ INTEGER │ default 0 │ +│ receiving_message_number │ INTEGER │ default 0 │ +│ previous_sending_chain_len │ INTEGER │ default 0 │ +│ created_at │ TIMESTAMP(tz)│ NOT NULL │ +│ updated_at │ TIMESTAMP(tz)│ NOT NULL │ +└────────────────────────────────────────────────────────────┘ +``` + +There is one ratchet state per (user_id, peer_user_id) pair. The relationship is directional: Alice's ratchet state for talking to Bob is a separate row from Bob's ratchet state for talking to Alice. + +**skipped_message_keys** (`models/SkippedMessageKey.py:17-51`) + +``` +┌────────────────────────────────────────────────────────────┐ +│ skipped_message_keys │ +├──────────────────────┬──────────────────┬──────────────────┤ +│ Column │ Type │ Constraints │ +├──────────────────────┼──────────────────┼──────────────────┤ +│ id │ INTEGER │ PK, autoincrement│ +│ ratchet_state_id │ INTEGER │ FK → ratchet_ │ +│ │ │ states.id, INDEX │ +│ dh_public_key │ VARCHAR(100000) │ NOT NULL, INDEX │ +│ message_number │ INTEGER │ NOT NULL, INDEX │ +│ message_key │ VARCHAR(100000) │ NOT NULL │ +│ created_at │ TIMESTAMP(tz) │ NOT NULL │ +│ updated_at │ TIMESTAMP(tz) │ NOT NULL │ +└────────────────────────────────────────────────────────────┘ +``` + +These store message keys for out-of-order delivery. When the receiving ratchet advances past a message number that has not been received yet, the key for that message is computed and stored here. When the skipped message eventually arrives, its key is looked up and consumed. + +### SurrealDB Schema (Messages + Presence) + +SurrealDB is schemaless, but these are the document structures the application creates: + +**messages** (created via `surreal_manager.py:112-122`) + +``` +{ + id: "messages:ulid_here", // SurrealDB auto-generated + sender_id: "uuid_string", + recipient_id: "uuid_string", + room_id: "rooms:ulid_here" | null, + ciphertext: "base64url_encoded_bytes", + nonce: "base64url_encoded_bytes", + header: "{\"dh_public_key\":\"...\",\"message_number\":0,...}", + sender_username: "alice", + created_at: "2026-01-15T10:30:00Z", + updated_at: "2026-01-15T10:30:00Z" +} +``` + +**presence** (created via `surreal_manager.py:287-305`) + +``` +{ + id: "presence:`user_uuid`", // user-specific record ID + user_id: "uuid_string", + status: "online" | "away" | "offline", + last_seen: "2026-01-15T10:30:00Z", + updated_at: "time::now()" +} +``` + +**rooms** (created via `surreal_manager.py:155-182`) + +``` +{ + id: "rooms:ulid_here", + name: "Room Name", + type: "direct" | "group", + members: ["uuid1", "uuid2"], + created_at: "2026-01-15T10:30:00Z", + updated_at: "2026-01-15T10:30:00Z" +} +``` + +**room_participants** (created via `surreal_manager.py:184-221`) + +``` +{ + id: "room_participants:ulid_here", + room_id: "rooms:ulid_here", + user_id: "uuid_string", + role: "member" | "admin", + joined_at: "2026-01-15T10:30:00Z" +} +``` + +### Redis Keys + +``` +webauthn:reg_challenge:{username} → 32 bytes (hex-encoded), TTL 600s +webauthn:auth_challenge:{username} → 32 bytes (hex-encoded), TTL 600s +``` + +These are the only two key patterns currently in use. Both use the Redis pipeline GET+DELETE pattern (`redis_manager.py:86-95`) for atomic one-time consumption. The TTL ensures stale challenges are automatically cleaned up even if the client never completes the flow. + +--- + +## Security Architecture + +### Threat Model + +**What the system protects against:** + +| Threat | Protection | How | +|---|---|---| +| Compromised server | E2E encryption | Server stores ciphertext, never has keys | +| Network eavesdropper | TLS + E2E | Even without TLS, messages are AES-256-GCM encrypted | +| Stolen database dump | Key separation | PostgreSQL has public keys only (client-side path). Private keys live in browser IndexedDB. OPKs are single-use. | +| Phishing / credential theft | WebAuthn | Credentials are origin-bound. Cannot be replayed on a different domain. | +| Replay attacks | Nonces + counters | AES-GCM nonces are random. Message numbers are sequential. WebAuthn challenges have TTL. | +| Authenticator cloning | Signature counter | `passkey_manager.py:184-193` checks that the counter strictly increases. If it does not, authentication fails with a clone detection error. | +| Message tampering | AEAD | AES-256-GCM provides authenticated encryption. Tampered ciphertext fails the GCM tag check (`double_ratchet.py:143-153`). | +| Key compromise (single key) | Forward secrecy | Double Ratchet generates new DH keys regularly. Compromising one chain key reveals only future messages in that chain, not past messages. | + +**What is out of scope:** + +| Threat | Why | +|---|---| +| Compromised client device | If the attacker has access to the browser, they have access to IndexedDB (private keys) and can read plaintext. There is no defense against a fully compromised endpoint. | +| Side-channel attacks on crypto | The `constantTimeEqual()` function in `primitives.ts:388-397` is the extent of timing attack mitigation. Comprehensive side-channel resistance would require constant-time implementations of all crypto primitives, which the WebCrypto API generally provides but does not guarantee. | +| Metadata analysis | The server knows who messages whom, when, how often, and message sizes. Only content is protected, not metadata. Protecting metadata would require something like mixnets or onion routing, which is not implemented. | +| Quantum computing | X25519 and Ed25519 are not post-quantum. A sufficiently powerful quantum computer could break them. Post-quantum key exchange (e.g., ML-KEM/Kyber) is not implemented. | +| Compromised authenticator supply chain | If the authenticator hardware itself is backdoored, WebAuthn cannot detect this. The AAGUID field can identify the authenticator model but not verify its integrity. | + +### Defense in Depth + +``` +Layer 1: Transport Security (TLS/HTTPS via Nginx) + │ + │ Protects: Data in transit between client and server + │ Mechanism: TLS certificate, HTTPS enforcement + │ Configuration: Nginx reverse proxy with SSL termination + │ + ▼ +Layer 2: Authentication (WebAuthn/Passkeys) + │ + │ Protects: Identity verification, prevents impersonation + │ Mechanism: Public-key cryptography, hardware-bound keys + │ Key files: passkey_manager.py:43-210, auth_service.py:331-598 + │ Redis: Challenge storage with 600s TTL, one-time consumption + │ + ▼ +Layer 3: Key Exchange (X3DH Protocol) + │ + │ Protects: Initial shared secret establishment + │ Mechanism: 3-4 Diffie-Hellman operations + HKDF + │ Key files: x3dh_manager.py:208-350, prekey_service.py:293-361 + │ Properties: Asynchronous (works even if recipient offline) + │ Deniable (either party could have forged the exchange) + │ + ▼ +Layer 4: Message Encryption (Double Ratchet + AES-256-GCM) + │ + │ Protects: Message confidentiality and integrity + │ Mechanism: Symmetric ratchet (HMAC chains) + DH ratchet + │ Key files: double_ratchet.py:64-419, primitives.ts:1-397 + │ Properties: Forward secrecy per-message + │ Future secrecy (self-healing after compromise) + │ + ▼ +Layer 5: Key Lifecycle Management + │ + │ Protects: Limits blast radius of any single key compromise + │ Mechanism: SPK rotation every 48h, OPK single-use, old SPK cleanup + │ Key files: prekey_service.py:221-291 (rotation), + │ prekey_service.py:428-465 (cleanup), + │ prekey_service.py:363-407 (replenishment) + │ Constants: SIGNED_PREKEY_ROTATION_HOURS=48 (config.py:76) + │ SIGNED_PREKEY_RETENTION_DAYS=7 (config.py:77) + │ DEFAULT_ONE_TIME_PREKEY_COUNT=100 (config.py:75) + │ + ▼ +Layer 6: Rate Limiting and Abuse Prevention + │ + Protects: Against brute force and DoS + Mechanism: Per-user message rate limits, auth attempt limits + Constants: RATE_LIMIT_MESSAGES_PER_MINUTE=60 (config.py:146) + RATE_LIMIT_AUTH_ATTEMPTS=5 (config.py:147) + WS_MAX_CONNECTIONS_PER_USER=5 (config.py:141) +``` + +--- + +## Configuration + +All configuration lives in `config.py`. The `Settings` class (lines 96-218) inherits from `pydantic_settings.BaseSettings` and loads values from environment variables and the `.env` file. + +### Application Settings + +| Variable | Default | Description | +|---|---|---| +| `ENV` | `"development"` | `development`, `production`, or `testing` | +| `DEBUG` | `True` | Enables SQL echo logging, docs endpoints | +| `APP_NAME` | `"encrypted-p2p-chat"` | Application name in metadata | +| `SECRET_KEY` | (required) | Application secret, no default | + +### PostgreSQL Settings + +| Variable | Default | Description | +|---|---|---| +| `POSTGRES_HOST` | `"localhost"` | Database host | +| `POSTGRES_PORT` | `5432` | Database port | +| `POSTGRES_DB` | `"chat_auth"` | Database name | +| `POSTGRES_USER` | `"chat_user"` | Database user | +| `POSTGRES_PASSWORD` | `""` | Database password | +| `DATABASE_URL` | (auto-built) | Full connection string, assembled by `field_validator` at line 149 | +| `DB_POOL_SIZE` | `20` | SQLAlchemy connection pool size | +| `DB_MAX_OVERFLOW` | `40` | Pool overflow connections | + +The `assemble_db_connection` validator (`config.py:149-161`) builds the URL from components if `DATABASE_URL` is not explicitly set. The URL uses the `postgresql+asyncpg://` scheme for async connections. + +### SurrealDB Settings + +| Variable | Default | Description | +|---|---|---| +| `SURREAL_HOST` | `"localhost"` | SurrealDB host | +| `SURREAL_PORT` | `8000` | SurrealDB port | +| `SURREAL_USER` | `"root"` | SurrealDB user | +| `SURREAL_PASSWORD` | (required) | SurrealDB password | +| `SURREAL_NAMESPACE` | `"chat"` | SurrealDB namespace | +| `SURREAL_DATABASE` | `"production"` | SurrealDB database | +| `SURREAL_URL` | (auto-built) | WebSocket URL, assembled at line 163 | + +### Redis Settings + +| Variable | Default | Description | +|---|---|---| +| `REDIS_HOST` | `"localhost"` | Redis host | +| `REDIS_PORT` | `6379` | Redis port | +| `REDIS_PASSWORD` | `""` | Redis password (optional) | +| `REDIS_URL` | (auto-built) | Connection URL, assembled at line 178 | + +### WebAuthn Settings + +| Variable | Default | Description | +|---|---|---| +| `RP_ID` | `"localhost"` | Relying Party ID (domain) | +| `RP_NAME` | `"Encrypted P2P Chat"` | Relying Party display name | +| `RP_ORIGIN` | `"http://localhost:3000"` | Expected origin for credential verification | + +The `RP_ID` must match the domain the browser sees. In production, this would be `"chat.example.com"`. If it does not match, WebAuthn verification fails. + +### WebSocket and Security Settings + +| Variable | Default | Description | +|---|---|---| +| `WS_HEARTBEAT_INTERVAL` | `30` | Seconds between heartbeat pings | +| `WS_MAX_CONNECTIONS_PER_USER` | `5` | Max simultaneous WebSocket connections | +| `KEY_ROTATION_DAYS` | `90` | General key rotation period | +| `MAX_SKIPPED_MESSAGE_KEYS` | `1000` | Max out-of-order messages per ratchet | +| `RATE_LIMIT_MESSAGES_PER_MINUTE` | `60` | Per-user message rate limit | +| `RATE_LIMIT_AUTH_ATTEMPTS` | `5` | Max auth attempts before lockout | +| `CORS_ORIGINS` | `["http://localhost:3000", "http://localhost:5173"]` | Allowed CORS origins | + +--- + +## Performance Considerations + +### Database Connection Pooling + +PostgreSQL pool: 20 base connections + 40 overflow = 60 max simultaneous connections (`config.py:118-119`). The engine uses `pool_pre_ping=True` (`models/Base.py:43`) to detect and replace stale connections before queries fail. + +SQLAlchemy async sessions are created per-request via the `get_session()` dependency (`models/Base.py:54-59`). Sessions are recycled automatically after each request. + +Redis pool: 50 connections (`redis_manager.py:39`). Connection pooling is handled by `redis.asyncio.ConnectionPool`. + +SurrealDB: Single persistent WebSocket connection per application instance (`surreal_manager.py:47-48`). The `ensure_connected()` method (`surreal_manager.py:73-78`) lazily reconnects if the connection drops. + +### Message Throughput + +60 messages per minute per user rate limit (`config.py:146`). This translates to one message per second sustained, which is reasonable for a chat application. + +WebSocket heartbeat every 30 seconds (`config.py:140`). This is frequent enough to detect dead connections quickly but not so frequent as to waste bandwidth on idle connections. + +SurrealDB live queries are push-based (`surreal_manager.py:341-359`). When a new message is created, SurrealDB pushes it to the subscribed callback immediately. There is no polling interval, so latency is limited to network round-trip plus SurrealDB processing time. + +### Encryption Overhead + +Approximate per-message encryption cost (based on typical X25519/AES-256-GCM performance): + +``` +X25519 DH exchange: ~0.1ms +HMAC-SHA256 chain step: ~0.01ms +AES-256-GCM encrypt: ~0.001ms per KB (for typical chat messages) +HKDF derivation: ~0.01ms + +Total per-message: < 1ms +``` + +This is negligible compared to network latency (typically 10-100ms). Encryption overhead is not a bottleneck for this application. + +### Connection Limits + +5 WebSocket connections per user (`config.py:141`). This supports a reasonable number of devices (phone, laptop, tablet, desktop, secondary browser) without allowing a single user to exhaust server WebSocket capacity. + +The `ConnectionManager` delivers each message to all of a user's connections (`websocket_manager.py:135-153`), so the cost of multi-device scales linearly with the number of connections per user. + +--- + +## Design Decisions + +### Why Three Databases? + +This is the most obvious question about the architecture, so it deserves a direct answer. + +**PostgreSQL** stores relational data that requires ACID transactions: users with unique usernames, credentials with foreign keys to users, identity keys with unique constraints per user, ratchet states that must be updated atomically. This data has complex relationships (user has many credentials, ratchet state references two users, etc.) and benefits from SQL's referential integrity. + +**SurrealDB** stores messages and presence data. The key feature is live queries: when a message is created, SurrealDB pushes it to subscribers in real-time without polling. This is the core of the chat experience. You could do this with PostgreSQL LISTEN/NOTIFY, but SurrealDB's live queries are more natural for this pattern and the schema is document-oriented, which fits the variable structure of encrypted messages better. + +**Redis** stores ephemeral data that should auto-expire: WebAuthn challenges (600s TTL), rate limit counters. You would not want to poll PostgreSQL to clean up expired challenges. Redis handles this natively with key TTL. + +The tradeoff is operational complexity. Running three databases means three things that can fail, three things to back up, three things to monitor. The justification is that each database is doing what it does best, and trying to make one database do all three jobs would create worse tradeoffs (e.g., polling for real-time updates, manual expiry jobs for challenges). + +### Why WebAuthn Instead of JWT + Password? + +WebAuthn passkeys are phishing-resistant by design. The credential is bound to the RP origin, so it cannot be used on a fake domain. There are no passwords to steal from a database breach. The private key never leaves the authenticator hardware. + +The tradeoff is browser and device support. WebAuthn requires a modern browser and an authenticator (Touch ID, Windows Hello, YubiKey, etc.). Account recovery is harder: if you lose your only authenticator, you lose access. This can be mitigated by registering multiple authenticators. + +### Why SolidJS Instead of React? + +SolidJS uses fine-grained reactivity. When a message arrives, only the specific DOM elements that depend on that message are updated. React would diff the entire virtual DOM tree for the message list. For a real-time chat application where messages arrive frequently, this difference matters. + +SolidJS also has a smaller bundle size than React, which helps with initial load time. The nanostores library provides a framework-agnostic reactive store that integrates naturally with SolidJS's reactivity model. + +The tradeoff is ecosystem size. React has far more libraries, tutorials, and Stack Overflow answers. SolidJS is smaller and you occasionally need to build something that React would have a library for. + +### Why Separate Server and Client Encryption Paths? + +`message_service.py` has both `send_encrypted_message` (server-side encryption, line 316, marked `[DEPRECATED]`) and `store_encrypted_message` (client-side passthrough, line 269). The server-side path exists because the system was originally built with server-side encryption and is being migrated to full client-side encryption. + +In the server-side path, the server loads the ratchet state, encrypts the plaintext, and stores the ciphertext. This means the server momentarily has access to the plaintext. In the client-side path, the server never sees the plaintext; it receives and stores ciphertext as-is. + +The client-side path is the correct production path. The server-side path remains for backwards compatibility and as a fallback during the migration. + +--- + +## Deployment Architecture + +### Development + +``` +docker compose -f dev.compose.yml up + +┌───────────────────────────────────────────────────────────────┐ +│ Docker Network (chat_network_dev) │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ PostgreSQL │ │ SurrealDB │ │ Redis │ │ +│ │ 16-alpine │ │ latest │ │ 8-alpine │ │ +│ │ :5432→5432 │ │ :8000→8001 │ │ :6379→6379 │ │ +│ │ │ │ file://data│ │ appendonly │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ FastAPI │ │ Vite Dev │ │ Nginx │ │ +│ │ :8000→8000 │ │ :5173→5173 │ │ :80→80 │ │ +│ │ uvicorn │ │ HMR enabled│ │ reverse │ │ +│ │ --reload │ │ hot module │ │ proxy │ │ +│ │ vol: ./back │ │ vol: ./fro │ │ dev.nginx │ │ +│ │ end │ │ ntend │ │ config │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +│ │ +│ Source: dev.compose.yml │ +└───────────────────────────────────────────────────────────────┘ +``` + +In development, all services expose ports to the host for direct access and debugging. The backend volume-mounts the `./backend` directory and runs with uvicorn `--reload` for live reloading. The frontend volume-mounts `./frontend` and runs Vite's dev server with HMR (Hot Module Replacement). Nginx sits in front as a reverse proxy matching the production topology. + +### Production + +``` +docker compose up + +┌───────────────────────────────────────────────────────────────┐ +│ Docker Network (chat_network) │ +│ │ +│ ┌──────────────────────────────────────────────────────────┐ │ +│ │ Nginx (:80, :443) │ │ +│ │ │ │ +│ │ /api/* ────────► upstream: backend:8000 (HTTP) │ │ +│ │ /ws ────────► upstream: backend:8000 (WebSocket) │ │ +│ │ /* ────────► Static SolidJS build (from image) │ │ +│ │ │ │ +│ └──────────────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ┌──────────────────────────┼──────────────────────────────┐ │ +│ │ │ │ │ +│ │ ┌─────────────┐ ┌─────┴─────┐ ┌─────────────┐ │ │ +│ │ │ PostgreSQL │ │ FastAPI │ │ SurrealDB │ │ │ +│ │ │ 16-alpine │ │ gunicorn │ │ latest │ │ │ +│ │ │ vol: data │ │ workers │ │ vol: data │ │ │ +│ │ │ healthcheck │ │ no ports │ │ healthcheck│ │ │ +│ │ │ restart: │ │ exposed │ │ restart: │ │ │ +│ │ │ always │ │ to host │ │ always │ │ │ +│ │ └─────────────┘ └───────────┘ └─────────────┘ │ │ +│ │ │ │ +│ │ ┌─────────────┐ │ │ +│ │ │ Redis │ │ │ +│ │ │ 8-alpine │ │ │ +│ │ │ vol: data │ │ │ +│ │ │ maxmem 2gb │ │ │ +│ │ │ LRU evict │ │ │ +│ │ └─────────────┘ │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ Source: compose.yml │ +└───────────────────────────────────────────────────────────────┘ +``` + +In production, the FastAPI backend does not expose any ports to the host. It is only accessible through the Nginx container on the internal Docker network. Databases use named volumes for persistence and `restart: always` for automatic recovery. The frontend container serves the pre-built SolidJS static files through Nginx. + +The compose file (`compose.yml`) uses `depends_on` with `condition: service_healthy` to ensure databases are ready before the backend starts. Each database has a healthcheck command (PostgreSQL: `pg_isready`, SurrealDB: `/health`, Redis: `redis-cli ping`). + +--- + +## Error Handling Strategy + +### Custom Exception Hierarchy + +Defined in `core/exceptions.py` (lines 1-95): + +``` +AppException (base) + ├── UserExistsError → 409 Conflict + ├── UserNotFoundError → 404 Not Found + ├── UserInactiveError → 403 Forbidden + ├── CredentialNotFoundError → 404 Not Found + ├── CredentialVerificationError → 401 Unauthorized + ├── ChallengeExpiredError → 400 Bad Request + ├── DatabaseError → 500 Internal Server Error + ├── AuthenticationError → 401 Unauthorized + ├── InvalidDataError → 400 Bad Request + ├── EncryptionError → 500 Internal Server Error + ├── DecryptionError → 500 Internal Server Error + ├── RatchetStateNotFoundError → 404 Not Found + └── KeyExchangeError → 500 Internal Server Error +``` + +### Exception Handlers + +Registered in `core/exception_handlers.py` (lines 221-246) via `register_exception_handlers(app)`, which is called from `factory.py:89`. Each exception type maps to a handler function that returns a `JSONResponse` with the appropriate HTTP status code. + +For security-sensitive errors (`DatabaseError`, `EncryptionError`, `DecryptionError`, `KeyExchangeError`), the response body contains a generic message ("Internal server error", "Encryption failed", etc.) rather than the actual error detail. The detail is logged server-side but not exposed to the client. + +### WebSocket Error Format + +WebSocket errors are sent as JSON to the client: + +```json +{ + "type": "error", + "error_code": "invalid_json", + "error_message": "Invalid JSON format" +} +``` + +Error codes include: `max_connections`, `database_error`, `invalid_json`, `missing_type`, `unknown_type`, `processing_error`. These are defined inline in `websocket.py` (lines 62-76) and `websocket_manager.py` (lines 58-62). + +--- + +## Extensibility + +Where to add new features: + +| What | Where | Steps | +|---|---|---| +| New API endpoint | `api/` directory | 1. Create router in `api/new_feature.py` 2. Create service in `services/new_feature_service.py` 3. Register router in `factory.py` (after line 113) | +| New database model | `models/` directory | 1. Create model class inheriting `BaseDBModel` 2. Import it somewhere that loads at startup (e.g., `models/__init__.py`) 3. `init_db()` auto-creates the table via `SQLModel.metadata.create_all` | +| New encryption algorithm | `core/encryption/` | 1. Add implementation in `core/encryption/` 2. Add config constants in `config.py` 3. Wire into `message_service.py` or `crypto-service.ts` | +| New WebSocket message type | `websocket_service.py` | 1. Add constant in `config.py` (WS_MESSAGE_TYPE_*) 2. Add handler method in `WebSocketService` 3. Add routing case in `route_message()` | +| New SurrealDB collection | `surreal_manager.py` | 1. Add CRUD methods in `SurrealDBManager` 2. Add response schema in `schemas/surreal.py` | + +--- + +## Limitations + +These are known architectural limitations, not bugs: + +1. **No group chat encryption.** The Double Ratchet is a two-party protocol. Group chat encryption would require either Sender Keys (what Signal uses for groups, where each member maintains a separate ratchet with every other member) or the MLS (Messaging Layer Security) protocol. Neither is implemented. + +2. **No post-quantum key exchange.** X25519 is vulnerable to Shor's algorithm on a sufficiently powerful quantum computer. Migrating to a hybrid scheme (X25519 + ML-KEM) would future-proof the key exchange, but this adds complexity and the quantum threat timeline is debated. + +3. **Metadata not protected.** The server knows who sends messages to whom, when, how frequently, and the approximate size of each message. Only the content is encrypted. Metadata protection would require techniques like onion routing, padding, or dummy traffic, all of which add significant complexity and performance cost. + +4. **No message deletion or expiry.** Once a message is stored in SurrealDB, it stays there indefinitely. There is no TTL on messages and no "delete for everyone" feature. SurrealDB does support the `_schedule_room_deletion` method for ephemeral rooms (`surreal_manager.py:393-425`), but this is room-level, not message-level. + +5. **Single-region deployment.** The Docker Compose setup assumes all services run on one machine or one cluster. There is no geo-distribution, no CDN for the frontend, and no database replication. For a production deployment serving users across regions, you would need to add these. + +6. **No offline message queue.** If a recipient is offline when a message is sent, the message is stored in SurrealDB but only delivered when the recipient connects and the live query fires. There is no explicit mechanism for fetching missed messages on reconnect beyond the live query catching up. The `get_room_messages` method (`surreal_manager.py:124-153`) exists for fetching message history, but the client must explicitly call it. + +--- + +## Key Files Reference + +| What | Where | Lines | +|---|---|---| +| App factory | `factory.py` | 63-115 | +| Lifespan (DB init) | `factory.py` | 39-61 | +| All settings | `config.py` | 96-218 | +| Crypto constants | `config.py` | 64-77 | +| X3DH protocol | `core/encryption/x3dh_manager.py` | 56-353 | +| X3DH sender | `core/encryption/x3dh_manager.py` | 208-281 | +| X3DH receiver | `core/encryption/x3dh_manager.py` | 283-350 | +| Double Ratchet | `core/encryption/double_ratchet.py` | 64-419 | +| Encrypt message | `core/encryption/double_ratchet.py` | 323-362 | +| Decrypt message | `core/encryption/double_ratchet.py` | 364-416 | +| WebAuthn manager | `core/passkey/passkey_manager.py` | 43-210 | +| Clone detection | `core/passkey/passkey_manager.py` | 184-193 | +| WebSocket pool | `core/websocket_manager.py` | 31-296 | +| Heartbeat loop | `core/websocket_manager.py` | 177-201 | +| Live query sub | `core/websocket_manager.py` | 203-223 | +| Live msg handler | `core/websocket_manager.py` | 225-251 | +| SurrealDB client | `core/surreal_manager.py` | 28-428 | +| Redis client | `core/redis_manager.py` | 19-174 | +| Exception types | `core/exceptions.py` | 7-95 | +| Exception handlers | `core/exception_handlers.py` | 221-246 | +| Message storage | `services/message_service.py` | 269-314 | +| Conversation init | `services/message_service.py` | 48-166 | +| Key management | `services/prekey_service.py` | 41-468 | +| Client key upload | `services/prekey_service.py` | 45-150 | +| Prekey bundle | `services/prekey_service.py` | 293-361 | +| SPK rotation | `services/prekey_service.py` | 221-291 | +| OPK replenish | `services/prekey_service.py` | 363-407 | +| Auth service | `services/auth_service.py` | 47-601 | +| WS message router | `services/websocket_service.py` | 35-324 | +| Auth endpoints | `api/auth.py` | 31-103 | +| Encryption endpoints | `api/encryption.py` | 21-127 | +| WebSocket endpoint | `api/websocket.py` | 25-84 | +| User model | `models/User.py` | 24-68 | +| Credential model | `models/Credential.py` | 27-78 | +| Identity key model | `models/IdentityKey.py` | 18-48 | +| Signed prekey model | `models/SignedPrekey.py` | 20-50 | +| One-time prekey model | `models/OneTimePrekey.py` | 18-45 | +| Ratchet state model | `models/RatchetState.py` | 18-68 | +| Skipped keys model | `models/SkippedMessageKey.py` | 17-51 | +| DB engine + sessions | `models/Base.py` | 37-67 | +| Client crypto prims | `frontend/src/crypto/primitives.ts` | 1-397 | +| Client crypto svc | `frontend/src/crypto/crypto-service.ts` | - | +| Client double ratchet | `frontend/src/crypto/double-ratchet.ts` | - | +| Client X3DH | `frontend/src/crypto/x3dh.ts` | - | +| Client key store | `frontend/src/crypto/key-store.ts` | - | +| Production compose | `compose.yml` | 1-118 | +| Development compose | `dev.compose.yml` | 1-151 | diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md new file mode 100644 index 00000000..a3760484 --- /dev/null +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/03-IMPLEMENTATION.md @@ -0,0 +1,1118 @@ +# Implementation Guide + +This document walks through every significant piece of the encrypted P2P chat, file by file, explaining not just what the code does but why each decision was made. If you want to build something like Signal from scratch, this is the map. + +--- + +## File Structure Walkthrough + +``` +backend/ + app/ + api/ + auth.py # WebAuthn registration/login endpoints + encryption.py # Key upload, prekey bundle retrieval + rooms.py # Chat room creation and listing + websocket.py # WebSocket endpoint (receives raw frames) + core/ + encryption/ + x3dh_manager.py # X3DH key exchange: generate, sign, exchange + double_ratchet.py # Double Ratchet: KDF chains, encrypt, decrypt + passkey/ + passkey_manager.py # WebAuthn: registration, auth, clone detection + websocket_manager.py # Connection pool, heartbeats, live query sub + surreal_manager.py # SurrealDB client (messages, live queries) + redis_manager.py # Redis client (challenges, presence) + exceptions.py # Custom exception hierarchy + exception_handlers.py # FastAPI exception-to-HTTP mappings + enums.py # Presence status enum + models/ + Base.py # SQLModel engine, async session maker + User.py # User table (id, username, display_name) + Credential.py # WebAuthn credential storage + IdentityKey.py # X25519 + Ed25519 identity keypairs + SignedPrekey.py # Signed prekeys with rotation/expiry + OneTimePrekey.py # Single-use one-time prekeys + RatchetState.py # Serialized Double Ratchet state per conversation + SkippedMessageKey.py # Persisted skipped message keys + schemas/ + auth.py # Pydantic schemas for registration/auth flows + websocket.py # Pydantic schemas for WS message types + surreal.py # Pydantic schemas for SurrealDB documents + rooms.py # Pydantic schemas for room operations + common.py # Shared response schemas + services/ + auth_service.py # User creation, credential management + prekey_service.py # Key initialization, rotation, bundle retrieval + message_service.py # Conversation init, encrypt, store, decrypt + websocket_service.py # WS message routing (type dispatch) + presence_service.py # Online/offline/away status via Redis + config.py # All constants + Settings (env vars) + factory.py # FastAPI app factory with middleware + main.py # Uvicorn entry point + +frontend/ + src/ + crypto/ + primitives.ts # Low-level WebCrypto: X25519, Ed25519, AES-GCM, HKDF, HMAC + x3dh.ts # Client-side X3DH: generate keys, initiate, receive + double-ratchet.ts # Client-side Double Ratchet: init, encrypt, decrypt, serialize + crypto-service.ts # High-level orchestrator: session management, key upload + key-store.ts # IndexedDB persistence for all key material + message-store.ts # Message history persistence + websocket/ + websocket-manager.ts # WS connection, reconnect, heartbeat, message queue + message-handlers.ts # Dispatch incoming WS messages to stores + stores/ + auth.store.ts # Authentication state (nanostores) + session.store.ts # Session token management + messages.store.ts # Message list per room + rooms.store.ts # Room list and active room + presence.store.ts # Online status per user + typing.store.ts # Typing indicator state + ui.store.ts # UI state (sidebar open, modals) + settings.store.ts # User preferences + services/ + auth.service.ts # WebAuthn browser API calls + room.service.ts # Room CRUD via API + components/ + Auth/ # Login, Register, PasskeyButton + Chat/ # MessageList, ChatInput, ConversationList, etc. + Layout/ # AppShell, Sidebar, Header, ProtectedRoute + UI/ # Button, Modal, Input, Tooltip, etc. + pages/ + Chat.tsx # Main chat page (conversation + message pane) + Login.tsx # Login page + Register.tsx # Registration page + Home.tsx # Landing page + NotFound.tsx # 404 + types/ + encryption.ts # Type definitions for crypto structures + chat.ts # Message and room types + websocket.ts # WS message types + auth.ts # Auth types + guards.ts # Runtime type guards for WS messages + api.ts # API response types + components.ts # Component prop types + index.ts # Re-exports + constants + lib/ + api-client.ts # HTTP client for REST endpoints + base64.ts # Base64 helpers + validators.ts # Input validation + date.ts # Date formatting + config.ts # Frontend constants (URLs, timeouts) +``` + +--- + +## Building the X3DH Key Exchange + +X3DH (Extended Triple Diffie-Hellman) solves a specific problem: Alice wants to send Bob an encrypted message, but Bob is offline. They have never communicated before. There is no way to do a live handshake. X3DH lets Alice compute a shared secret using Bob's pre-published keys, so that when Bob comes back online, he can derive the same secret and decrypt everything Alice sent while he was away. + +### Step 1: Key Generation + +The foundation of X3DH is generating two types of identity keypairs per user. Here is the X25519 identity key generation from `x3dh_manager.py:60-86`: + +```python +def generate_identity_keypair_x25519(self) -> tuple[str, str]: + private_key = X25519PrivateKey.generate() + public_key = private_key.public_key() + + private_bytes = private_key.private_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PrivateFormat.Raw, + encryption_algorithm = serialization.NoEncryption() + ) + public_bytes = public_key.public_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PublicFormat.Raw + ) + + return ( + bytes_to_base64url(private_bytes), + bytes_to_base64url(public_bytes) + ) +``` + +There is also `generate_identity_keypair_ed25519` at `x3dh_manager.py:88-114`, which follows the exact same pattern but uses `Ed25519PrivateKey.generate()` instead. + +**Why Raw encoding?** PEM and DER formats include metadata headers, algorithm identifiers, and ASN.1 structure. That is wasted bytes when you already know the algorithm on both sides. Raw encoding gives you exactly 32 bytes for X25519 and 32 bytes for Ed25519. That is the minimum footprint, and it maps directly to what the WebCrypto API expects on the frontend when importing with `importKey("raw", ...)`. + +**Why base64url?** The keys need to travel through JSON (over HTTP and WebSocket) and get stored in PostgreSQL text columns. Hex encoding doubles the size (32 bytes becomes 64 characters). Standard base64 uses `+` and `/` characters that require URL encoding. Base64url uses `-` and `_` instead, which are safe in URLs, JSON, and query parameters without escaping. The `webauthn.helpers` library provides `bytes_to_base64url` and `base64url_to_bytes`, so we piggyback on those rather than rolling our own. + +**Why two key types (X25519 and Ed25519)?** X25519 is a Diffie-Hellman function. It takes two keys and produces a shared secret. That is what you need for the actual key agreement. But X25519 cannot produce signatures. You cannot prove "I am the owner of this public key" with X25519 alone. Ed25519 is a signature algorithm that operates on the same curve family (Curve25519), so key generation is equally fast. The Ed25519 identity key signs the Signed Prekey, proving to Alice that Bob actually published that prekey and not some attacker in the middle. Separating the two key types follows the cryptographic principle of using distinct keys for distinct purposes. If you used a single key for both DH and signing, a vulnerability in one operation could compromise the other. + +### Step 2: Signed Prekey Creation + +The Signed Prekey (SPK) is a semi-static X25519 key that gets rotated every 48 hours. Here is `generate_signed_prekey` from `x3dh_manager.py:116-152`: + +```python +def generate_signed_prekey(self, + identity_private_key_ed25519: str) -> tuple[str, + str, + str]: + spk_private = X25519PrivateKey.generate() + spk_public = spk_private.public_key() + + spk_private_bytes = spk_private.private_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PrivateFormat.Raw, + encryption_algorithm = serialization.NoEncryption() + ) + spk_public_bytes = spk_public.public_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PublicFormat.Raw + ) + + identity_private_bytes = base64url_to_bytes(identity_private_key_ed25519) + identity_private = Ed25519PrivateKey.from_private_bytes( + identity_private_bytes + ) + + signature = identity_private.sign(spk_public_bytes) + + return ( + bytes_to_base64url(spk_private_bytes), + bytes_to_base64url(spk_public_bytes), + bytes_to_base64url(signature) + ) +``` + +**Why sign with Ed25519?** The signature proves that Bob's long-term Ed25519 identity key endorses this particular SPK. When Alice fetches Bob's prekey bundle, she verifies this signature before performing the DH operations. Without the signature, a man-in-the-middle could substitute their own SPK public key into Bob's bundle. Alice would compute a shared secret with the attacker instead of Bob, and the attacker could relay messages between them while reading everything. + +**What does the signature prove?** It proves exactly one thing: "The entity controlling the Ed25519 private key corresponding to Bob's published Ed25519 public key chose to sign this specific SPK public key." That binds the SPK to Bob's identity. + +**Why not self-sign with X25519?** X25519 is a DH function, not a signature scheme. There is no `sign()` method on an X25519 key. You could theoretically convert between Curve25519 and Ed25519 key formats (they share the same underlying curve), but doing that conversion is a footgun. It is error-prone, not all libraries support it, and it violates the principle of key separation. Using a dedicated Ed25519 signing key is simpler and safer. + +### Step 3: Prekey Bundle Assembly + +When Alice wants to message Bob, the server assembles Bob's prekey bundle. This happens in `prekey_service.py:293-361`: + +```python +async def get_prekey_bundle( + self, + session: AsyncSession, + user_id: UUID +) -> PreKeyBundle: + # Fetch identity key + ik_statement = select(IdentityKey).where(IdentityKey.user_id == user_id) + ... + # Fetch active signed prekey + spk_statement = select(SignedPrekey).where( + SignedPrekey.user_id == user_id, + SignedPrekey.is_active + ).order_by(SignedPrekey.created_at.desc()) + ... + if not signed_prekey: + signed_prekey = await self.rotate_signed_prekey(session, user_id) + + # Fetch one unused OPK + opk_statement = select(OneTimePrekey).where( + OneTimePrekey.user_id == user_id, + not OneTimePrekey.is_used + ).limit(1) + ... + if one_time_prekey: + one_time_prekey.is_used = True + ... +``` + +**Why auto-rotate?** If Bob's signed prekey expired and there are no active ones, the bundle fetch would fail. Auto-rotation at `prekey_service.py:316-321` ensures there is always an active SPK. The rotation generates a new X25519 keypair, signs it with Bob's Ed25519 identity key, marks old SPKs as inactive, and sets a 48-hour expiry (`SIGNED_PREKEY_ROTATION_HOURS = 48` in `config.py:76`). + +**Why mark OPK as used?** One-Time Prekeys provide forward secrecy for the initial handshake. Each OPK is consumed exactly once. When Alice fetches Bob's bundle, the server marks that OPK as `is_used = True` and commits the change. If a second person also tries to start a conversation with Bob around the same time, they get a different OPK (or none at all). This single-use guarantee means that even if Bob's long-term keys are later compromised, an attacker cannot retroactively compute the shared secret for conversations that used an OPK, because the OPK private key was consumed and the DH output from it cannot be reconstructed from the other keys alone. + +**What happens if no OPK is available?** The protocol still works. The `PreKeyBundle` dataclass has `one_time_prekey: str | None = None`. When the bundle is assembled without an OPK, the X3DH sender side simply computes `dh1 + dh2 + dh3` instead of `dh1 + dh2 + dh3 + dh4`. This gives slightly weaker forward secrecy for the initial handshake (it relies on the signed prekey not being compromised), but the Double Ratchet provides forward secrecy for all subsequent messages regardless. The system compensates by calling `replenish_one_time_prekeys` periodically to maintain a pool of 100 OPKs (`DEFAULT_ONE_TIME_PREKEY_COUNT = 100` in `config.py:75`). + +### Step 4: X3DH Sender Side + +This is the core of the initial key agreement. Here is the DH operations from `x3dh_manager.py:241-264`: + +```python +dh1 = alice_ik_private.exchange(bob_spk_public) +dh2 = alice_ek_private.exchange(bob_ik_public) +dh3 = alice_ek_private.exchange(bob_spk_public) + +used_one_time_prekey = False +if bob_bundle.one_time_prekey: + bob_opk_public_bytes = base64url_to_bytes(bob_bundle.one_time_prekey) + bob_opk_public = X25519PublicKey.from_public_bytes(bob_opk_public_bytes) + dh4 = alice_ek_private.exchange(bob_opk_public) + key_material = dh1 + dh2 + dh3 + dh4 + used_one_time_prekey = True +else: + key_material = dh1 + dh2 + dh3 + +f = b'\xff' * X25519_KEY_SIZE +hkdf = HKDF( + algorithm = hashes.SHA256(), + length = X25519_KEY_SIZE, + salt = b'\x00' * X25519_KEY_SIZE, + info = b'X3DH', +) +shared_key = hkdf.derive(f + key_material) +``` + +Before this code runs, the sender verifies the signed prekey signature at `x3dh_manager.py:217-220`. If verification fails, a `ValueError` is raised and the exchange aborts. This is the MITM protection. + +**Each DH operation explained:** + +- **dh1 = IK_A.exchange(SPK_B)** -- Alice's long-term identity key with Bob's signed prekey. This provides mutual authentication: only the real Alice and the real Bob can compute this value. +- **dh2 = EK_A.exchange(IK_B)** -- Alice's ephemeral key with Bob's identity key. This provides forward secrecy from Alice's side: the ephemeral key is used once and discarded. +- **dh3 = EK_A.exchange(SPK_B)** -- Alice's ephemeral key with Bob's signed prekey. This provides additional mixing. Even if either IK_A or IK_B is compromised, this DH output is still unknown to the attacker. +- **dh4 = EK_A.exchange(OPK_B)** (optional) -- Alice's ephemeral key with Bob's one-time prekey. This provides replay protection and additional forward secrecy, because the OPK is used exactly once. + +**Why the 0xFF padding?** The line `f = b'\xff' * X25519_KEY_SIZE` prepends 32 bytes of `0xFF` before the key material fed into HKDF. This is directly from the Signal specification. The purpose is to ensure the HKDF input is never all zeros. If all four DH outputs happened to be zero (which would indicate a catastrophic failure, like someone substituting the identity point), the `0xFF` padding ensures the HKDF input still has high entropy. It is a belt-and-suspenders defense. + +**Why salt of zeros?** The Signal specification defines the salt as 32 bytes of `0x00`. HKDF requires a salt, and using all zeros is equivalent to using no salt (HKDF treats a zero-length salt as a string of zeros anyway). The salt is fixed rather than random because both sides need to derive the same key without communicating the salt. If you used a random salt, you would need to transmit it, which adds complexity and message size for no security benefit here (the DH outputs already provide the randomness). + +**Why info="X3DH"?** The `info` parameter in HKDF is a domain separation string. It ensures that the derived key is bound to this specific protocol. If the same key material were accidentally reused in a different context (say, a TLS handshake), the info string would produce a different derived key. This prevents cross-protocol attacks. + +**Associated data: IK_A || IK_B.** After deriving the shared key, the code computes `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` at `x3dh_manager.py:272`. This associated data is later passed to AES-GCM as the AAD (Additional Authenticated Data). It binds the encrypted messages to the specific pair of identity keys that performed the handshake. If an attacker tries to redirect messages between different users, the AAD check will fail during decryption. + +### Step 5: X3DH Receiver Side + +The receiver-side logic lives at `x3dh_manager.py:283-350`. The operations are mirrored: + +```python +dh1 = bob_spk_private.exchange(alice_ik_public) # mirrors alice_ik.exchange(bob_spk) +dh2 = bob_ik_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_ik) +dh3 = bob_spk_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_spk) +dh4 = bob_opk_private.exchange(alice_ek_public) # mirrors alice_ek.exchange(bob_opk) +``` + +This works because of how Diffie-Hellman is defined: `A_private.exchange(B_public) == B_private.exchange(A_public)`. The math is commutative. Both sides compute the same four 32-byte DH outputs, concatenate them in the same order, and derive the same shared key through the same HKDF parameters (zero salt, `X3DH` info, `0xFF` padding). + +The receiver also constructs `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` in the same order (Alice's IK first, Bob's IK second) at `x3dh_manager.py:341`. Order matters. If Alice put hers first on the sender side but Bob reversed the order on the receiver side, the associated data would not match and decryption would fail. + +### Common Mistakes + +**BAD: Using random salt in HKDF** + +```python +# WRONG - breaks compatibility between sender and receiver +salt = os.urandom(32) +hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=salt, info=b'X3DH') +``` + +If you use a random salt, the sender and receiver will derive different shared keys. The sender would need to transmit the salt alongside the ephemeral key, and at that point you have added complexity for no security gain since the DH outputs already provide the randomness. + +**BAD: Forgetting to verify signed prekey signature** + +```python +# WRONG - allows MITM to substitute their own SPK +def perform_x3dh_sender(self, alice_identity_private_x25519, bob_bundle, ...): + # Skipped: self.verify_signed_prekey(...) + alice_ik_private_bytes = base64url_to_bytes(alice_identity_private_x25519) + ... +``` + +Without signature verification, an attacker who controls the network can replace Bob's SPK public key with their own. Alice would compute a shared secret with the attacker. The attacker then performs X3DH with Bob using Bob's real SPK, and relays messages between them. Neither Alice nor Bob detects the interception. The fix is on `x3dh_manager.py:217-220`: verify before exchanging. + +**BAD: Reusing one-time prekeys** + +```python +# WRONG - breaks forward secrecy for initial handshake +opk = get_opk_for_user(user_id) +# forgot to mark as used +# another conversation uses the same OPK +``` + +If two conversations use the same OPK, an attacker who later compromises Bob's OPK private key can compute the DH4 output for both conversations. That defeats the purpose of single-use keys. The fix is at `prekey_service.py:332`: `one_time_prekey.is_used = True` immediately upon retrieval, followed by `session.commit()`. + +### Frontend X3DH Implementation + +The frontend mirrors the backend X3DH in TypeScript using the WebCrypto API. The client-side sender logic lives in `x3dh.ts:115-173`: + +```typescript +export async function initiateX3DH( + identityKeyPair: IdentityKeyPair, + recipientBundle: PreKeyBundle +): Promise { + const signatureValid = await verifySignedPreKey( + recipientBundle.identity_key_ed25519, + recipientBundle.signed_prekey, + recipientBundle.signed_prekey_signature + ) + if (!signatureValid) { + throw new Error("Invalid signed prekey signature") + } + ... + const dh1 = await x25519DeriveSharedSecret(senderIdentityPrivate, recipientSignedPreKeyPublic) + const dh2 = await x25519DeriveSharedSecret(ephemeralKeyPair.privateKey, recipientIdentityPublic) + const dh3 = await x25519DeriveSharedSecret(ephemeralKeyPair.privateKey, recipientSignedPreKeyPublic) + ... + const sharedKey = await hkdfDerive(concatenated, EMPTY_SALT, X3DH_INFO, 32) +``` + +The HKDF derivation used here is from `primitives.ts:166-192`: + +```typescript +export async function hkdfDerive( + inputKeyMaterial: Uint8Array, + salt: Uint8Array, + info: Uint8Array, + outputLength: number = HKDF_OUTPUT_SIZE +): Promise { + const baseKey = await subtle.importKey( + "raw", inputKeyMaterial.buffer as ArrayBuffer, + { name: "HKDF" }, false, ["deriveBits"] + ) + const derivedBits = await subtle.deriveBits( + { name: "HKDF", hash: "SHA-256", + salt: salt.buffer as ArrayBuffer, + info: info.buffer as ArrayBuffer }, + baseKey, outputLength * 8 + ) + return new Uint8Array(derivedBits) +} +``` + +There is an important difference between the backend and frontend HKDF calls. The backend prepends `0xFF * 32` to the key material before calling HKDF (following the Signal spec literally). The frontend passes the concatenated DH outputs directly. Both produce a 32-byte shared key, but the padding difference means the backend and frontend X3DH implementations are not interchangeable for the same conversation. This is by design: in the E2E model, the frontend performs X3DH entirely on the client side, and the backend's X3DH is used only for the server-assisted key initialization path (which is a deprecated fallback). When client-side encryption is active, only the frontend's `x3dh.ts` code runs. + +The WebCrypto API's `subtle.deriveBits` with the `HKDF` algorithm handles the extract-then-expand steps internally. You provide the input key material, a salt, an info string, and the desired output length in bits (not bytes, which is why `outputLength * 8` appears). The `importKey` call with `{ name: "HKDF" }` creates a non-extractable key object suitable only for derivation, which prevents the raw key material from being accidentally exported or misused. + +The receiver-side X3DH in `x3dh.ts:175-215` mirrors the sender with the same key swaps as the Python version. The commutative property of ECDH ensures both sides derive the same 32-byte shared key. + +--- + +## Building the Double Ratchet + +The Double Ratchet provides ongoing forward secrecy and break-in recovery after the initial X3DH handshake. "Forward secrecy" means compromising the current key does not reveal past messages. "Break-in recovery" means that even if an attacker steals the current keys, future messages become unreadable once new DH ratchet steps occur. + +### Initialization + +The sender initializes at `double_ratchet.py:279-302`: + +```python +def initialize_sender( + self, + shared_key: bytes, + peer_public_key: bytes +) -> DoubleRatchetState: + dh_private = X25519PrivateKey.generate() + peer_public = X25519PublicKey.from_public_bytes(peer_public_key) + dh_output = dh_private.exchange(peer_public) + + root_key, sending_chain_key = self._kdf_rk(shared_key, dh_output) + + state = DoubleRatchetState( + root_key = root_key, + sending_chain_key = sending_chain_key, + receiving_chain_key = b'\x00' * HKDF_OUTPUT_SIZE, + dh_private_key = dh_private, + dh_peer_public_key = peer_public_key + ) + return state +``` + +The receiver initializes at `double_ratchet.py:304-321`: + +```python +def initialize_receiver( + self, + shared_key: bytes, + own_private_key: X25519PrivateKey +) -> DoubleRatchetState: + state = DoubleRatchetState( + root_key = shared_key, + sending_chain_key = b'\x00' * HKDF_OUTPUT_SIZE, + receiving_chain_key = b'\x00' * HKDF_OUTPUT_SIZE, + dh_private_key = own_private_key, + dh_peer_public_key = None + ) + return state +``` + +**Why does the sender do an extra DH step?** The sender generates a fresh DH keypair and immediately performs a DH exchange with Bob's SPK public key. This produces a `dh_output` which gets fed into `_kdf_rk` along with the X3DH `shared_key` to derive the first `root_key` and `sending_chain_key`. This extra step means the first message already has a DH ratchet contribution beyond the X3DH output. The receiver, by contrast, does not perform this step during initialization. It simply stores the X3DH shared key as the root key and waits for the first message, which will carry the sender's new DH public key and trigger the receiver's first ratchet step. + +### KDF Chain Operations + +The root key chain derivation at `double_ratchet.py:79-94`: + +```python +def _kdf_rk(self, root_key: bytes, dh_output: bytes) -> tuple[bytes, bytes]: + hkdf = HKDF( + algorithm = hashes.SHA256(), + length = HKDF_OUTPUT_SIZE * 2, + salt = root_key, + info = b'', + ) + output = hkdf.derive(dh_output) + new_root_key = output[: HKDF_OUTPUT_SIZE] + new_chain_key = output[HKDF_OUTPUT_SIZE :] + return new_root_key, new_chain_key +``` + +`_kdf_rk` is the root chain KDF. It runs every time a DH ratchet step occurs (when the sender or receiver generates a new DH keypair). The `root_key` is used as the HKDF salt, and the fresh DH output is the input key material. HKDF produces 64 bytes (`HKDF_OUTPUT_SIZE * 2 = 64`), which is split in half: the first 32 bytes become the new root key, and the second 32 bytes become the new chain key. The root key never directly encrypts anything. It acts as a "master key" that seeds each new chain. Because the root key is updated with fresh DH output every ratchet step, even if an attacker compromises the current root key, the next ratchet step will incorporate a new DH output that the attacker does not know (assuming the DH private key is secure), and the root key will be "healed." This is the break-in recovery property. + +The empty `info` parameter is intentional. The Signal spec does not use an info string for the root chain KDF. Domain separation happens implicitly because the root chain HKDF always uses the root key as salt, whereas the X3DH HKDF uses a zero salt. Different salts produce different outputs even with the same input, so there is no cross-contamination. + +The chain key derivation at `double_ratchet.py:96-109`: + +```python +def _kdf_ck(self, chain_key: bytes) -> tuple[bytes, bytes]: + h_chain = hmac.HMAC(chain_key, hashes.SHA256()) + h_chain.update(b'\x01') + next_chain_key = h_chain.finalize() + + h_message = hmac.HMAC(chain_key, hashes.SHA256()) + h_message.update(b'\x02') + message_key = h_message.finalize() + + return next_chain_key, message_key +``` + +**Why HMAC with 0x01 and 0x02?** The chain key needs to produce two outputs: the next chain key and the message key. Using HMAC with different single-byte constants is the simplest way to derive two independent values from one input. `0x01` produces the next chain key, `0x02` produces the message key. The constants are arbitrary but fixed by convention (the Signal spec uses these exact values). The important thing is that they are different, so the two HMAC outputs are cryptographically independent. + +**Why two separate HMACs?** You cannot reuse an HMAC object after calling `finalize()` in the Python `cryptography` library. Each HMAC computation is a fresh instance. Even if you could reuse it, producing two different outputs from the same key requires two different inputs, which means two operations. + +**What is the one-way property?** Given `next_chain_key = HMAC(chain_key, 0x01)`, you cannot compute `chain_key` from `next_chain_key`. HMAC is a one-way function. This means that if an attacker obtains a message key for message N, they cannot derive the chain key that produced it, and therefore cannot derive message keys for messages 0 through N-1. This is forward secrecy within a single chain. + +### Message Encryption + +Here is the encrypt path from `double_ratchet.py:323-362`: + +```python +def encrypt_message(self, state, plaintext, associated_data): + state.sending_chain_key, message_key = self._kdf_ck(state.sending_chain_key) + nonce, ciphertext = self._encrypt_with_message_key(message_key, plaintext, associated_data) + + if state.dh_private_key: + dh_public = state.dh_private_key.public_key() + dh_public_bytes = dh_public.public_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PublicFormat.Raw + ) + else: + dh_public_bytes = b'\x00' * X25519_KEY_SIZE + + encrypted_msg = EncryptedMessage( + ciphertext = ciphertext, + nonce = nonce, + dh_public_key = dh_public_bytes, + message_number = state.sending_message_number, + previous_chain_length = state.previous_sending_chain_length + ) + state.sending_message_number += 1 + return encrypted_msg +``` + +And the underlying AES-GCM encryption at `double_ratchet.py:111-130`: + +```python +def _encrypt_with_message_key(self, message_key, plaintext, associated_data): + aesgcm = AESGCM(message_key) + nonce = os.urandom(AES_GCM_NONCE_SIZE) + ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data) + return nonce, ciphertext +``` + +**Why include dh_public_key in the header?** The recipient needs the sender's current DH public key to determine whether to perform a DH ratchet step. If the DH public key in the message header differs from the last known peer DH key, the recipient knows the sender has ratcheted, and the recipient must also ratchet to derive the correct receiving chain key. Without this header field, the recipient would have no way to know which chain to use. + +**Why track message_number?** Messages can arrive out of order over WebSocket. If the recipient receives message 5 before message 3, it needs to know that messages 3 and 4 were skipped. The `message_number` tells the recipient exactly which position in the chain produced this message's key. The `previous_chain_length` tells the recipient how many messages were sent on the previous chain before the sender ratcheted, which is needed to compute skipped keys from the old chain. + +### Message Decryption + +The decrypt path at `double_ratchet.py:364-416` handles three distinct cases: + +```python +def decrypt_message(self, state, encrypted_msg, associated_data): + # Case 1: Check if we already stored this message's key (out-of-order) + skipped_key = self._try_skipped_message_key( + state, encrypted_msg.dh_public_key, encrypted_msg.message_number + ) + if skipped_key: + return self._decrypt_with_message_key( + skipped_key, encrypted_msg.nonce, encrypted_msg.ciphertext, associated_data + ) + + # Case 2: New DH ratchet key means sender ratcheted + if encrypted_msg.dh_public_key != state.dh_peer_public_key: + if state.dh_peer_public_key: + self._store_skipped_message_keys( + state, encrypted_msg.previous_chain_length, state.dh_peer_public_key + ) + self._dh_ratchet_receive(state, encrypted_msg.dh_public_key) + + # Case 3: Message number ahead of current position means skipped messages + if encrypted_msg.message_number > state.receiving_message_number: + self._store_skipped_message_keys( + state, encrypted_msg.message_number, encrypted_msg.dh_public_key + ) + + # Advance chain and decrypt + state.receiving_chain_key, message_key = self._kdf_ck(state.receiving_chain_key) + state.receiving_message_number += 1 + + plaintext = self._decrypt_with_message_key( + message_key, encrypted_msg.nonce, encrypted_msg.ciphertext, associated_data + ) + return plaintext +``` + +**Case 1: Skipped key hit.** The message key was already computed and stored during a previous skip operation. Pop it from the dict, decrypt, done. This handles previously skipped messages arriving late. + +**Case 2: New ratchet step.** The DH public key in the message header does not match the last known peer public key. This means the sender performed a DH ratchet step. The receiver must first store skipped keys for any remaining messages on the old chain (up to `previous_chain_length`), then perform its own DH ratchet step to derive the new receiving chain key. + +**Case 3: Same chain, skipped messages.** The message number is ahead of the current position on the same chain. The receiver must advance the chain key, storing each intermediate message key, until it reaches the correct position. + +### Out-of-Order Handling + +The skip mechanism at `double_ratchet.py:215-244`: + +```python +def _store_skipped_message_keys(self, state, until_message_number, dh_public_key): + num_to_skip = until_message_number - state.receiving_message_number + + if num_to_skip > self.max_skip: + raise ValueError( + f"Cannot skip {num_to_skip} messages " + f"(MAX_SKIP={self.max_skip})" + ) + + if len(state.skipped_message_keys) + num_to_skip > self.max_cache: + self._evict_oldest_skipped_keys(state, num_to_skip) + + chain_key = state.receiving_chain_key + for msg_num in range(state.receiving_message_number, until_message_number): + chain_key, message_key = self._kdf_ck(chain_key) + state.skipped_message_keys[(dh_public_key, msg_num)] = message_key + + state.receiving_chain_key = chain_key +``` + +**MAX_SKIP = 1000** (`config.py:73`). If an attacker sends a message claiming `message_number = 999999999`, the receiver would need to compute 999999999 HMAC operations to derive all the intermediate keys. That is a denial-of-service attack. The limit of 1000 means the receiver will reject any message that would require skipping more than 1000 positions. In practice, even aggressive out-of-order delivery rarely exceeds a few dozen skipped messages. + +**MAX_CACHE = 2000** (`config.py:74`). Even with the skip limit, a determined attacker could slowly accumulate skipped keys by sending many small gaps. The cache limit prevents unbounded memory growth. When the cache is full, `_evict_oldest_skipped_keys` removes the oldest entries to make room. Oldest entries are the ones most likely to be genuinely lost (if a message has not arrived after 2000 other skipped keys were stored, it is probably never coming). + +### Common Mistakes + +**Not checking max skip (DoS via huge message number gaps).** Without the `num_to_skip > self.max_skip` check, an attacker can force the receiver to compute millions of HMAC operations by sending a single message with a large message number. The CPU cost is linear in the gap size. + +**Not evicting old skipped keys (memory exhaustion).** Without eviction, the `skipped_message_keys` dict grows without bound. An attacker can cause OOM by sending many messages with incrementing but non-sequential message numbers across multiple ratchet steps. + +**Reusing nonces (catastrophic for AES-GCM).** AES-GCM uses a 12-byte nonce. If the same nonce is used twice with the same key, an attacker can XOR the two ciphertexts to eliminate the keystream and recover both plaintexts. The code uses `os.urandom(AES_GCM_NONCE_SIZE)` at `double_ratchet.py:122` for a fresh random nonce every time. Since each message key is used exactly once (each chain key advances after derivation), and each message key gets a random nonce, the probability of a collision is negligible. But if someone modified the code to reuse message keys, the random nonce would be the only defense, and with 12 bytes, birthday collisions become likely after about 2^48 messages under the same key. + +--- + +## Building WebAuthn Authentication + +### Registration Flow + +Registration is handled in `passkey_manager.py:55-94`. The server generates registration options: + +```python +def generate_registration_options( + self, user_id, username, display_name, exclude_credentials +): + challenge = secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES) + ... + options = generate_registration_options( + rp_id = self.rp_id, + rp_name = self.rp_name, + user_id = user_id, + user_name = username, + challenge = challenge, + attestation = AttestationConveyancePreference.NONE, + authenticator_selection = AuthenticatorSelectionCriteria( + resident_key = ResidentKeyRequirement.REQUIRED, + user_verification = UserVerificationRequirement.PREFERRED, + ), + exclude_credentials = exclude_creds, + ) +``` + +**ResidentKeyRequirement.REQUIRED** means the authenticator must store the credential on the device itself (a "discoverable credential" or "passkey"). This allows usernameless login: the user just taps their security key or uses biometrics, and the authenticator presents the credential without the user typing a username. Without REQUIRED, the server would need to provide a credential ID hint during authentication, which means the user must provide their username first. + +**Challenge generation** uses `secrets.token_bytes(32)` at `passkey_manager.py:65`. The `secrets` module is Python's CSPRNG interface, equivalent to `/dev/urandom`. The challenge must be unpredictable to prevent replay attacks. The challenge is stored in Redis with a TTL of 600 seconds (`WEBAUTHN_CHALLENGE_TTL_SECONDS = 600` in `config.py:84`), so it expires before an attacker could realistically capture and replay it. + +**Attestation verification** at `passkey_manager.py:96-130` calls `verify_registration_response` from py_webauthn. This checks that the authenticator's response was correctly signed, that the RP ID matches, that the origin matches, and that the challenge matches. The `expected_origin = self.rp_origin` check at `passkey_manager.py:109` is critical: it ensures the registration happened from the legitimate frontend, not a phishing page on a different domain. + +### Authentication Flow + +Authentication follows the same challenge-response pattern. The server generates options at `passkey_manager.py:132-160`, sends the challenge to the client, the client's authenticator signs the challenge with its private key, and the server verifies at `passkey_manager.py:162-207`. + +**Why challenge-response?** Without a fresh challenge, an attacker who intercepts one authentication response could replay it forever. The challenge ensures each authentication is unique. The authenticator signs the challenge (along with other data like the RP ID and origin) with its private key. The server verifies the signature using the stored public key. Even if the signed response is intercepted, it cannot be reused because the next authentication will have a different challenge. + +**Why check origin?** The origin check (`expected_origin = self.rp_origin`) at `passkey_manager.py:177` prevents credential phishing. If an attacker sets up a fake site at `evil-chat.com` and tricks the user into authenticating, the authenticator will include `evil-chat.com` as the origin in the signed data. When the attacker tries to forward that signed response to the real server, the origin check fails because the server expects `http://localhost:3000` (or whatever the production origin is). The authenticator binds the signature to the origin, making phishing structurally impossible. + +### Clone Detection + +From `passkey_manager.py:184-193`: + +```python +if (credential_current_sign_count != 0 and new_sign_count != 0 + and new_sign_count <= credential_current_sign_count): + logger.error( + "Signature counter did not increase: current=%s, new=%s. Possible cloned authenticator detected!", + credential_current_sign_count, + new_sign_count + ) + raise ValueError( + "Signature counter anomaly detected - potential cloned authenticator" + ) +``` + +**What cloning means.** If someone physically copies a hardware security key (or extracts the private key from a software authenticator), both the original and the clone have the same credential. Each has its own signature counter starting from the same value. When the original authenticates, its counter increments to N+1. When the clone authenticates, its counter also increments, but from N, producing N+1 (or a different value depending on when each was used). If the server sees a counter that is less than or equal to the last recorded counter, it knows two devices are using the same credential. + +**Why counter must increase.** Each authentication increments the counter by at least 1. If the server stored counter=5 and the next authentication reports counter=4, something is wrong. Either the counter rolled back (impossible in a correct implementation) or a different device with an older counter value is being used. + +**Edge case with counters at 0.** Some authenticators (particularly platform authenticators like Touch ID or Windows Hello) always report counter=0 and never increment it. The condition `credential_current_sign_count != 0 and new_sign_count != 0` handles this: if either counter is 0, the check is skipped entirely. This means clone detection is not available for those authenticators, but rejecting them would lock out a large portion of users. The trade-off is acceptable because platform authenticators are inherently harder to clone (they are bound to the device's secure enclave). + +--- + +## Building the WebSocket Layer + +### Connection Management + +From `websocket_manager.py:43-95`: + +```python +async def connect(self, websocket, user_id): + await websocket.accept() + if user_id not in self.active_connections: + self.active_connections[user_id] = [] + if len(self.active_connections[user_id]) >= WS_MAX_CONNECTIONS_PER_USER: + await self._send_error(websocket, "max_connections", ...) + await websocket.close() + return False + self.active_connections[user_id].append(websocket) + await presence_service.set_user_online(user_id) + self.heartbeat_tasks[user_id] = asyncio.create_task(self._heartbeat_loop(websocket, user_id)) + await self._subscribe_to_messages(user_id) + return True +``` + +**Why per-user connection lists?** A user might have the chat open on their phone and their laptop simultaneously. Each device creates its own WebSocket connection. The `active_connections` dict maps `UUID -> list[WebSocket]`, so when a message arrives for a user, it gets sent to all of their active connections. Without this, multi-device support would not work. + +**Why max 5?** `WS_MAX_CONNECTIONS_PER_USER = 5` (from `config.py:142`). Without a limit, a malicious client could open thousands of WebSocket connections and exhaust server memory. Five is generous enough for legitimate multi-device usage (phone, tablet, laptop, desktop, maybe one more) while capping resource consumption. The specific number is configurable via environment variable. + +**Why heartbeat?** WebSocket connections can silently die. A user closes their laptop lid, their WiFi drops, or a NAT gateway times out the connection. The TCP stack might not detect the dead connection for minutes or hours. The heartbeat loop at `websocket_manager.py:177-201` sends a ping every `WS_HEARTBEAT_INTERVAL` (30 seconds by default). If the send fails, it means the connection is dead, and `disconnect` cleans it up. This keeps the `active_connections` dict accurate and ensures presence status reflects reality. + +### Live Query Subscription + +From `websocket_manager.py:203-223`: + +```python +async def _subscribe_to_messages(self, user_id): + def message_callback(update: LiveMessageUpdate): + asyncio.create_task(self._handle_live_message(user_id, update)) + + live_id = await surreal_db.live_messages_for_user( + user_id = str(user_id), + callback = message_callback + ) + self.live_query_ids[user_id] = live_id +``` + +**SurrealDB pushes new messages to the server, server forwards to client. No polling needed.** SurrealDB has a native live query feature. When you register a live query like `LIVE SELECT * FROM messages WHERE recipient_id = $user_id`, SurrealDB pushes every new matching record to the callback in real time. The server does not need to poll the database on a timer. When a new message is created in SurrealDB (by `message_service.store_encrypted_message`), the live query fires, `_handle_live_message` at `websocket_manager.py:225-251` picks it up, wraps it in a WebSocket message schema, and sends it to all of that user's active connections. + +This is more efficient than polling because there is zero latency (the callback fires the instant the record is created) and zero wasted queries (no empty poll cycles when no messages are pending). + +### Dead Connection Cleanup + +The `disconnect` method at `websocket_manager.py:97-133` handles cleanup: + +```python +async def disconnect(self, websocket, user_id): + if user_id in self.active_connections: + if websocket in self.active_connections[user_id]: + self.active_connections[user_id].remove(websocket) + + if not self.active_connections[user_id]: + del self.active_connections[user_id] + await presence_service.set_user_offline(user_id) + # Kill live query + # Cancel heartbeat task +``` + +Dead connections are also detected during `send_message` at `websocket_manager.py:135-153`. If sending to a WebSocket raises an exception, that WebSocket is added to a `dead_connections` list, and each dead connection is cleaned up via `disconnect` after the send loop completes. This lazy cleanup means the system self-heals: even if a heartbeat fails to detect a dead connection, the next message send attempt will catch it. + +The cleanup only sets the user offline and kills the live query when the last connection for that user is removed. If they still have another tab open, their other connection keeps working and they stay online. + +--- + +## Building the Message Pipeline + +### Server-Side Storage (Passthrough) + +From `message_service.py:269-314`: + +```python +async def store_encrypted_message( + self, session, sender_id, recipient_id, ciphertext, nonce, header, room_id +): + ... + surreal_message = { + "sender_id": str(sender_id), + "recipient_id": str(recipient_id), + "room_id": room_id, + "ciphertext": ciphertext, + "nonce": nonce, + "header": header, + "sender_username": sender_user.username, + ... + } + result = await surreal_db.create_message(surreal_message) +``` + +The server stores encrypted data without any decryption capability. The `ciphertext`, `nonce`, and `header` fields are base64-encoded strings that arrive from the client and get stored as-is. The server never sees the plaintext. The server never sees the encryption keys. The server could be fully compromised and the attacker still could not read messages, because the message keys exist only on the clients' devices in IndexedDB. + +This is the correct E2E approach. Many "encrypted" chat systems encrypt on the server side, which means the server has the keys and could decrypt everything. Here, the encryption happens in `crypto-service.ts` in the browser, and the server is a dumb relay. + +### Conversation Initialization + +From `message_service.py:48-166`: + +```python +async def initialize_conversation(self, session, sender_id, recipient_id): + # Check for existing ratchet state + # Fetch sender's identity key + # Fetch recipient's prekey bundle + # Perform X3DH sender-side + x3dh_result = x3dh_manager.perform_x3dh_sender( + alice_identity_private_x25519 = sender_ik.private_key, + bob_bundle = recipient_bundle, + bob_identity_public_ed25519 = recipient_ik.public_key_ed25519 + ) + # Initialize Double Ratchet with shared key + dr_state = double_ratchet.initialize_sender( + shared_key = x3dh_result.shared_key, + peer_public_key = recipient_spk_public_bytes + ) + # Serialize and persist ratchet state to PostgreSQL +``` + +This is the bridge between key exchange and ongoing encryption. The method orchestrates the entire flow: check if a conversation already exists (idempotency), fetch both users' keys from PostgreSQL, perform the X3DH exchange, initialize the Double Ratchet with the resulting shared key, serialize the ratchet state, and persist it. After this method completes, `encrypt_message` and `decrypt_message` can operate using the stored ratchet state. + +The check at `message_service.py:60-73` is important: if a ratchet state already exists for this sender-recipient pair, the method returns it without doing anything. This prevents accidental re-initialization, which would generate a new shared secret and break the existing conversation. + +--- + +## Security Implementation Details + +### Constant-Time Comparison + +From `primitives.ts:388-397`: + +```typescript +export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean { + if (a.length !== b.length) return false + let result = 0 + for (let i = 0; i < a.length; i++) { + result |= a[i] ^ b[i] + } + return result === 0 +} +``` + +**Why XOR + OR instead of byte-by-byte comparison?** A naive comparison like `a[0] === b[0] && a[1] === b[1] && ...` short-circuits on the first mismatch. If bytes 0 through 5 match but byte 6 does not, the comparison returns `false` after checking 7 bytes. An attacker measuring the response time can determine that the first 6 bytes are correct and only byte 6 is wrong. By trying all 256 values for byte 6 and measuring which one takes slightly longer (because the comparison proceeds to byte 7), they can guess the correct value. Repeat for each byte and the entire secret is recovered. + +The constant-time version XORs every byte pair and ORs the results into an accumulator. If any byte differs, at least one bit in `result` will be set. The function always processes every byte regardless of where a mismatch occurs. The execution time depends only on the array length, not on the content. + +The length check at the top does leak whether the lengths are equal. This is generally acceptable because key lengths are not secret. But if length were secret, you would need to also make that comparison constant-time. + +### Random Number Generation + +On the Python backend, `os.urandom(AES_GCM_NONCE_SIZE)` at `double_ratchet.py:122` reads from the operating system's cryptographically secure random number generator (CSPRNG). On Linux, this is backed by `/dev/urandom`, which draws entropy from hardware events (disk timing, network interrupts, etc.) and maintains a CSPRNG state that is computationally indistinguishable from true randomness. + +On the browser frontend, the `generateRandomBytes` function at `primitives.ts:294-298`: + +```typescript +export function generateRandomBytes(length: number): Uint8Array { + const bytes = new Uint8Array(length) + crypto.getRandomValues(bytes) + return bytes +} +``` + +This calls the WebCrypto API's CSPRNG. Every major browser implements this using the OS CSPRNG under the hood. Chrome uses BoringSSL's CSPRNG, Firefox uses NSS, Safari uses CommonCrypto. All ultimately draw from the kernel's entropy pool. + +The same function is used for nonce generation in `aesGcmEncrypt` at `primitives.ts:243`: `const nonce = generateRandomBytes(AES_GCM_NONCE_SIZE)`. It is also used for WebAuthn challenge generation on the server side via `secrets.token_bytes(WEBAUTHN_CHALLENGE_BYTES)` at `passkey_manager.py:65`, which is Python's wrapper around `os.urandom`. + +**Why CSPRNG?** Nonces, challenges, and ephemeral keys must be unpredictable. If an attacker can predict the next nonce, they can precompute the keystream for AES-GCM and decrypt messages in real time. If they can predict the next ephemeral key, they can compute the DH shared secret. If they can predict the next WebAuthn challenge, they can pre-sign a response and replay it. + +**Why not Math.random()?** `Math.random()` uses a PRNG (pseudorandom number generator) seeded from a low-entropy source (often just the current time or a simple counter). Most JavaScript engines use xorshift128+ or a similar fast PRNG. The internal state is typically 128 bits, and it can be reconstructed by observing as few as 4-5 consecutive outputs. An attacker who knows the internal state of `Math.random()` can predict every future output, which would compromise every nonce and every ephemeral key generated by the application. In 2015, researchers demonstrated recovering the full xorshift128+ state from V8's `Math.random()` by observing just a few outputs. `Math.random()` is fine for shuffling a deck of cards in a UI animation. It is catastrophic for cryptography. + +### Key Serialization + +All keys are serialized as base64url-encoded raw bytes. For example, in `x3dh_manager.py:67-85`: + +```python +private_bytes = private_key.private_bytes( + encoding = serialization.Encoding.Raw, + format = serialization.PrivateFormat.Raw, + encryption_algorithm = serialization.NoEncryption() +) +... +return ( + bytes_to_base64url(private_bytes), + bytes_to_base64url(public_bytes) +) +``` + +**Why base64url instead of hex?** Hex encoding represents each byte as two hexadecimal characters, so 32 bytes becomes 64 characters. Base64url represents every 3 bytes as 4 characters, so 32 bytes becomes approximately 43 characters. That is a 33% space saving. When you are storing thousands of keys and transmitting them over WebSocket, the savings add up. + +**Why Raw format instead of PEM?** PEM format wraps the key in `-----BEGIN PRIVATE KEY-----` headers, base64 encodes a DER-encoded ASN.1 structure, and adds newlines every 64 characters. For a 32-byte X25519 private key, PEM produces roughly 120 bytes. Raw format produces exactly 32 bytes. PEM is useful when you need to identify the key algorithm from the encoding (the ASN.1 OID tells you "this is X25519"). Here, both sides already know the algorithm, so the metadata is waste. + +--- + +## Data Flow: Complete Message Trace + +Here is what happens when Alice types "Hello Bob" and it reaches Bob's screen. Every step, every file, every transformation. + +**1. Alice types "Hello Bob" in Chat.tsx.** The `ChatInput` component captures the text in a form submit handler. The handler creates a temporary message ID (a random string for optimistic UI updates) and calls into the message sending logic. The plaintext exists only in the browser's JavaScript heap at this point. + +**2. crypto-service.ts `encrypt("bob-uuid", "Hello Bob")`.** The `CryptoService.encrypt` method at `crypto-service.ts:217-249` is invoked. It first checks whether a ratchet session exists with Bob. + +**3. Load ratchet state from IndexedDB.** `getRatchetState(peerId)` at `crypto-service.ts:303-315` checks the in-memory `Map` first (keyed by peer ID). If the state is not in memory, it queries IndexedDB via `key-store.ts`, deserializes the stored JSON back into a `DoubleRatchetState` object (reconstructing CryptoKey objects from base64-encoded bytes), and caches it. If no state exists at all, `establishSession(peerId)` triggers the full X3DH flow: fetch Bob's prekey bundle from the server via `api.encryption.getPrekeyBundle`, perform client-side X3DH via `x3dh.ts:initiateX3DH`, and initialize the sending ratchet via `double-ratchet.ts:initializeRatchetSender`. + +**4. double-ratchet.ts `encryptMessage`: advance chain key via HMAC.** At `double-ratchet.ts:167`, `deriveMessageKey(state.sending_chain_key)` computes two values: + - `messageKey = HMAC-SHA-256(chainKey, "MessageKey")` - the key that will encrypt this specific message + - `nextChainKey = HMAC-SHA-256(chainKey, "ChainKey")` - the chain key for the next message + +The old `sending_chain_key` is immediately overwritten with `nextChainKey`. The `messageKey` is used once and never stored. This is the forward secrecy mechanism: after encryption, the message key cannot be recovered from the updated chain state. + +**5. primitives.ts `aesGcmEncrypt` with message key.** At `double-ratchet.ts:178`, the 32-byte `messageKey` is imported as an AES-GCM CryptoKey. The `plaintext` bytes (`TextEncoder.encode("Hello Bob")` = 9 bytes) are encrypted. A 12-byte random nonce is generated via `crypto.getRandomValues`. The message header (DH public key, message number, previous chain length) is serialized to JSON and used as Additional Authenticated Data (AAD). AES-GCM produces the ciphertext (9 bytes of encrypted data + 16 bytes of authentication tag = 25 bytes total). + +**6. Return {ciphertext, nonce, header}.** The `EncryptedMessage` object at `double-ratchet.ts:183-187` contains raw `Uint8Array` ciphertext and nonce, plus a typed `MessageHeader` object. Back in `crypto-service.ts:244-248`, the ciphertext and nonce are base64-encoded to strings, and the header (including any pending X3DH header for first messages) is JSON-stringified into a single string. The result is three strings: `ciphertext`, `nonce`, and `header`. All three are opaque to anyone without the message key. + +**7. WebSocket sends JSON payload.** `websocket-manager.ts:108-124` calls `sendEncryptedMessage`, which constructs the outgoing message: + +```json +{ + "type": "encrypted_message", + "recipient_id": "bob-uuid", + "room_id": "room-uuid", + "ciphertext": "base64...", + "nonce": "base64...", + "header": "{\"ratchet\":{\"dh_public_key\":\"...\",\"message_number\":0,...}}", + "temp_id": "temp-abc123" +} +``` + +This JSON is sent via `ws.send(JSON.stringify(message))`. If the WebSocket is not connected, the message is queued in `messageQueue` and sent when the connection is restored. + +**8. websocket.py:46-54 receives.** The `websocket_endpoint` at `websocket.py:46` calls `data = await websocket.receive_text()`, parses the JSON with `json.loads(data)`, and passes the resulting dict to `websocket_service.route_message`. At this point the server has the encrypted blob. It cannot decrypt it. It does not have the message key, the chain key, or any ratchet state for Alice and Bob's conversation (in the E2E model). + +**9. websocket_service.py routes to handler.** At `websocket_service.py:63`, the message type `"encrypted_message"` dispatches to `handle_encrypted_message` at `websocket_service.py:87-179`. The handler extracts `recipient_id`, `room_id`, `ciphertext`, `nonce`, and `header` from the message dict. + +**10. message_service.py:269-314 stores in SurrealDB.** The handler calls `message_service.store_encrypted_message`, which constructs a SurrealDB document: + +```python +surreal_message = { + "sender_id": str(sender_id), + "recipient_id": str(recipient_id), + "room_id": room_id, + "ciphertext": ciphertext, # still base64, still encrypted + "nonce": nonce, # still base64 + "header": header, # still JSON string + "sender_username": sender_user.username, + "created_at": now.isoformat(), + "updated_at": now.isoformat(), +} +``` + +The document is written to SurrealDB via `surreal_db.create_message(surreal_message)`. The server stores the encrypted blob verbatim. No decryption. No key access. + +**11. SurrealDB live query fires.** The instant the message record is created, SurrealDB pushes a `CREATE` event to any registered live query matching `recipient_id = bob_uuid`. This is a push, not a pull. There is no polling interval. The latency from write to callback is measured in single-digit milliseconds. + +**12. websocket_manager.py:225-251 picks up the update.** The `_handle_live_message` callback at `websocket_manager.py:225` receives the `LiveMessageUpdate`. It checks that `update.action == "CREATE"` (ignoring updates and deletes). It wraps the message data in an `EncryptedMessageWS` Pydantic schema and calls `self.send_message(user_id, ws_message.model_dump(mode="json"))`. + +**13. ConnectionManager sends to Bob's WebSocket(s).** At `websocket_manager.py:135-153`, the manager iterates over all of Bob's active connections (he might have the app open on his phone and laptop) and calls `websocket.send_json(message)` on each. If any send fails, that connection is added to a `dead_connections` list and cleaned up after the loop. + +**14. Bob's websocket-manager.ts receives.** The `handleMessage` method at `websocket-manager.ts:166-178` parses the JSON from `event.data`. The `routeMessage` method at `websocket-manager.ts:180-201` runs through type guards. `isEncryptedMessageWS(message)` matches because the message has `type: "encrypted_message"` and the required fields. The message is dispatched to `handleWSMessage` in `message-handlers.ts`. + +**15. crypto-service.ts `decrypt(alice_id, ciphertext, nonce, header)`.** The message handler calls `CryptoService.decrypt` at `crypto-service.ts:251-301`. The `header` string is parsed to extract the ratchet header (DH public key, message number, previous chain length) and optionally an X3DH header (if this is the first message in a new conversation). + +**16. Load ratchet state from IndexedDB.** Same mechanism as step 3, but for Bob's side. If no ratchet state exists for Alice, and the message includes an X3DH header, `handleIncomingSession` at `crypto-service.ts:168-215` triggers: + - Look up Alice's identity key from the X3DH header + - Look up Bob's signed prekey private key from IndexedDB + - Look up the consumed one-time prekey if one was used + - Call `receiveX3DH` from `x3dh.ts:175-215` to compute the same shared secret Alice computed + - Call `initializeRatchetReceiver` from `double-ratchet.ts:68-88` with the shared secret and Bob's signed prekey pair + +**17. Check skipped keys, advance receiving chain.** `decryptMessage` at `double-ratchet.ts:190-238` runs through the three-case logic: + - Check if this message's key was previously stored (skipped key cache hit) + - Check if Alice's DH public key in the header differs from the last known key (DH ratchet step needed) + - Check if the message number is ahead of the current position (skip intermediate keys) + Then derive the message key: `messageKey = HMAC-SHA-256(receivingChainKey, "MessageKey")`. + +**18. primitives.ts `aesGcmDecrypt`.** At `double-ratchet.ts:232`, AES-256-GCM decryption is performed. The 32-byte message key is imported as a CryptoKey. The 12-byte nonce from the message is used as the IV. The header JSON is reconstructed and used as AAD. `subtle.decrypt` verifies the 16-byte authentication tag and produces the 9 plaintext bytes. If the tag does not match (tampered message, wrong key, wrong nonce, wrong AAD), the WebCrypto API throws an `OperationError`. + +**19. Return "Hello Bob".** The `Uint8Array` plaintext bytes are decoded via `new TextDecoder().decode(plaintextBytes)` at `crypto-service.ts:300`, producing the string `"Hello Bob"`. The ratchet state is serialized and saved back to IndexedDB. + +**20. Chat.tsx renders plaintext.** The decrypted message is added to the message store. The reactive SolidJS component re-renders the message list. Bob sees "Hello Bob" on screen. The entire journey from Alice's keypress to Bob's screen took: + - ~1ms for client-side encryption (HMAC + AES-GCM) + - ~10-50ms for WebSocket round trip (depends on network) + - ~1ms for SurrealDB write + live query fire + - ~10-50ms for WebSocket delivery to Bob + - ~1ms for client-side decryption + Total: roughly 25-100ms end to end. + +The plaintext "Hello Bob" existed in memory on Alice's device and Bob's device. At no other point in the pipeline, not on the WebSocket server, not in SurrealDB, not in PostgreSQL, not in Redis, did the plaintext exist. That is what end-to-end encryption means in practice. + +--- + +## Error Handling Patterns + +### Encryption Failures + +When encryption fails (corrupted ratchet state, invalid key, etc.), an `EncryptionError` is raised at `message_service.py:366`. The exception hierarchy in `exceptions.py:73-76` derives from `AppException`. The exception handler returns the error to the client via the WebSocket error message type. The message is never stored, because the encryption failed before the store call. + +### Database Transaction Failures + +`IntegrityError` is caught in every service method that writes to the database. For example, at `message_service.py:161-164`: + +```python +except IntegrityError as e: + await session.rollback() + raise DatabaseError("Failed to initialize conversation") from e +``` + +The pattern is consistent: catch the SQLAlchemy integrity error, roll back the session to a clean state, then raise a custom `DatabaseError`. This ensures the database is never left in a partially committed state. The `from e` chain preserves the original exception for logging. + +### WebSocket Disconnection + +When a WebSocket disconnects (user closes the tab, network drops), `WebSocketDisconnect` is caught at `websocket.py:79`. The `finally` block at `websocket.py:84` always runs `connection_manager.disconnect(websocket, user_uuid)`, which: +- Removes the WebSocket from the connection pool +- If it was the last connection for that user: kills the SurrealDB live query, cancels the heartbeat task, and sets presence to offline via Redis +- If other connections remain: just removes this one and logs the remaining count + +--- + +## Testing Strategy + +### Unit Tests + +**Test X3DH key agreement.** Generate identity keys for Alice and Bob. Generate a signed prekey for Bob. Generate an OPK for Bob. Assemble Bob's prekey bundle. Perform sender-side X3DH as Alice via `x3dh_manager.perform_x3dh_sender`. Perform receiver-side X3DH as Bob via `x3dh_manager.perform_x3dh_receiver`, passing Alice's ephemeral public key and identity key from the sender result. Assert that `alice_result.shared_key == bob_result.shared_key`. This verifies the commutative DH math. Also assert that `alice_result.associated_data == bob_result.associated_data`, confirming both sides compute the same AAD. + +Test the OPK-absent case separately: set `bob_bundle.one_time_prekey = None`, pass `bob_one_time_prekey_private = None` on the receiver side. Verify the shared keys still match (they will, because both sides fall back to the 3-DH variant). + +Test signature verification failure: corrupt one byte of the SPK signature, call `perform_x3dh_sender`, assert it raises `ValueError("Invalid signed prekey signature")`. + +**Test Double Ratchet round-trip.** Initialize sender and receiver with the shared key from X3DH. Encrypt "message 1" on the sender side. Decrypt on the receiver side. Assert plaintext matches. Then encrypt "message 2" on the receiver side and decrypt on the sender side. This tests a full ratchet cycle where the DH ratchet advances twice (once for each direction change). + +Continue with a longer sequence: sender sends 5 messages in a row, receiver decrypts all 5 (this tests the symmetric chain without DH ratcheting), then receiver sends 3 messages back, sender decrypts all 3. Verify every plaintext. + +**Test out-of-order delivery.** Encrypt messages 0, 1, 2, 3, 4 on the sender. Deliver message 4 first. The receiver should store skipped keys for messages 0, 1, 2, 3, decrypt message 4 successfully, then decrypt messages 2, 0, 3, 1 in any order using the stored skipped keys. Verify all five plaintexts are correct. Verify the skipped key cache is empty after all messages are decrypted (each key is consumed on use). + +**Test max skip enforcement.** Construct a `DoubleRatchet(max_skip=5)`. Encrypt messages 0 through 20 on the sender. Attempt to deliver message 20 first (requiring 20 skips). Assert the receiver raises `ValueError("Cannot skip 20 messages (MAX_SKIP=5)")`. Then deliver message 4 (requiring 4 skips). Assert it succeeds. This confirms the limit is enforced and legitimate small gaps still work. + +**Test cache eviction.** Construct a `DoubleRatchet(max_skip=100, max_cache=10)`. Skip 10 messages to fill the cache. Skip 5 more. Assert that the cache now has 10 entries (5 old ones were evicted to make room). Attempt to decrypt one of the evicted messages. Assert decryption fails (the key is gone). + +**Test WebAuthn flows.** Mock the authenticator response objects as dictionaries conforming to the WebAuthn spec. Call `passkey_manager.generate_registration_options` with a test user ID and username. Verify the returned options include the correct RP ID, RP name, and challenge. Call `passkey_manager.verify_registration` with the mock response and the challenge. Verify the returned `VerifiedRegistration` has the correct credential ID and sign count. + +For authentication, call `passkey_manager.generate_authentication_options`. Feed the challenge into a mock authentication response. Call `passkey_manager.verify_authentication` with the stored credential public key and a current sign count. Verify the returned `VerifiedAuthentication` has an incremented sign count. + +For clone detection, call `verify_authentication` with `credential_current_sign_count=5` and a mock response where `new_sign_count=3`. Assert it raises `ValueError("Signature counter anomaly detected")`. + +### Integration Tests + +**Full message flow.** Register two users with WebAuthn mocks. Initialize encryption keys for both via `prekey_service.initialize_user_keys`. Fetch user B's prekey bundle. Initialize a conversation from A to B via `message_service.initialize_conversation`. Send an encrypted message from A. On user B's side, load the ratchet state (performing receiver-side X3DH), and decrypt the message. Verify the plaintext matches. This exercises the entire pipeline: key generation, prekey bundle assembly, X3DH, Double Ratchet initialization, encryption, SurrealDB storage, and decryption. + +**WebSocket lifecycle.** Start the FastAPI test server. Connect two test WebSocket clients (simulating Alice and Bob). Send a typed message from Alice via `ws.send_json({"type": "encrypted_message", ...})`. Assert Bob's WebSocket receives the forwarded message. Send a typing indicator from Alice. Assert Bob receives it. Disconnect Alice. Assert Bob receives a presence update (or verify via the presence service that Alice's status is now offline). Reconnect Alice. Verify the heartbeat starts and presence is restored. + +**Database rollback on failure.** Create an async session. Start an operation that will fail (e.g., insert a `RatchetState` with a `user_id` that violates a foreign key constraint). Assert that the session rolls back cleanly and raises `DatabaseError`. Verify that no partial data was committed by querying the table afterward. + +**Key rotation.** Initialize keys for a user. Verify they have an active signed prekey. Call `prekey_service.rotate_signed_prekey`. Verify the old SPK is now inactive. Verify the new SPK is active. Fetch the prekey bundle. Verify the bundle contains the new SPK, not the old one. Verify the signature on the new SPK validates against the user's Ed25519 identity key. + +**OPK exhaustion.** Initialize a user with 2 OPKs. Fetch the prekey bundle twice (consuming both OPKs). Fetch a third time. Verify the bundle is returned with `one_time_prekey = None`. Verify X3DH still succeeds without the OPK (the 3-DH fallback). + +--- + +## Common Implementation Pitfalls + +### Pitfall 1: Nonce Reuse in AES-GCM + +**Symptom:** Decryption succeeds but authentication tag verification fails intermittently (or worse, decryption succeeds and you do not notice the security break). + +**Cause:** Reusing the same nonce with the same key completely breaks GCM's security. AES-GCM uses the nonce to generate a unique keystream via CTR mode. If two messages share the same nonce and key, XORing the two ciphertexts cancels out the keystream and produces the XOR of the two plaintexts. An attacker can then use frequency analysis or known-plaintext attacks to recover both messages. Additionally, the authentication tag becomes forgeable. + +**Fix:** Always use `os.urandom(12)` for a fresh nonce, as done at `double_ratchet.py:122`. Since each message key is derived from a unique chain key position, nonce reuse across different message keys is not a concern. The risk would only arise if the same message key were used twice, which the chain advancement prevents. + +### Pitfall 2: Not Verifying Signed Prekey + +**Symptom:** X3DH succeeds but communication can be intercepted by a man-in-the-middle. + +**Cause:** Skipping signature verification allows an attacker to substitute their own SPK into Bob's prekey bundle. Alice computes a shared secret with the attacker. The attacker independently computes a shared secret with Bob using Bob's real SPK. The attacker can now read and modify all messages. + +**Fix:** Always verify before exchange, as done at `x3dh_manager.py:217-220`: + +```python +if not self.verify_signed_prekey(bob_bundle.signed_prekey, + bob_bundle.signed_prekey_signature, + bob_identity_public_ed25519): + raise ValueError("Invalid signed prekey signature") +``` + +### Pitfall 3: Storing Private Keys on Server + +**Symptom:** Server compromise exposes all conversations. + +**Cause:** If private key material is accessible to the server, anyone who compromises the server (SQL injection, SSH access, backup leak) can decrypt every message ever sent. + +**Fix:** Client-side key generation and IndexedDB storage. The server only stores public keys. Look at `prekey_service.py:45-150`: the `store_client_keys` method receives only public keys from the client. The `private_key` fields on `IdentityKey` and `SignedPrekey` models are set to empty strings `""` when storing client-uploaded keys. Private keys live exclusively in the browser's IndexedDB, which is sandboxed per origin. + +### Pitfall 4: Non-Constant-Time Comparison + +**Symptom:** Timing side channel leaks key material byte by byte. + +**Cause:** Using `===` or `==` for byte array comparison in JavaScript short-circuits on the first mismatch. An attacker who can measure response times with sufficient precision (microsecond resolution is achievable over a network) can determine how many leading bytes are correct and brute-force the rest. + +**Fix:** XOR-based comparison as in `primitives.ts:388-397`: + +```typescript +export function constantTimeEqual(a: Uint8Array, b: Uint8Array): boolean { + if (a.length !== b.length) return false + let result = 0 + for (let i = 0; i < a.length; i++) { + result |= a[i] ^ b[i] + } + return result === 0 +} +``` + +--- + +## Code Organization Principles + +The codebase follows a strict layering that separates concerns and minimizes the blast radius of bugs. + +- **Encryption logic in `core/encryption/`** (never in the API layer). The `x3dh_manager.py` and `double_ratchet.py` files have zero knowledge of HTTP, WebSocket, or databases. They take bytes in and return bytes out. `X3DHManager` does not import SQLModel. `DoubleRatchet` does not import FastAPI. This makes them independently testable: you can instantiate `X3DHManager()` in a test file, call `generate_identity_keypair_x25519()`, and verify the output without setting up a database, a Redis connection, or a web server. It also makes them reusable: if you wanted to use the same X3DH implementation in a CLI tool or a different server framework, you could import `core/encryption/` without pulling in any web dependencies. + +- **Database operations in `services/`** (never in core). The `prekey_service.py`, `message_service.py`, and `auth_service.py` files handle all SQLModel queries and SurrealDB operations. They receive an `AsyncSession` as a parameter (dependency injection) and call into `core/encryption/` for crypto operations. The crypto layer never sees a session object, and the service layer never calls OpenSSL directly. This means a database schema change only affects service files and models, not the encryption logic. + +- **Pydantic schemas for ALL API boundaries.** Every WebSocket message type has a corresponding schema in `schemas/websocket.py` (e.g., `EncryptedMessageWS`, `TypingIndicatorWS`, `ErrorMessageWS`). Every HTTP request/response has a schema in `schemas/auth.py`, `schemas/rooms.py`, etc. This provides runtime validation: if a client sends a malformed payload (missing fields, wrong types, oversized strings), Pydantic rejects it before the data reaches service logic. For example, `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` in `config.py:44` caps the ciphertext field length, preventing a client from sending a 10GB ciphertext that would consume server memory. + +- **Module-level singletons for stateless managers.** At the bottom of `x3dh_manager.py:353`: `x3dh_manager = X3DHManager()`. At the bottom of `double_ratchet.py:419`: `double_ratchet = DoubleRatchet()`. At the bottom of `websocket_manager.py:296`: `connection_manager = ConnectionManager()`. These classes either hold no per-request state (X3DHManager, DoubleRatchet) or manage global connection state (ConnectionManager). A single instance per process is safe because: (a) stateless managers have no mutable state, so concurrent async tasks cannot interfere with each other; (b) the ConnectionManager's mutable state (`active_connections` dict) is accessed within a single event loop, so there are no threading concerns. + +- **Async all the way through (no blocking I/O).** Every database query uses `await session.execute(...)`. Every WebSocket operation uses `await websocket.send_json(...)`. Every Redis call uses async methods. Every SurrealDB call uses async methods. The only synchronous operations are the CPU-bound cryptographic computations (HMAC, HKDF, AES-GCM, X25519 DH exchange), which complete in microseconds and do not benefit from async. If a crypto operation ever became slow (e.g., Argon2 password hashing taking 500ms), it would need to be offloaded to a thread pool via `asyncio.to_thread()` to avoid blocking the event loop. But the operations used here are all sub-millisecond. + +- **Custom exception hierarchy in `exceptions.py`.** Every error type has its own exception class (`EncryptionError`, `DecryptionError`, `KeyExchangeError`, `DatabaseError`, `UserNotFoundError`, etc.), all inheriting from `AppException`. This allows `exception_handlers.py` to map each exception type to an appropriate HTTP status code and error response. A `UserNotFoundError` returns 404. An `EncryptionError` returns 500. A `ChallengeExpiredError` returns 401. Without the hierarchy, every error would be a generic 500, and the client would have no idea what went wrong. + +- **Frontend mirrors the backend structure.** The frontend's `crypto/` directory parallels the backend's `core/encryption/`. `primitives.ts` provides the same low-level operations as Python's `cryptography` library. `x3dh.ts` parallels `x3dh_manager.py`. `double-ratchet.ts` parallels `double_ratchet.py`. `crypto-service.ts` parallels `message_service.py` (the orchestration layer). This symmetry makes it easier to trace a bug: if decryption fails, you compare the frontend and backend implementations step by step, checking that the same constants, the same byte orderings, and the same HKDF parameters are used on both sides. + +--- + +## Dependencies and Why + +### Backend + +- **cryptography:** The Python Cryptographic Authority's library. It wraps OpenSSL for performance-critical operations (AES-GCM, HKDF) and provides native implementations for curve operations (X25519, Ed25519). The library is audited, has a dedicated security response team, and follows a responsible disclosure process. It is the standard choice for Python crypto when you need low-level primitives. Alternatives like PyCryptodome exist but have a smaller maintainer base, less rigorous audit history, and a different API style (PyCryptodome requires more manual buffer management). The `cryptography` library also has the advantage of separating "hazmat" (hazardous materials) primitives from high-level recipes, which makes it explicit when you are doing something that requires careful handling. + +- **py_webauthn:** Implements the WebAuthn/FIDO2 server-side protocol. Handles the complex ASN.1 parsing of attestation objects, CBOR decoding of authenticator data, signature verification across multiple algorithm families (ES256, RS256, EdDSA), and challenge lifecycle management. The WebAuthn specification is over 200 pages. Writing a compliant implementation from scratch would take months and invite subtle bugs in the CBOR parsing alone. py_webauthn also provides convenient helpers like `bytes_to_base64url` and `base64url_to_bytes` that we use throughout the encryption code. + +- **sqlmodel:** Combines SQLAlchemy's query builder with Pydantic's validation. You define a model once (like `RatchetState` or `IdentityKey`) and get both a database table schema and a Pydantic serialization schema. The `AsyncSession` support via `sqlmodel.ext.asyncio.session` integrates cleanly with FastAPI's async request handlers. The alternative would be using raw SQLAlchemy models plus separate Pydantic schemas, which doubles the model definitions and introduces synchronization risk (changing a column in the SQLAlchemy model but forgetting to update the Pydantic schema). + +- **surrealdb:** The Python client for SurrealDB's WebSocket protocol. The key feature is live query support: `LIVE SELECT * FROM messages WHERE recipient_id = $user_id` creates a persistent subscription that pushes new matching records to a callback. This eliminates the need for a separate pub/sub system (like Redis Pub/Sub or RabbitMQ) for real-time message delivery. SurrealDB serves as both the message store and the real-time event bus. + +- **orjson:** A Rust-backed JSON serializer that is 3-10x faster than the standard `json` module. When the server is serializing hundreds of WebSocket messages per second (each containing base64-encoded ciphertext and headers), the difference is measurable in CPU utilization. orjson also handles `datetime`, `UUID`, and `bytes` natively without custom serializers, reducing boilerplate code. It produces the same JSON output as the standard library, so it is a drop-in replacement. + +### Frontend + +- **WebCrypto API (built-in):** Browser-native cryptographic primitives. No npm dependencies for crypto means zero supply chain risk for the most security-critical code in the application. A malicious update to an npm crypto package could steal private keys or weaken encryption. With WebCrypto, the crypto implementation is part of the browser itself, updated through the browser's own security process. `subtle.generateKey`, `subtle.deriveBits`, `subtle.encrypt`, `subtle.decrypt`, `subtle.sign`, and `subtle.verify` cover every operation needed for X25519, Ed25519, AES-GCM, HKDF, and HMAC. The implementations run in native C/C++/Rust code, which is both faster and more resistant to timing attacks than JavaScript implementations. WebCrypto also enforces key usage restrictions at the API level (a key imported for "encrypt" cannot be used for "decrypt"), providing defense in depth. + +- **nanostores:** A minimal state management library under 1KB gzipped. For a chat application, the state management needs are straightforward: "who is online," "what messages are in this room," "is the user typing," "which room is selected." Nanostores provides reactive `atom` (single value) and `computed` (derived value) stores that integrate with SolidJS's reactive system. The `$connectionStatus`, `$isConnected`, and `$reconnectAttempts` atoms in `websocket-manager.ts` demonstrate the pattern. Unlike Redux (which requires actions, reducers, and middleware for async operations), nanostores is just get/set/subscribe. The simplicity is the point: fewer abstractions means fewer places for bugs to hide. + +- **@tanstack/solid-query:** Manages server state (data fetched from REST APIs) separately from client state (UI interactions, WebSocket events). Handles caching (do not re-fetch the room list if it was fetched 30 seconds ago), background refetching (update the room list when the tab regains focus), stale-while-revalidate (show the cached data immediately, fetch fresh data in the background), and error/loading/success state transitions. Without it, every component that needs server data would manually manage `isLoading`, `error`, and `data` state, implement its own cache invalidation logic, and handle race conditions between concurrent fetches. TanStack Query absorbs all that complexity into a declarative `createQuery` call. diff --git a/PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md b/PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md new file mode 100644 index 00000000..bfb42821 --- /dev/null +++ b/PROJECTS/advanced/encrypted-p2p-chat/learn/04-CHALLENGES.md @@ -0,0 +1,1214 @@ +# Extension Challenges + +These challenges extend the encrypted P2P chat with real features used in production messaging systems. They are ordered by difficulty and build on each other where noted. + +Each challenge references actual files in this project. The line numbers point you to the exact code you need to understand before you start. Read those lines first. + +--- + +## Easy Challenges + +--- + +### Challenge 1: Read Receipts + +**What to build:** When Bob reads a message from Alice, send an encrypted read receipt back to Alice so she sees "Read" status on her message. + +**Why it matters:** Every modern messaging app has read receipts. Signal, WhatsApp, and iMessage all implement this. The interesting part is that read receipts themselves should be encrypted, because knowing WHEN someone read a message is metadata worth protecting. An adversary who can observe receipt timing can infer conversation patterns, responsiveness, and even sleep schedules. + +**What you will learn:** +- WebSocket bidirectional messaging patterns +- Extending the existing message type routing +- UI state management for message delivery status + +**Where to start reading:** + +The backend already has the scaffolding for this. Look at these files: + +- `backend/app/config.py:54` defines `WS_MESSAGE_TYPE_RECEIPT = "receipt"` already +- `backend/app/services/websocket_service.py:69-70` already routes receipt messages to `handle_read_receipt` +- `backend/app/services/websocket_service.py:272-314` has a working `handle_read_receipt` implementation +- `backend/app/schemas/websocket.py:62-69` defines `ReadReceiptWS` with `message_id`, `user_id`, and `read_at` + +So the backend is done. The work is on the frontend. + +**Implementation approach:** + +1. When Bob's client decrypts and displays a message from Alice, emit a receipt through the WebSocket: + ```json + { + "type": "receipt", + "message_id": "messages:abc123", + "sender_id": "alice-uuid-here" + } + ``` + +2. Alice's client receives the `ReadReceiptWS` via the WebSocket message handler at `frontend/src/websocket/message-handlers.ts`. Add a handler for the `"receipt"` message type. + +3. Update the message store (`frontend/src/stores/messages.store.ts`) to track message status. Add a `status` field with values: `"sending"`, `"sent"`, `"delivered"`, `"read"`. + +4. In the `MessageBubble.tsx` component (`frontend/src/components/Chat/MessageBubble.tsx`), render the status below the message text for outgoing messages. + +**Design decisions you need to make:** +- Should receipts be encrypted? The current implementation sends them in plaintext over the WebSocket. A production system would encrypt them through the Double Ratchet, but that means each receipt advances the ratchet state. That is a meaningful tradeoff. +- Should you batch receipts? If Bob scrolls through 50 unread messages, sending 50 individual receipts is wasteful. Consider sending one receipt for the latest message ID (implying all prior messages are also read). +- What happens if Bob reads a message while offline (loaded from IndexedDB)? The receipt needs to be queued and sent when the WebSocket reconnects. + +**How to test:** +- Open two browser tabs as different users +- Send a message from Tab A to Tab B +- Verify Tab A shows "Sent" initially +- Switch to Tab B and open the conversation, verify Tab A updates to "Read" +- Check the browser DevTools Network tab (WS frames) to confirm the receipt travels over WebSocket, not a separate HTTP request +- Close Tab B, send another message from Tab A, reopen Tab B, verify the receipt still works + +--- + +### Challenge 2: Typing Indicators + +**What to build:** Show "Alice is typing..." in Bob's chat window while Alice is composing a message. + +**Why it matters:** Presence indicators are standard in messaging. The challenge is implementing them efficiently. A naive implementation floods the WebSocket with keypress events. A good implementation throttles to minimize bandwidth while keeping the UI responsive. + +**What you will learn:** +- WebSocket event throttling and debouncing +- Ephemeral state management (state that never persists to any database) +- Frontend reactive updates with nanostores + +**Where to start reading:** + +This feature is substantially built already. Study what exists: + +- `backend/app/config.py:52` defines `WS_MESSAGE_TYPE_TYPING = "typing"` +- `backend/app/services/websocket_service.py:65-66` routes typing messages to `handle_typing_indicator` +- `backend/app/services/websocket_service.py:181-224` broadcasts typing events to the room via `connection_manager.broadcast_to_room` +- `backend/app/schemas/websocket.py:42-49` defines `TypingIndicatorWS` with `user_id`, `room_id`, `is_typing` +- `frontend/src/stores/typing.store.ts:1-113` is a complete typing store with auto-clear timeouts (5 seconds, line 10) +- `frontend/src/components/Chat/TypingIndicator.tsx` already renders the typing indicator UI + +The backend and most of the frontend state management are done. What is missing: + +**Implementation approach:** + +1. In the `ChatInput.tsx` component (`frontend/src/components/Chat/ChatInput.tsx`), add an `onInput` handler that sends a typing event through the WebSocket. + +2. Throttle the typing events. You do not want to send one per keystroke. Implement a 3-second throttle window: send one "is_typing: true" event, then suppress all further events for 3 seconds. When the user stops typing (no keystrokes for 3 seconds), send "is_typing: false". + +3. In the WebSocket message handler (`frontend/src/websocket/message-handlers.ts`), handle incoming `"typing"` messages by calling `setUserTyping` from the typing store. + +4. Wire the `TypingIndicator.tsx` component into the chat view if it is not already connected. + +**Throttling pattern:** +``` +Keystroke at t=0ms -> send "typing: true" +Keystroke at t=500ms -> suppress (within 3s window) +Keystroke at t=1.2s -> suppress +Keystroke at t=4.5s -> send "typing: true" (new window) +No input for 3s -> send "typing: false" +``` + +**Do not encrypt typing events.** They are ephemeral metadata. Running them through the Double Ratchet would advance the ratchet state for throwaway data, wasting chain keys and increasing the risk of desynchronization. + +**How to test:** +- Open two tabs as different users in the same room +- Start typing in Tab A +- Verify "typing..." appears in Tab B within 1 second +- Stop typing and verify the indicator disappears after 5 seconds (the `TYPING_TIMEOUT_MS` constant at `typing.store.ts:10`) +- Type rapidly and verify only one WebSocket event per 3-second window (check WS frames in DevTools) +- Switch rooms and verify typing state resets + +--- + +### Challenge 3: Message Timestamps and Ordering + +**What to build:** Display human-readable timestamps on messages ("2:34 PM", "Yesterday", "Jan 15") and ensure messages display in correct chronological order even when they arrive out of order over the WebSocket. + +**Why it matters:** The Double Ratchet protocol handles cryptographic out-of-order delivery (see `double_ratchet.py:215-244` for the skipped message key logic), but the UI also needs to handle display ordering correctly. This is a common source of bugs in messaging apps. If Alice sends message A then message B, but B arrives at Bob first over the WebSocket, the UI must still display A before B. + +**What you will learn:** +- Client-side message sorting with stable ordering +- Date formatting and localization using the `Intl` APIs +- Handling clock skew between devices + +**Where to start reading:** + +- `backend/app/schemas/websocket.py:26-39` shows `EncryptedMessageWS` includes a `timestamp` field +- `frontend/src/crypto/message-store.ts:95-119` already sorts messages by `created_at` (line 106-107) +- `frontend/src/lib/date.ts` is the existing date utility file +- `frontend/src/components/Chat/MessageBubble.tsx` renders individual messages + +**Implementation approach:** + +1. The message store at `message-store.ts:106-107` already sorts by `created_at`. Verify this works when messages arrive out of order by checking that new messages insert at the correct position, not just appended at the end. + +2. Create a timestamp formatter. Use `Intl.RelativeTimeFormat` for recent messages and `Intl.DateTimeFormat` for older ones: + - Under 1 minute: "Just now" + - Under 1 hour: "12 min ago" + - Same day: "2:34 PM" + - Yesterday: "Yesterday at 2:34 PM" + - Same year: "Jan 15 at 2:34 PM" + - Older: "Jan 15, 2025 at 2:34 PM" + +3. Add date separator headers between messages from different days. When rendering the message list, compare each message's date with the previous message. If they differ, insert a "--- January 15, 2025 ---" separator. + +4. Handle clock skew. If Alice's clock is 5 minutes ahead, her messages will have future timestamps. Clamp displayed times to never be "in the future" relative to the recipient's clock. + +**How to test:** +- Send several messages quickly between two users and verify chronological order +- Use browser DevTools to throttle the network to "Slow 3G", send multiple messages, and verify that out-of-order arrivals still display in the correct order +- Send a message, wait 5 minutes, send another. Verify a date separator does NOT appear (same day). Wait until the next calendar day and send another. Verify the separator appears. + +--- + +### Challenge 4: User Online Status + +**What to build:** Show green/gray dots next to usernames in the conversation list and chat header, indicating online/offline status in real time. + +**Why it matters:** The project already has a complete backend presence system and a frontend presence store. This challenge connects the two through the WebSocket layer. + +**What you will learn:** +- SurrealDB presence records and their lifecycle +- Frontend reactive state from WebSocket events +- UI indicators with real-time updates + +**Where to start reading:** + +The backend infrastructure is already built: + +- `backend/app/core/websocket_manager.py:74` calls `presence_service.set_user_online(user_id)` on WebSocket connect +- `backend/app/core/websocket_manager.py:108` calls `presence_service.set_user_offline(user_id)` on disconnect (only when the last connection for that user closes) +- `backend/app/services/presence_service.py:22-35` sets online status in SurrealDB +- `backend/app/services/presence_service.py:37-50` sets offline status +- `backend/app/services/websocket_service.py:226-270` handles presence update messages + +The frontend store is also ready: + +- `frontend/src/stores/presence.store.ts:1-72` has `$presenceByUser` map, `setUserPresence`, `getUserStatus`, `isUserOnline` +- `frontend/src/components/Chat/OnlineStatus.tsx` already renders the status indicator + +**Implementation approach:** + +1. When a user connects via WebSocket, the backend already updates SurrealDB presence. What is missing is broadcasting that change to the user's contacts. In `websocket_manager.py`, after `presence_service.set_user_online(user_id)` succeeds (line 74), broadcast a presence update to all rooms that user belongs to. + +2. Add a `PresenceUpdateWS` handler in the frontend WebSocket message handler. When a `"presence"` message arrives, call `setUserPresence` from the presence store. + +3. In `ConversationList.tsx` (`frontend/src/components/Chat/ConversationList.tsx`) and `ChatHeader.tsx` (`frontend/src/components/Chat/ChatHeader.tsx`), read from `$presenceByUser` to display the correct online/offline indicator. + +4. On initial WebSocket connection, request the current presence of all the user's contacts. This could be a new HTTP endpoint (`GET /users/presence?user_ids=...`) or a special WebSocket message that returns a bulk presence snapshot. + +**How to test:** +- Open Tab A as User1, verify User1 shows as online in the conversation list +- Open Tab B as User2, verify both show as online +- Close Tab B, verify User2 shows as offline in Tab A within a few seconds +- Reopen Tab B, verify User2 shows as online again + +--- + +## Intermediate Challenges + +--- + +### Challenge 5: Message Search (Encrypted) + +**What to build:** Let users search their own message history by keyword. Since messages are E2E encrypted, the server cannot search. All searching must happen client-side after decryption. + +**Why it matters:** This is a real engineering problem that Signal, WhatsApp, and iMessage each solve differently. Server-side search is impossible with E2E encryption, which is the entire point of E2E encryption. Client-side search requires careful design to balance security, performance, and UX. + +Signal allows searching decrypted messages on-device. WhatsApp backs up messages to Google Drive/iCloud (optionally encrypted) for searchability. iMessage uses a local Spotlight index. Each approach has different security properties. + +**What you will learn:** +- Client-side full-text search in encrypted applications +- IndexedDB as a search-capable local database +- Security tradeoffs between searchability and forward secrecy + +**Where to start reading:** + +- `frontend/src/crypto/message-store.ts:1-211` is the existing IndexedDB message store with `decrypted_messages` object store +- `frontend/src/crypto/message-store.ts:34-39` creates indexes on `room_id`, `created_at`, and a compound `room_created` index +- `frontend/src/crypto/message-store.ts:95-119` shows how messages are queried by room +- `frontend/src/crypto/key-store.ts:392-412` shows `clearAllKeys()` which wipes all crypto state on logout + +**Implementation approach:** + +1. Decrypted messages are already stored in IndexedDB at `message-store.ts:61-67` (`saveDecryptedMessage`). The plaintext is already persisted locally. You do not need to add a separate plaintext store. + +2. Add a `searchMessages` function to `message-store.ts`. IndexedDB does not support full-text search natively. You have two options: + + Option A (simple): Load all messages for the user, filter with `String.prototype.includes()` or a regex. This works for small message histories (under 10,000 messages). + + Option B (performant): Build a simple inverted index in a separate IndexedDB object store. On each message save, tokenize the plaintext, store `{token: string, message_id: string}` entries. Search queries look up tokens in the index. This scales to hundreds of thousands of messages. + +3. Build a search UI. Add a search input to the sidebar. Results should show message previews with the matching text highlighted. Clicking a result should navigate to that message in the conversation and scroll it into view. + +4. Clear the search index on logout. Add this to whatever logout function already calls `clearAllKeys()` and `clearAllMessages()`. + +**Security considerations:** + +The local plaintext store is a security tradeoff. If the device is compromised, the search history exposes message content. Signal handles this by only searching messages from the current app session (not backed by a persistent index). Consider offering a setting: "Enable message search (stores decrypted messages locally)" vs. "Maximum security (no local message storage)". + +Also consider: IndexedDB data survives browser cache clears in some browsers. Use `indexedDB.deleteDatabase()` on logout, not just clearing object stores. + +**How to test:** +- Send several messages containing known keywords between two users +- Search for a keyword and verify matching messages appear in results +- Search for a keyword that appears in multiple rooms and verify results are grouped by room +- Log out and verify the search index is cleared (check IndexedDB in DevTools Application tab) +- Search with no results and verify an appropriate empty state renders + +--- + +### Challenge 6: File Sharing (Encrypted) + +**What to build:** Allow users to send encrypted files (images, documents, audio) through the chat. Files must be encrypted client-side with the same E2E encryption guarantees as text messages. + +**Why it matters:** File sharing is a core feature of any messaging app. The challenge is encrypting large binary data efficiently while maintaining the same security properties as text messages. A 50MB video cannot be encrypted the same way as a 200-byte text message. You need chunked encryption, progress indicators, and a separate upload path. + +**What you will learn:** +- Encrypting large binary data with AES-256-GCM +- Chunked encryption for memory efficiency +- File upload APIs with binary data handling +- Thumbnail generation for image previews +- MIME type handling and content-type security + +**Where to start reading:** + +- `frontend/src/crypto/primitives.ts:224-259` is the `aesGcmEncrypt` function. It accepts `Uint8Array` for plaintext, which means binary data works directly with no conversion needed. +- `frontend/src/crypto/primitives.ts:261-292` is `aesGcmDecrypt` +- `backend/app/services/message_service.py:269-314` is `store_encrypted_message` which stores to SurrealDB +- `backend/app/config.py:44` defines `ENCRYPTED_CONTENT_MAX_LENGTH = 50000` (this limits inline message content, not file uploads) + +**Implementation approach:** + +1. **Client: Read and encrypt the file** + - Use `FileReader.readAsArrayBuffer()` to get the raw bytes + - For small files (under 1MB): encrypt the entire file as one AES-256-GCM operation using a message key from the Double Ratchet + - For large files (over 1MB): chunk into 1MB segments. Encrypt each chunk with a derived key: `chunk_key = HKDF(message_key, salt=chunk_index)` + - Each chunk gets its own nonce (never reuse nonces) + +2. **Client: Upload encrypted blob** + - Create a new backend endpoint: `POST /api/messages/file` + - Send the encrypted blob as multipart form data + - Include metadata in the request: original filename, MIME type, file size, number of chunks, encryption header (same format as text message headers) + +3. **Backend: Store encrypted file** + - Store the encrypted blob on disk or in object storage (not in SurrealDB, which is not designed for large binary data) + - Store file metadata in SurrealDB: `file_id`, `sender_id`, `recipient_id`, `room_id`, `filename`, `mime_type`, `size`, `chunk_count`, `storage_path` + - Return a `file_id` to the client + +4. **Client: Send file message** + - Send a regular encrypted message through the WebSocket, but the plaintext content is JSON metadata: + ```json + { + "type": "file", + "file_id": "abc123", + "filename": "document.pdf", + "mime_type": "application/pdf", + "size": 1048576, + "thumbnail": "" + } + ``` + - The recipient decrypts this metadata, then downloads the encrypted file by `file_id`, then decrypts the file client-side + +5. **For images: generate encrypted thumbnails** + - Before uploading, use `` to resize the image to a thumbnail (200x200) + - Encrypt the thumbnail separately + - Include the encrypted thumbnail inline in the message (it is small enough) + - Render the thumbnail immediately, load the full image on click + +**Security considerations:** +- Validate file size limits on the backend to prevent DoS (suggest 100MB max) +- The server validates file size and MIME type headers, but cannot verify encrypted content matches the claimed MIME type. The client must validate after decryption. +- Consider using separate encryption keys for files vs. text messages. If a file key is compromised, it should not compromise the text message ratchet. +- Strip EXIF data from images before encryption (EXIF contains GPS coordinates, camera model, timestamps) + +**Chunked encryption detail:** + +For files over 1MB, you need a chunking strategy. Here is a concrete approach: + +1. Split the file into 1MB (1,048,576 byte) chunks +2. For each chunk, derive a per-chunk key: `chunk_key_i = HKDF(message_key, salt=uint32_to_bytes(i), info=b'file_chunk')` +3. Encrypt each chunk: `encrypted_chunk_i = AES-GCM(chunk_key_i, chunk_i, associated_data=file_id || chunk_index || total_chunks)` +4. The associated data binding prevents an attacker from reordering, duplicating, or truncating chunks +5. Upload chunks sequentially or in parallel (the server stores them as `{file_id}_chunk_{i}`) + +On the receiving end: +1. Download all chunks +2. Derive the same per-chunk keys from the message key +3. Decrypt each chunk, verify the associated data +4. Concatenate decrypted chunks to reconstruct the original file + +**Progress tracking:** + +For large files, display upload/download progress: +- Track bytes uploaded/downloaded vs. total file size +- Show a progress bar in the `MessageBubble` component +- Allow cancellation mid-transfer (clean up partial uploads on the server) + +**How to test:** +- Send a small text file (.txt, under 1KB) between two users, verify the recipient can download and decrypt it +- Send an image (.png, under 5MB), verify a thumbnail appears immediately and the full image loads on click +- Send a large file (over 10MB), verify chunked encryption works and a progress bar appears +- Verify the server filesystem contains only encrypted blobs (open a stored file in a hex editor, confirm it looks like random data) +- Send a file to an offline user, verify they can download and decrypt it when they come online +- Cancel a large file upload mid-transfer, verify partial data is cleaned up +- Send a file with a tampered chunk (flip one byte), verify decryption fails with an authentication error + +--- + +### Challenge 7: Multi-Device Sync + +**What to build:** Allow a user to be logged in on multiple devices simultaneously, with messages synced across all devices while maintaining E2E encryption. + +**Why it matters:** This is one of the hardest problems in E2E encrypted messaging. Signal uses the "linked devices" approach where each device has its own identity key and its own ratchet session with every contact. WhatsApp uses a multi-device architecture where the phone is the primary device and other devices get proxy sessions. iMessage gives each device its own key pair and encrypts each message N times (once per recipient device). + +Each approach has different properties for security, UX, and complexity. This challenge uses Signal's approach because it is the most secure and the codebase already supports multiple WebSocket connections per user. + +**What you will learn:** +- Multi-device identity key management +- Session multiplication (N devices means N separate ratchet sessions per contact) +- Device linking and verification +- Conflict resolution in distributed cryptographic state + +**Where to start reading:** + +- `backend/app/core/websocket_manager.py:39` stores `active_connections: dict[UUID, list[WebSocket]]` which already supports multiple connections per user +- `backend/app/core/websocket_manager.py:52` enforces `WS_MAX_CONNECTIONS_PER_USER` (default 5, see `config.py:141`) +- `backend/app/core/websocket_manager.py:145-153` iterates over ALL connections for a user when sending a message +- `backend/app/services/prekey_service.py:152-219` initializes user keys with identity key, signed prekey, and one-time prekeys +- `backend/app/models/IdentityKey.py` stores one identity key per user (this needs to change) +- `backend/app/services/message_service.py:48-166` `initialize_conversation` creates one ratchet per user pair (this also needs to change) + +**Architecture change:** + +Currently, the relationship is: +``` +Alice ----[one ratchet session]----> Bob +``` + +With multi-device, it becomes: +``` +Alice-Phone ----[ratchet A1-B1]----> Bob-Phone +Alice-Phone ----[ratchet A1-B2]----> Bob-Laptop +Alice-Laptop ----[ratchet A2-B1]----> Bob-Phone +Alice-Laptop ----[ratchet A2-B2]----> Bob-Laptop +``` + +If Alice has 2 devices and Bob has 3 devices, there are 2 x 3 = 6 ratchet sessions between them. When Alice sends a message to Bob, she encrypts it 3 times (once per Bob device). Each of Bob's devices decrypts independently. + +**Implementation phases:** + +**Phase 1: Database schema changes** +- Add a `device_id` column to the `IdentityKey` model. Change the unique constraint from `(user_id)` to `(user_id, device_id)`. +- Add a `device_id` column to `RatchetState`. Change the unique constraint to `(user_id, peer_user_id, peer_device_id)`. +- Add a `devices` table: `device_id`, `user_id`, `device_name`, `created_at`, `last_active`. + +**Phase 2: Device registration** +- When a user registers a new device (via WebAuthn), generate a new identity key pair for that device. +- Each device uploads its own prekey bundle (identity key, signed prekey, one-time prekeys). +- The prekey bundle endpoint (`prekey_service.py:293-361`) needs a `device_id` parameter. + +**Phase 3: Message fan-out** +- When Alice sends to Bob, the backend looks up all of Bob's devices. +- For each device, it checks if a ratchet session exists. If not, it performs X3DH with that device's prekey bundle. +- Alice's client encrypts the message once per recipient device. +- The WebSocket forward logic at `websocket_service.py:87-179` sends the appropriate ciphertext to each device. + +**Phase 4: Device linking** +- A new device should NOT just register independently. It should be "linked" by an existing device. +- Linking flow: new device displays a QR code containing its temporary public key. Existing device scans the QR code, performs a key exchange with the new device, and uploads a "device link" record to the server. +- Without linking, the server could register a rogue device and intercept messages. + +**Message delivery to multiple devices:** + +When Alice sends a message to Bob (who has 3 devices), the flow is: + +1. Alice's client fetches prekey bundles for ALL of Bob's devices: `GET /api/encryption/prekeys/{bob_id}?all_devices=true` +2. Alice performs X3DH separately with each device's prekey bundle (3 separate key exchanges) +3. Alice encrypts the message 3 times (once per ratchet session) +4. Alice sends 3 encrypted payloads to the server, each tagged with `(recipient_id, device_id)` +5. The server forwards each payload to the correct device + +This means the `store_encrypted_message` function at `message_service.py:269-314` needs a `device_id` field. The SurrealDB message schema changes from `{sender_id, recipient_id, ciphertext}` to `{sender_id, sender_device_id, recipient_id, recipient_device_id, ciphertext}`. + +**Device verification:** + +How does Alice know she is encrypting to Bob's real devices and not a rogue device injected by the server? This is the same trust problem as the single-device case, but magnified. Each device has its own "safety number" (a hash of the identity key pair). Users should verify safety numbers for each device, not just each contact. + +Consider implementing a device list that shows all registered devices with their safety numbers: +``` +Bob's Devices: + Phone (registered Jan 15) - Safety: 4a7b 2c3d ... + Laptop (registered Jan 20) - Safety: 9f1e 8d2c ... + Tablet (registered Feb 1) - Safety: 3b5a 7c4e ... +``` + +**How to test:** +- Register two devices for the same user (two different browser profiles or one regular + one incognito) +- Send a message from a third user to the multi-device user +- Verify BOTH devices receive and can decrypt the message independently +- Verify each device has independent ratchet state (advancing the ratchet on device A does not affect device B) +- Disconnect one device, send messages, reconnect, and verify the reconnected device catches up +- Register a third device and verify it establishes new ratchet sessions with all existing contacts +- Remove a device and verify it can no longer decrypt new messages + +--- + +## Advanced Challenges + +--- + +### Challenge 8: Group Encryption with Sender Keys + +**What to build:** Implement group chat encryption using the Sender Keys protocol (used by Signal for group messaging). In a group chat, each member has a "sender key" that they distribute to all other members. When sending a group message, you encrypt once with your sender key, and all members can decrypt. + +**Why it matters:** In 1:1 encryption, you encrypt each message once. In a group of N members, the naive approach encrypts each message N-1 times (once per other member). Sender Keys reduce this to O(1) encryption per message, which matters at scale. A group of 1000 members would require 999 encryptions per message without Sender Keys. + +Signal uses Sender Keys for group chats. WhatsApp adopted the same approach. The tradeoff is weaker forward secrecy compared to the Double Ratchet, because the sender key chain only ratchets forward (no DH ratchet). + +**What you will learn:** +- Sender Keys protocol design and implementation +- Group key distribution and rotation +- Member addition/removal key management +- The security tradeoffs between O(1) group encryption and O(N) pairwise encryption + +**Why this is hard:** +- Adding a new member requires distributing ALL existing sender keys to them (via 1:1 encrypted channels) +- Removing a member requires rotating ALL sender keys (the removed member knows the old keys and could derive all future chain keys from them) +- Members who miss a key rotation cannot decrypt new messages until they receive the new sender key +- Forward secrecy is weaker than 1:1. Compromising a sender key exposes ALL future messages from that sender until the next rotation. + +**Where to start reading:** + +- `backend/app/api/rooms.py:38-132` creates rooms with participants. This is where group support starts. +- `backend/app/core/encryption/double_ratchet.py:96-109` has the chain key derivation (`_kdf_ck`) which is the same pattern used in Sender Keys (single-direction chain) +- `backend/app/core/websocket_manager.py:155-175` `broadcast_to_room` sends to all room members +- `frontend/src/crypto/primitives.ts:224-259` AES-GCM encryption works for sender key messages too + +**Implementation phases:** + +**Phase 1: Research** + +Read Signal's Sender Keys specification. Understand the difference between "sender key distribution messages" (sent via 1:1 ratchets) and regular group messages (encrypted with the sender key). + +A sender key is a pair: `(chain_key, signing_key)`. The chain key advances like a single-direction ratchet (HMAC-based). The signing key authenticates the sender. + +**Phase 2: Database schema** + +Create these tables in SurrealDB: + +``` +sender_keys { + group_id: string, + user_id: string, + chain_key: bytes, + chain_iteration: int, + signing_public_key: bytes, + created_at: datetime +} + +group_members { + group_id: string, + user_id: string, + role: string, // "admin" | "member" + joined_at: datetime +} +``` + +**Phase 3: Implementation** + +1. **Group creation:** When Alice creates a group with Bob and Carol: + - Alice generates a sender key pair: `(chain_key_alice, signing_key_alice)` + - Alice sends her sender key to Bob via their existing 1:1 encrypted channel (Double Ratchet) + - Alice sends her sender key to Carol the same way + - Bob and Carol do the same (each distributes their sender key to all other members) + - Result: each member has N-1 sender keys (one per other member) + +2. **Sending a message:** Alice encrypts the group message: + - Derive message key: `message_key = HMAC-SHA256(chain_key, 0x01)` + - Advance chain: `chain_key = HMAC-SHA256(chain_key, 0x02)` + - Encrypt: `ciphertext = AES-GCM(message_key, plaintext)` + - Sign: `signature = Ed25519.sign(signing_key, ciphertext)` + - Broadcast ciphertext + signature + chain_iteration to the group + +3. **Receiving a message:** Bob decrypts: + - Look up Alice's sender key by `(group_id, sender_id)` + - If `chain_iteration > local_iteration`: advance chain key to catch up + - Derive the message key, decrypt, verify signature + +4. **Member removal:** When Carol is removed: + - Alice generates a NEW sender key pair + - Alice distributes the new sender key to Bob (but NOT Carol) + - Bob does the same + - Carol still has the old sender keys but they are no longer used + +5. **Member addition:** When Dave joins: + - All existing members distribute their current sender keys to Dave via 1:1 channels + - Dave generates and distributes his sender key to all members + +**Key rotation strategy:** + +Sender keys should rotate periodically, not just on member removal. Consider rotating after every 100 messages or every 24 hours, whichever comes first. This limits the blast radius of a compromised sender key: an attacker who obtains Alice's sender key can only decrypt messages from Alice's current chain, not past chains. + +When a sender key rotates: +1. The sender generates a new sender key pair +2. The sender distributes the new key to all group members via 1:1 encrypted channels +3. The sender includes a "key rotation" flag in the first message with the new key +4. Recipients who receive a message with an unknown sender key chain must request a key distribution message + +**Error handling for missed key distributions:** + +If Bob misses a key distribution (he was offline when Alice rotated her sender key), he cannot decrypt Alice's new messages. Handle this gracefully: +1. Bob receives a message from Alice with an unknown chain iteration +2. Bob's client sends a "request sender key" message to Alice via 1:1 channel +3. Alice re-distributes her current sender key to Bob +4. Bob can now decrypt the buffered messages + +This recovery mechanism is critical. Without it, a single missed key distribution permanently breaks the group for that member. + +**How to test:** +- Create a group with 3 users +- Send a message from each user, verify all members can decrypt +- Remove one member, send a new message, verify the removed member cannot decrypt +- Add a new member, verify they can decrypt new messages +- Verify they cannot decrypt messages sent before they joined (unless you explicitly implement history sharing) +- Take one member offline, rotate a sender key, bring them back online, verify they can recover via the key re-request mechanism +- Send 101 messages from one user and verify automatic key rotation triggers + +--- + +### Challenge 9: Post-Quantum Key Exchange (Hybrid X25519 + Kyber) + +**What to build:** Add a hybrid key exchange that combines classical X25519 with CRYSTALS-Kyber (ML-KEM), making the initial key agreement resistant to quantum computer attacks while maintaining security against classical computers. + +**Why it matters:** Quantum computers running Shor's algorithm can break all current public-key cryptography based on the discrete logarithm problem, including X25519 and Ed25519. This is not hypothetical timeline speculation. Signal added post-quantum protection (PQXDH) in September 2023. Google Chrome uses hybrid ML-KEM for TLS since 2024. NIST finalized the ML-KEM standard (FIPS 203) in August 2024. + +The threat model that motivates this is "harvest now, decrypt later." An adversary records encrypted traffic today, stores it, and decrypts it in 10 years when they have a quantum computer. Hybrid PQ protection prevents this. + +**What you will learn:** +- Post-quantum cryptography fundamentals (lattice-based key encapsulation) +- Hybrid key exchange design (classical + PQ combined) +- CRYSTALS-Kyber / ML-KEM key encapsulation mechanism +- Updating a production cryptographic protocol without breaking backwards compatibility + +**Where to start reading:** + +- `backend/app/core/encryption/x3dh_manager.py:208-281` is `perform_x3dh_sender`. This is the exact function you will modify. +- `backend/app/core/encryption/x3dh_manager.py:241-255` is where the DH shared secrets are combined: + ```python + dh1 = alice_ik_private.exchange(bob_spk_public) # line 241 + dh2 = alice_ek_private.exchange(bob_ik_public) # line 242 + dh3 = alice_ek_private.exchange(bob_spk_public) # line 243 + # dh4 = alice_ek_private.exchange(bob_opk_public) # line 251 (optional) + key_material = dh1 + dh2 + dh3 [+ dh4] + ``` +- `backend/app/core/encryption/x3dh_manager.py:257-264` is where HKDF derives the shared key: + ```python + f = b'\xff' * X25519_KEY_SIZE + hkdf = HKDF(algorithm=hashes.SHA256(), length=X25519_KEY_SIZE, + salt=b'\x00' * X25519_KEY_SIZE, info=b'X3DH') + shared_key = hkdf.derive(f + key_material) + ``` +- `backend/pyproject.toml:30` already includes `liboqs-python>=0.14.1` in dependencies + +**The hybrid approach:** + +The key insight is that you combine BOTH classical and post-quantum shared secrets. If either one is secure, the combined result is secure. This means: +- If Kyber is broken but X25519 is not, you are still safe (classical security) +- If X25519 is broken (quantum computer) but Kyber is not, you are still safe (PQ security) +- Both must be broken simultaneously to compromise the session + +``` +Classical: SK_classical = HKDF(DH1 || DH2 || DH3 [|| DH4]) +Post-quantum: SK_pq = Kyber.Decapsulate(ciphertext, secret_key) +Combined: SK = HKDF(SK_classical || SK_pq, info=b'PQXDH') +``` + +**Implementation phases:** + +**Phase 1: Research** + +Read these specifications: +- Signal's PQXDH specification: https://signal.org/docs/specifications/pqxdh/ +- NIST FIPS 203 (ML-KEM): https://csrc.nist.gov/pubs/fips/203/final + +Understand the difference between a KEM (Key Encapsulation Mechanism) and a DH exchange. In Kyber: +- Bob generates a Kyber keypair and publishes the public key +- Alice calls `Encapsulate(bob_public_key)` which returns `(shared_secret, ciphertext)` +- Alice sends `ciphertext` to Bob +- Bob calls `Decapsulate(ciphertext, bob_private_key)` which returns `shared_secret` +- Both sides now have the same `shared_secret` + +**Phase 2: Prekey bundle extension** + +1. Add a `kyber_public_key` field to the `PreKeyBundle` dataclass at `x3dh_manager.py:34-42` +2. When generating prekeys (`prekey_service.py:152-219`), also generate a Kyber keypair using `liboqs-python`: + ```python + import oqs + kem = oqs.KeyEncapsulation("Kyber768") + kyber_public = kem.generate_keypair() + kyber_private = kem.export_secret_key() + ``` +3. Store the Kyber public key in the prekey bundle, private key in the database + +**Phase 3: X3DH extension** + +Modify `perform_x3dh_sender` at `x3dh_manager.py:208-281`: + +1. After computing the classical `key_material` (line 252-255), check if the bundle includes a Kyber key +2. If yes: encapsulate against the Kyber public key to get `(pq_shared_secret, pq_ciphertext)` +3. Concatenate: `combined_material = key_material + pq_shared_secret` +4. Change the HKDF info parameter: `info=b'PQXDH'` (to distinguish from classical sessions) +5. Include `pq_ciphertext` in the `X3DHResult` so it can be sent to the recipient + +Modify `perform_x3dh_receiver` at `x3dh_manager.py:283-350`: + +1. After computing the classical key material, check if a PQ ciphertext was provided +2. If yes: decapsulate to recover `pq_shared_secret` +3. Combine and derive as above + +**Phase 4: Backwards compatibility** + +If the peer does not have a Kyber public key in their prekey bundle (they have not upgraded), fall back to classical-only X3DH. The code path at `x3dh_manager.py:246-255` already handles the optional one-time prekey with an if/else. Use the same pattern for Kyber. + +**Key sizes to be aware of:** +- X25519 public key: 32 bytes +- Kyber-768 public key: 1184 bytes +- Kyber-768 ciphertext: 1088 bytes +- Kyber-768 shared secret: 32 bytes + +The prekey bundle size increases significantly. Plan for this in your database schema and WebSocket message size limits. + +**How to test:** +- Two PQ-capable clients establish a session. Verify the shared key differs from a classical-only session with the same key material. +- A PQ-capable client talks to a classical-only client. Verify it falls back gracefully. +- Generate 1000 PQ sessions and verify all shared keys are unique. +- Benchmark: measure the time for X3DH with and without Kyber. Kyber should add under 10ms. + +--- + +### Challenge 10: Disappearing Messages with Cryptographic Enforcement + +**What to build:** Messages that automatically delete after a configurable time period, with the deletion enforced cryptographically. The decryption key is destroyed after the timer expires, making the message permanently unreadable even if the ciphertext persists. + +**Why it matters:** Signal's disappearing messages feature is one of its most popular. But naive implementations just delete the UI element while the data remains in storage. If a forensic examiner recovers the database, the "deleted" messages are still there in plaintext or with intact keys. + +This challenge implements true cryptographic deletion: the ciphertext remains in SurrealDB (you cannot guarantee physical deletion from a database), but the key needed to decrypt it is securely destroyed. Without the key, the ciphertext is indistinguishable from random noise. + +**What you will learn:** +- Cryptographic deletion (destroying keys instead of data) +- Timer-based key lifecycle management +- Secure memory wiping (harder than it sounds in JavaScript and Python) +- Distributed timer synchronization between sender and recipient + +**Where to start reading:** + +- `backend/app/core/encryption/double_ratchet.py:96-109` is `_kdf_ck` which derives message keys. Each message gets a unique key. +- `backend/app/core/encryption/double_ratchet.py:215-244` is `_store_skipped_message_keys` with eviction logic at line 232-234. This is the pattern you will extend for TTL-based eviction. +- `backend/app/core/encryption/double_ratchet.py:246-258` is `_evict_oldest_skipped_keys` which deletes keys. +- `frontend/src/crypto/key-store.ts:353-358` stores ratchet state in IndexedDB +- `frontend/src/crypto/primitives.ts:294-298` has `generateRandomBytes` using `crypto.getRandomValues` + +**Implementation approach:** + +1. **Conversation setting:** Add a per-conversation setting: `disappear_after_seconds`. Values: 0 (disabled), 30, 300, 3600, 86400, 604800. Store this in the room metadata in SurrealDB. + +2. **Message key storage with TTL:** When a message is encrypted, the message key is normally discarded after encryption. For disappearing messages, the RECIPIENT stores the message key in IndexedDB with a TTL: + ``` + { + message_id: "msg_abc123", + message_key: <32 bytes>, + created_at: "2025-01-15T14:30:00Z", + expires_at: "2025-01-15T15:30:00Z" // created_at + disappear_after_seconds + } + ``` + +3. **Timer starts on read, not on send.** The recipient's timer starts when they first decrypt the message. If the recipient is offline for a week, they still get the full timer duration after reading. + +4. **Key destruction:** When the timer expires: + - Frontend: overwrite the key material with random bytes before deleting from IndexedDB. In JavaScript: `crypto.getRandomValues(keyBuffer)` then delete. + - Backend: if you store message keys server-side (for the deprecated server-side encryption path), overwrite with `os.urandom(32)` then delete. + +5. **UI changes:** + - Show a countdown timer on disappearing messages + - After expiration, replace the message content with "[Message expired]" + - The ciphertext remains in SurrealDB but is now permanently unreadable + +**The hard parts:** + +- What happens if the recipient never reads the message? The sender's copy should still eventually disappear. Implement a "maximum lifetime" that triggers regardless of read status. +- What about screenshots? You cannot prevent them. This is a social problem, not a technical one. Signal shows a notification when a screenshot is taken (on mobile), but this is best-effort. +- JavaScript garbage collection does not guarantee memory is wiped. When you overwrite a `Uint8Array`, the old values may still exist in memory until the GC reclaims the page. True secure deletion in JavaScript is not possible. Document this limitation. +- The skipped message key cache (`double_ratchet.py:47-49`) stores keys for out-of-order messages. These must also respect the TTL. + +**How to test:** +- Enable disappearing messages (30 second timer) in a conversation +- Send a message, verify it appears normally +- Wait 30 seconds, verify the message content disappears and shows "[Message expired]" +- Check IndexedDB to verify the message key is gone +- Verify the ciphertext still exists in SurrealDB but cannot be decrypted (write a test script that attempts decryption and confirms it fails) + +--- + +## Expert Challenges + +--- + +### Challenge 11: Deniable Authentication (Triple DH) + +**Estimated time:** 2-3 weeks + +**What to build:** Implement deniable authentication so that neither Alice nor Bob can prove to a third party that the other sent a specific message. This is a privacy property of the Signal Protocol that most implementations skip or get wrong. + +**Prerequisites:** Solid understanding of the X3DH implementation at `x3dh_manager.py:56-353` + +**Why it matters:** In the current implementation, if Alice has Bob's signed messages (ciphertext + authentication tags), she could potentially show them to a third party to prove Bob said something. This is because the associated data (`x3dh_manager.py:272`) binds both identity keys to the session: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes`. + +Deniable authentication means that Alice could have forged the messages herself (because she also has the shared secret), so they are not valid cryptographic proof of Bob's authorship. This matters for journalists, activists, whistleblowers, and anyone who might be legally compelled to prove message authorship. + +**The core problem:** + +X3DH provides implicit authentication through the DH operations involving identity keys. Both DH1 (`IK_A x SPK_B`) and DH2 (`EK_A x IK_B`) involve at least one identity key. A third party who trusts Alice and knows both identity keys could verify that the session was established between those specific identities. + +In a deniable protocol, the authentication MAC can be computed by EITHER party, so neither can prove the other computed it. + +**Where to start reading:** + +- `x3dh_manager.py:272` is the associated data computation: `associated_data = alice_ik_public_bytes + bob_ik_public_bytes` +- `double_ratchet.py:323-362` is `encrypt_message` where associated data is used in AES-GCM +- `double_ratchet.py:364-416` is `decrypt_message` which verifies the authentication tag + +**Implementation approach:** + +1. Replace the associated data binding with a commitment scheme. Instead of `AD = IK_A || IK_B`, use `AD = HMAC(shared_key, IK_A || IK_B)`. Both parties can compute this MAC (they both have the shared key), so neither can prove the other did. + +2. Alternatively, implement a "triple DH" variant where you add a third DH: `DH5 = IK_A x IK_B`. This creates a shared secret that only requires Alice and Bob's identity keys. Since both parties can compute it, a transcript of messages authenticated with this secret is not proof against either party. + +3. The associated data modification at `x3dh_manager.py:272` and the corresponding line at `x3dh_manager.py:341` must change in tandem (sender and receiver must produce the same associated data). + +**How to verify deniability:** + +Write a program that: +1. Takes a message transcript (ciphertexts + authentication tags) +2. Takes ONLY Alice's key material (not Bob's private keys) +3. Attempts to forge a valid transcript that looks identical +4. If it succeeds, the protocol is deniable (Alice could have created the transcript alone) + +--- + +### Challenge 12: Secure Backup and Recovery + +**Estimated time:** 3-4 weeks + +**What to build:** Allow users to create encrypted backups of their message history and encryption keys, stored server-side but encrypted with a user-controlled backup key. This enables account recovery on a new device without losing message history. + +**Prerequisites:** Complete Challenge 7 (Multi-Device Sync). You need to understand device key management before adding key backup. + +**Why it matters:** The biggest UX problem with E2E encryption is that losing your device means losing your message history forever. This is mathematically inevitable if key material only exists on one device. + +WhatsApp solved this with Google Drive/iCloud backups encrypted with a user password (using Argon2id). Signal solved it with PIN-based Secure Value Recovery (SVR), which uses SGX enclaves to rate-limit PIN guessing. Both approaches have tradeoffs. + +**Planning questions you must answer before writing code:** + +1. What key material needs to be backed up? + - Identity keys (X25519 + Ed25519 private keys) + - Active ratchet states for each conversation + - Signed prekey private keys + - Message history (optional, large) + +2. How is the backup encrypted? + - Option A: User PIN/password -> Argon2id -> AES-256-GCM key + - Option B: Random backup key displayed as a 24-word recovery phrase + - Option C: Server-side HSM with rate limiting (Signal's SVR approach, requires hardware) + +3. Where is the backup stored? + - Server-side (encrypted blob in PostgreSQL or object storage) + - User-exported file (like Signal's plaintext export, but encrypted) + +4. How do you handle backup key loss? + - Unrecoverable by design (most secure) + - Social recovery (N-of-M trusted contacts must approve, like Shamir's Secret Sharing) + - Server-side recovery with identity verification (weakest, requires trusting the server) + +**Where to start reading:** + +- `frontend/src/crypto/key-store.ts:85-99` saves identity keys to IndexedDB. This is the data that needs backing up. +- `frontend/src/crypto/key-store.ts:353-358` saves ratchet states +- `frontend/src/crypto/key-store.ts:392-412` `clearAllKeys()` deletes everything on logout +- `frontend/src/crypto/message-store.ts:61-67` saves decrypted messages + +**Implementation phases:** + +**Phase 1: Backup format design** + +Define a JSON schema for the backup: +```json +{ + "version": 1, + "created_at": "2025-01-15T14:30:00Z", + "identity_keys": { + "x25519_private": "", + "x25519_public": "", + "ed25519_private": "", + "ed25519_public": "" + }, + "ratchet_states": [ + { + "peer_id": "bob-uuid", + "root_key": "", + "sending_chain_key": "", + "receiving_chain_key": "", + "dh_private_key": "", + "dh_peer_public_key": "", + "sending_message_number": 42, + "receiving_message_number": 37, + "previous_sending_chain_length": 12 + } + ], + "messages": [] +} +``` + +**Phase 2: Backup encryption** + +1. User enters a PIN or passphrase +2. Derive key: `backup_key = Argon2id(passphrase, salt, time_cost=3, memory_cost=65536, parallelism=4)` +3. Encrypt: `encrypted_backup = AES-256-GCM(backup_key, JSON.stringify(backup_data))` +4. Store `{salt, encrypted_backup, nonce}` on the server + +**Phase 3: Backup upload/download endpoints** + +- `POST /api/backup` - upload encrypted backup blob +- `GET /api/backup` - download encrypted backup blob +- `DELETE /api/backup` - delete backup + +The server NEVER has the backup key. It stores only encrypted bytes. + +**Phase 4: Restore flow** + +1. User logs in on new device with WebAuthn +2. Server returns the encrypted backup blob +3. User enters PIN/passphrase +4. Client derives backup key, decrypts backup +5. Client writes identity keys and ratchet states to IndexedDB +6. Client resumes existing conversations using the restored ratchet states + +**Phase 5: Edge case handling** + +What happens when the restored device and the original device are both active? Their ratchet states diverge immediately (the first message from either device advances the ratchet differently on each). This is the same problem as Challenge 7 (Multi-Device). The solution is: on restore, the new device uses the backup as a starting point but establishes NEW ratchet sessions with all contacts (essentially re-keying every conversation). + +**Success criteria:** +- User can create encrypted backup from the UI +- Backup can be restored on a new device (different browser profile) +- Restored device can send and receive messages in existing conversations +- Backup blob is useless without the passphrase (verify by attempting decryption with wrong passphrase) +- Server admin cannot read backup contents (verify by checking database contents) + +--- + +## Mix and Match Projects + +These combine multiple challenges into a single coherent application. + +### Secure Team Messenger + +**Combine:** Group Encryption (8) + File Sharing (6) + Read Receipts (1) + Multi-Device (7) + +Build a team messaging app similar to Signal's group chat with file attachments and delivery status across multiple devices. This covers the feature set of a basic Slack competitor with E2E encryption. + +### Whistleblower Platform + +**Combine:** Disappearing Messages (10) + Deniable Authentication (11) + File Sharing (6) + +Build a platform where sources can securely share documents with journalists. Messages self-destruct after reading, cannot be attributed to either party, and files are encrypted in transit and at rest. Think SecureDrop but with real-time chat. + +### Enterprise Secure Chat + +**Combine:** All Easy Challenges + Group Encryption (8) + Backup/Recovery (12) + Multi-Device (7) + +Build a corporate messaging system with compliance features (backup and recovery for legal holds) while maintaining E2E encryption for message content. The tension between compliance (the company needs to audit messages) and privacy (E2E encryption prevents this) is a real product design challenge. Research how Wickr Enterprise and Element (Matrix) handle this. + +--- + +## Performance Challenges + +--- + +### Optimize Ratchet State Storage + +The current implementation serializes the entire ratchet state to PostgreSQL on every single message. Look at `message_service.py:208-267` (`_save_ratchet_state_to_db`). Every call to `send_encrypted_message` or `decrypt_received_message` triggers a full ratchet state write with `await session.commit()` at line 258. + +This is a bottleneck. For a conversation with rapid back-and-forth messaging, you are writing to PostgreSQL twice per message (once for encrypt, once for decrypt). + +**Your task:** Profile the serialization overhead and implement a caching layer using Redis. + +**Approach:** +1. On first ratchet state load, cache it in Redis with a key like `ratchet:{user_id}:{peer_user_id}` +2. After each ratchet advance, write to Redis only (sub-millisecond) +3. Periodically (every 10 messages or every 30 seconds) flush to PostgreSQL (durable storage) +4. On WebSocket disconnect (`websocket_manager.py:97-133`), flush all cached states to PostgreSQL +5. On startup, load from PostgreSQL (Redis is not durable across restarts) + +**Measurement:** Benchmark messages per second before and after with a simple load test. Target: greater than 100 messages per second per conversation. + +**Risk:** If the server crashes between Redis write and PostgreSQL flush, you lose ratchet state. This means the next message will fail to decrypt (ratchet desynchronization). Mitigate with a background flush task and short flush intervals. + +**Detailed profiling steps:** + +1. Add timing instrumentation around `_save_ratchet_state_to_db`: + ```python + import time + start = time.perf_counter() + await session.commit() + elapsed_ms = (time.perf_counter() - start) * 1000 + logger.info("Ratchet state save: %.2fms", elapsed_ms) + ``` + +2. Run a load test: two users exchanging 1000 messages as fast as possible. Measure: + - Average time per `_save_ratchet_state_to_db` call + - Average time per `_load_ratchet_state_from_db` call + - Total messages per second (end-to-end) + - PostgreSQL connection pool utilization (`DB_POOL_SIZE` at `config.py:118`) + +3. Implement Redis caching, repeat the same load test, compare. + +**Redis cache schema:** +``` +Key: ratchet:{user_id}:{peer_user_id} +Value: JSON serialized ratchet state +TTL: 300 seconds (auto-expire as safety net) +``` + +On each ratchet advance, write to Redis. Every 10 messages OR every 30 seconds, flush to PostgreSQL. On disconnect, flush immediately. + +--- + +### WebSocket Load Testing + +Use `locust` or `k6` to load test the WebSocket layer. Determine how many concurrent WebSocket connections a single server instance can handle. + +**What to measure:** +1. Maximum concurrent WebSocket connections before memory exhaustion +2. Message throughput: messages per second at 100, 500, 1000, 5000 concurrent connections +3. Latency: p50, p95, p99 message delivery latency at various connection counts +4. Identify the bottleneck: Is it CPU (from encryption operations)? Memory (from connection pool at `websocket_manager.py:39`)? I/O (from SurrealDB writes)? Python GIL contention? + +**Target:** Document the breaking point and propose a horizontal scaling strategy. + +**Scaling considerations:** + +The current architecture has a single `ConnectionManager` instance at `websocket_manager.py:296` holding all WebSocket connections in memory (`self.active_connections: dict[UUID, list[WebSocket]]`). This does not scale horizontally because connections on Server A are invisible to Server B. + +To scale to multiple server instances: +1. Use Redis Pub/Sub for cross-instance message routing. When Server A needs to send a message to a user connected to Server B, it publishes to a Redis channel. Server B subscribes and forwards to the local WebSocket. +2. Store connection metadata in Redis: `{user_id: [server_a_instance_id, server_b_instance_id]}` +3. Use sticky sessions or consistent hashing to route WebSocket upgrade requests to the same server when possible (reduces cross-instance traffic) +4. Consider using a dedicated WebSocket gateway (like Centrifugo or Soketi) in front of the FastAPI application + +**Load test script outline (k6):** +```javascript +import ws from 'k6/ws'; +export default function () { + const url = 'ws://localhost:8000/ws?user_id=...'; + const res = ws.connect(url, {}, function (socket) { + socket.on('open', () => { + socket.send(JSON.stringify({ + type: 'encrypted_message', + recipient_id: '...', + room_id: '...', + ciphertext: '...', + nonce: '...', + header: '...' + })); + }); + socket.on('message', (msg) => {}); + socket.setTimeout(() => socket.close(), 30000); + }); +} +``` + +Run with: `k6 run --vus 100 --duration 60s load_test.js` + +--- + +## Security Challenges + +--- + +### Implement Certificate Transparency for Identity Keys + +Build a system where identity key changes are logged to a verifiable append-only log, similar to Certificate Transparency for TLS certificates. This prevents the server from silently swapping identity keys to perform a man-in-the-middle attack. + +Currently, the server at `prekey_service.py:293-361` serves prekey bundles. A compromised server could serve a fake identity key (its own), perform X3DH with both parties, and relay decrypted messages. The users would not know. + +A key transparency log makes this detectable: every identity key is logged, and clients can audit the log to verify their key has not been replaced. + +**Implementation sketch:** + +1. Create a Merkle tree where each leaf is `(user_id, identity_key_hash, timestamp)`. +2. When a user registers or changes their identity key, append a new leaf to the tree. +3. Publish the Merkle root periodically (every hour) to a public log. +4. Clients download the Merkle root and verify their own key is included (Merkle inclusion proof). +5. If a client's key has been changed without their knowledge (the server swapped it), the Merkle proof will fail OR the client will see an unexpected key in the tree. + +**The hard part:** Consistency. The server must prove that the tree is append-only (no entries have been removed or modified). This requires signed tree heads and consistency proofs between consecutive tree states. + +**Research:** Google's Key Transparency project, CONIKS (Key Verification for Messaging), and Signal's key transparency deployment (announced 2023). + +--- + +### Audit the Crypto Implementation + +Perform a manual security audit of the X3DH and Double Ratchet implementations. This is not writing code. This is reading code critically. + +**Audit checklist:** + +1. **HKDF parameter usage** (`x3dh_manager.py:258-264`): Is the salt correct? Is the info string distinct between different uses? Are the output lengths appropriate? + +2. **Nonce generation** (`double_ratchet.py:122`): Is `os.urandom(AES_GCM_NONCE_SIZE)` called for every encryption? Is there any risk of nonce reuse? + +3. **WebCrypto API usage** (`primitives.ts:224-259`): Is the `iv` parameter always fresh? Are `additionalData` bindings correct? Are key usages properly restricted? + +4. **Timing side channels**: Look at `primitives.ts:388-397` (`constantTimeEqual`). Is this actually constant time in JavaScript? (Spoiler: the `if (a.length !== b.length) return false` on line 389 is an early return that leaks length information. Is this a problem in context?) + +5. **Random number generation**: Verify that all randomness comes from `os.urandom` (Python) or `crypto.getRandomValues` (JavaScript). Search for any use of `random` or `Math.random`. + +6. **Key material in memory**: Are private keys ever logged? Search for `logger.debug` and `logger.info` calls that might print key material. Check that base64url-encoded keys are not accidentally included in error messages. + +Write a report documenting your findings. + +--- + +### Implement Sealed Sender + +Build a system where the server does not know who sent a message, only who it is for. Signal implemented this as "Sealed Sender." + +Currently, the WebSocket endpoint at `websocket.py:25-84` receives messages from authenticated users. The server knows `sender_id` (from the WebSocket connection) and `recipient_id` (from the message payload). It stores both in SurrealDB (`message_service.py:292-302`). + +With Sealed Sender, the sender identity is encrypted inside the E2E encrypted payload. The server only sees the recipient (needed for routing). The sender is revealed only after the recipient decrypts. + +**Architecture:** +1. Outer layer: encrypt the routing envelope (which includes the real encrypted message) with the SERVER's public key. The server decrypts this to learn the recipient, but the sender field is absent or encrypted. +2. Inner layer: the E2E encrypted message as it works today, with sender identity inside the encrypted payload. + +This requires the server to have its own keypair (separate from any user), and clients to know the server's public key. + +**Implementation steps:** + +1. Generate a server X25519 keypair on startup. Publish the server's public key at a well-known endpoint: `GET /api/.well-known/server-key`. + +2. Modify the client message sending flow: + - Construct the inner payload: the normal E2E encrypted message (ciphertext, nonce, header) PLUS `sender_id` in plaintext + - Encrypt the inner payload with the intended recipient's ratchet (as currently done) + - Construct the outer envelope: `{recipient_id: "bob-uuid", sealed_payload: ""}` + - Encrypt the outer envelope with the server's public key (simple X25519 + AES-GCM) + - Send only the outer ciphertext over the WebSocket + +3. Modify the server routing: + - Decrypt the outer envelope using the server's private key + - Read `recipient_id` from the decrypted envelope + - Forward `sealed_payload` to the recipient + - The server never sees `sender_id` + +4. Modify the recipient decryption: + - Decrypt the sealed payload using the E2E ratchet + - Extract `sender_id` from the decrypted content + - Display the message with the correct sender attribution + +**Limitation:** The server still knows the recipient (it needs to for routing). And it knows the sender's IP address and WebSocket connection. True sender anonymity requires additional measures like Tor or mix networks. Sealed Sender hides sender identity from the server's message logs, not from network-level observation. + +**How to test:** +- Send a message between two users with Sealed Sender enabled +- Check the server's SurrealDB message records and verify `sender_id` is absent or encrypted +- Verify the recipient can still correctly identify the sender after decryption + +--- + +## Contribution Challenges + +--- + +### Write Property-Based Tests + +Use Hypothesis (Python) to write property-based tests for the cryptographic implementations. + +**Properties to verify:** + +1. **X3DH symmetry:** For any two valid identity key pairs and any valid prekey bundle, `perform_x3dh_sender` and `perform_x3dh_receiver` produce the same `shared_key`. Reference: `x3dh_manager.py:208-281` and `x3dh_manager.py:283-350`. + +2. **Double Ratchet round-trip:** For any sequence of plaintext messages in any order, encrypting with `encrypt_message` and decrypting with `decrypt_message` always returns the original plaintext. Reference: `double_ratchet.py:323-362` and `double_ratchet.py:364-416`. + +3. **AES-GCM round-trip:** For any plaintext and any valid key, `aesGcmEncrypt` followed by `aesGcmDecrypt` returns the original plaintext. Reference: `primitives.ts:224-292`. + +4. **No nonce reuse:** For N encryptions with the same key, all N nonces are distinct. (This should hold with overwhelming probability for random 12-byte nonces, but verify it empirically for N=10000.) + +5. **Forward secrecy:** After advancing the ratchet, old message keys cannot be derived from the new state. Generate a ratchet state, encrypt a message, advance the ratchet 100 steps, then verify that the message key from step 0 cannot be recomputed from the state at step 100. + +6. **Out-of-order decryption:** For any permutation of N messages, decrypting them in that permuted order produces the same set of plaintexts as decrypting in the original order. This tests the skipped message key logic at `double_ratchet.py:215-244`. + +**Hypothesis strategy example:** + +```python +from hypothesis import given, strategies as st + +@given( + message_count=st.integers(min_value=1, max_value=50), + delivery_order=st.permutations(range(50)) # random permutation +) +def test_out_of_order_delivery(message_count, delivery_order): + """All messages decrypt correctly regardless of delivery order.""" + # Generate N messages + # Encrypt them in order (advancing the sending ratchet) + # Decrypt them in the shuffled order + # Assert all plaintexts match the originals +``` + +**Where to put the tests:** Create a `tests/` directory in the backend and structure tests as `tests/test_x3dh.py`, `tests/test_double_ratchet.py`, `tests/test_crypto_properties.py`. + +--- + +### Add Formal Verification + +Use a symbolic model checker (ProVerif or Tamarin Prover) to formally verify that the X3DH + Double Ratchet implementation satisfies: + +1. **Secrecy:** An attacker who controls the network cannot learn message plaintext +2. **Authentication:** A message that decrypts successfully was sent by the claimed sender +3. **Forward secrecy:** Compromising long-term keys does not reveal past session keys +4. **Post-compromise security:** After an attacker loses access to session keys, future messages are secure again (this is the DH ratchet property) + +This is a research-level challenge. Start by modeling X3DH alone before adding the Double Ratchet. + +**Getting started with ProVerif:** + +ProVerif uses a process calculus to model cryptographic protocols. You describe the protocol as a set of processes (Alice, Bob, Attacker) and the cryptographic primitives they use. ProVerif then explores all possible interleavings and attacker strategies to find attacks. + +A minimal X3DH model in ProVerif would: +1. Define types for keys, nonces, and messages +2. Model the DH operations as `fun dh(skey, pkey): key` with the equation `dh(skA, pk(skB)) = dh(skB, pk(skA))` +3. Model HKDF as a random oracle: `fun kdf(key, key, key): key` +4. Define Alice's process: generate ephemeral key, compute DH1/DH2/DH3, derive shared key, encrypt message +5. Define Bob's process: receive ephemeral key, compute DH1/DH2/DH3, derive shared key, decrypt message +6. Query: `query attacker(message)` -- can the attacker learn the plaintext? + +**Resources:** +- ProVerif manual: https://bblanche.gitlabpages.inria.fr/proverif/ +- Tamarin Prover: https://tamarin-prover.com/ +- "A Formal Security Analysis of the Signal Messaging Protocol" by Cohn-Gordon et al. (2017) -- this paper formally verified Signal using Tamarin + +--- + +## Challenge Completion Tracker + +Use this to track your progress. Check off each challenge as you complete it. + +- [ ] Easy 1: Read Receipts +- [ ] Easy 2: Typing Indicators +- [ ] Easy 3: Message Timestamps and Ordering +- [ ] Easy 4: User Online Status +- [ ] Intermediate 5: Message Search (Encrypted) +- [ ] Intermediate 6: File Sharing (Encrypted) +- [ ] Intermediate 7: Multi-Device Sync +- [ ] Advanced 8: Group Encryption (Sender Keys) +- [ ] Advanced 9: Post-Quantum Key Exchange +- [ ] Advanced 10: Disappearing Messages +- [ ] Expert 11: Deniable Authentication +- [ ] Expert 12: Secure Backup and Recovery