diff --git a/PROJECTS/advanced/hsm-emulator/README.md b/PROJECTS/advanced/hsm-emulator/README.md index e1ae34dd..9dfd1b39 100644 --- a/PROJECTS/advanced/hsm-emulator/README.md +++ b/PROJECTS/advanced/hsm-emulator/README.md @@ -1,3 +1,6 @@ + + + ``` ██╗ ██╗███████╗███╗ ███╗ ███████╗███╗ ███╗██╗ ██╗██╗ █████╗ ████████╗ ██████╗ ██████╗ ██║ ██║██╔════╝████╗ ████║ ██╔════╝████╗ ████║██║ ██║██║ ██╔══██╗╚══██╔══╝██╔═══██╗██╔══██╗ @@ -13,21 +16,51 @@ [![Verified with](https://img.shields.io/badge/verified-pkcs11--tool-green?style=flat)](https://github.com/OpenSC/OpenSC) [![License: AGPLv3](https://img.shields.io/badge/License-AGPL_v3-purple.svg)](https://www.gnu.org/licenses/agpl-3.0) -> A software **Hardware Security Module** that compiles to a real Cryptoki (PKCS#11) shared object. Load it with `pkcs11-tool`, OpenSSL, or any PKCS#11 host the same way you would a real smartcard or HSM — it speaks the C ABI byte-for-byte. +> A software **Hardware Security Module** that compiles to a real Cryptoki (PKCS#11) shared object. Load it with `pkcs11-tool`, OpenSSL, or any PKCS#11 host the same way you would a real smartcard or HSM. It speaks the C ABI byte for byte, generates and stores keys, signs and encrypts, and keeps private key material sealed on disk and zeroized in RAM. ## Why PKCS#11 in Zig -PKCS#11 (Cryptoki) is the C-ABI standard that smartcards, YubiKeys, and cloud HSMs all speak. A conforming module is a `.so` that exports one function — `C_GetFunctionList` — returning a 68-entry table of function pointers in a *fixed canonical order*. Get one struct offset or one pointer slot wrong and the host loads garbage. +PKCS#11 (Cryptoki) is the C-ABI standard that smartcards, YubiKeys, and cloud HSMs all speak. A conforming module is a `.so` that exports one function, `C_GetFunctionList`, returning a 68-entry table of function pointers in a *fixed canonical order*. Get one struct offset or one pointer slot wrong and the host loads garbage. -That makes it a perfect showcase for Zig's C interop: `extern struct` with natural alignment, `callconv(.c)`, a version script that exports exactly one symbol, and a hand-written ABI that is **machine-checked against the official OASIS headers at build time**. +That makes it a near-perfect showcase for Zig's C interop: `extern struct` with natural alignment, `callconv(.c)`, a version script that exports exactly one symbol, and a hand-written ABI that is **machine-checked against the official OASIS headers at build time**. On top of that ABI sits a real HSM: a login model, an attribute-bag object store, a full set of cryptographic mechanisms, and encrypted-at-rest key storage. -## What Works Today (M0) +## What Works Today -- Loads cleanly under OpenSC `pkcs11-tool` 0.26.1 — enumerates the slot and token (`-L`) and advertises 19 mechanisms (`-M`) -- Exports **only** `C_GetFunctionList` (verified with `objdump -T`) +This is not a stub. A host can drive the module through a complete key lifecycle, and every capability below is exercised by a cross-process `pkcs11-tool` run, an in-process smoke harness against the built `.so`, and unit tests. + +**The ABI and the module** +- Loads under OpenSC `pkcs11-tool` 0.26.1, enumerates the slot and token (`-L`), advertises **21 mechanisms** (`-M`) +- Exports **only** `C_GetFunctionList` (verified with `objdump -T`); 34 `__ubsan_*` symbols are kept out of the dynamic table - The full v2.40 ABI hand-written in `src/ck.zig`: every type, 200+ constants, every struct, and the 68-entry `CK_FUNCTION_LIST` in canonical order -- A build-time cross-check (`zig build test`) that translates the vendored OASIS headers and asserts `@sizeOf` / `@offsetOf` / constant equality **and per-function C-ABI signatures** against `ck.zig` — the spec compliance is a compile-time invariant, not a hope -- General + slot/token entry points implemented for real; session, object, crypto, key-management, and RNG entry points are typed stubs returning `CKR_FUNCTION_NOT_SUPPORTED` until their milestone lands +- A build-time cross-check (`zig build test`) translates the vendored OASIS headers and asserts `@sizeOf` / `@offsetOf` / constant equality **and per-function C-ABI signatures** against `ck.zig`. Spec compliance is a compile-time invariant, not a hope + +**Tokens, sessions, login** +- `C_InitToken` / `C_InitPIN` / `C_SetPIN` / `C_Login` / `C_Logout` with a real Security Officer and User role split +- PINs are stretched with **Argon2id** (t=3, m=64 MiB, p=1); only salt and hash touch disk, never the PIN +- Three wrong attempts trip a lockout (`CKR_PIN_LOCKED`), reflected in the token flags +- Read-only versus read-write session enforcement and the full session state machine + +**Objects** +- `C_CreateObject` / `C_CopyObject` / `C_DestroyObject` / `C_GetObjectSize` +- Two-call `C_GetAttributeValue`, per-attribute, with `CKR_ATTRIBUTE_SENSITIVE` for sealed key material +- `C_FindObjects` triad with `CKA_PRIVATE` login gating (private objects are invisible before login) +- `CKA_MODIFIABLE` / `CKA_DESTROYABLE` honored + +**Cryptography** +- Digest: SHA-256 / SHA-384 / SHA-512, single-shot and multi-part +- HMAC: HMAC-SHA-256 / 384 / 512, with constant-time tag verification +- AES: CBC, CBC-PAD, and GCM (128 and 256-bit keys), encrypt and decrypt, with streaming GCM that buffers until the tag verifies +- ECDSA: P-256 and P-384 keygen, `CKM_ECDSA` and `CKM_ECDSA_SHA256`, cross-verified against OpenSSL +- RSA via libcrypto: 2048 to 4096-bit keygen, PKCS#1 v1.5, PSS, and OAEP; sign / verify / encrypt / decrypt and Sign / VerifyRecover +- Key management: `C_GenerateKey` (AES), `C_GenerateKeyPair` (RSA / EC), `C_WrapKey` / `C_UnwrapKey` (AES-KEY-WRAP RFC 3394 and RSA-OAEP), `C_DeriveKey` (ECDH), `C_DigestKey` +- RNG: `C_GenerateRandom` drawn from `std.Io.randomSecure` (the OS CSPRNG) + +**Encrypted at rest, zeroized in RAM** +- Token objects persist to a file. Sensitive attribute *values* are sealed with AES-256-GCM under a per-token master key +- The master key is wrapped under a single User-PIN keyslot (Argon2id-derived KEK). There is no Security-Officer keyslot for user secrets by design +- On `C_Logout`, `C_CloseAllSessions`, and `C_Finalize`, sealed secrets are re-sealed and the master key is wiped with `std.crypto.secureZero`. A failed re-seal fails closed (the plaintext is scrubbed in place) + +See [`learn/CONFORMANCE.md`](learn/CONFORMANCE.md) for the precise return code at every deliberate boundary. ## Quick Start @@ -37,11 +70,23 @@ cd Cybersecurity-Projects/PROJECTS/advanced/hsm-emulator ./install.sh ``` -`install.sh` checks for Zig 0.16, OpenSC, and OpenSSL, builds the module in ReleaseSafe, runs the ABI cross-check + smoke test, and confirms `pkcs11-tool` can load it. Then drive it like any real token: +`install.sh` checks for Zig 0.16, OpenSC, and OpenSSL, builds the module in ReleaseSafe, runs the ABI cross-check plus the smoke test, and confirms `pkcs11-tool` can load it. Then drive it like any real token. Point both storage paths at a scratch directory so you do not write into `$HOME`: ```bash -pkcs11-tool --module zig-out/lib/libhsm.so -L # list slots and token -pkcs11-tool --module zig-out/lib/libhsm.so -M # list mechanisms +export ANGELAMOS_HSM_TOKEN=/tmp/hsm-token +export ANGELAMOS_HSM_OBJECTS=/tmp/hsm-objects +MOD=zig-out/lib/libhsm.so + +pkcs11-tool --module $MOD -L # list slots and token +pkcs11-tool --module $MOD -M # list 21 mechanisms + +pkcs11-tool --module $MOD --init-token --label demo --so-pin 12345678 +pkcs11-tool --module $MOD --init-pin --so-pin 12345678 --pin 1234 +pkcs11-tool --module $MOD -l --pin 1234 \ + --keypairgen --key-type rsa:2048 --label signer +pkcs11-tool --module $MOD -l --pin 1234 \ + --sign --mechanism SHA256-RSA-PKCS --label signer \ + --input-file message.bin --output-file sig.bin ``` ``` @@ -51,10 +96,24 @@ Slot 0 (0x0): AngelaMos HSM Emulator Slot 0 ``` > [!TIP] -> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` to see everything. `just spy -L` wraps the module in `pkcs11-spy.so` and logs every Cryptoki call — the fastest way to watch the ABI work. +> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` to see everything. `just spy -L` wraps the module in `pkcs11-spy.so` and logs every Cryptoki call with its arguments and return code. It is the fastest way to watch the ABI work. > > Install: `curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin` +## Learn + +This project ships a full teaching track. Read it in order, or jump to what you need. + +| Doc | What it covers | +|-----|----------------| +| [`learn/00-OVERVIEW.md`](learn/00-OVERVIEW.md) | What an HSM is, why it exists, and a 10-minute tour | +| [`learn/01-CONCEPTS.md`](learn/01-CONCEPTS.md) | Key sensitivity, the login model, encryption at rest, constant-time, padding oracles, with real breaches | +| [`learn/02-ARCHITECTURE.md`](learn/02-ARCHITECTURE.md) | The three-layer design, object model, locking, the threat model | +| [`learn/03-IMPLEMENTATION.md`](learn/03-IMPLEMENTATION.md) | A code walkthrough of the ABI, an end-to-end operation, and the secret-handling patterns | +| [`learn/MECHANICS.md`](learn/MECHANICS.md) | How each cryptographic mechanism actually works, byte by byte | +| [`learn/CONFORMANCE.md`](learn/CONFORMANCE.md) | The v2.40 conformance statement: every narrowed behavior and its exact return code | +| [`learn/04-CHALLENGES.md`](learn/04-CHALLENGES.md) | Extension ideas from beginner to expert | + ## Architecture The same three-layer split SoftHSM2 uses: a thin C-ABI façade over typed core state over the store and crypto backends. @@ -70,37 +129,42 @@ The same three-layer split SoftHSM2 uses: a thin C-ABI façade over typed core s │ ┌───────────────────────┴─────────────────────┐ │ ABI façade src/ck.zig + src/api/*.zig │ hand-written Cryptoki ABI - │ general · slot_token · session · object · │ + per-call entry points - │ crypto_ops · keymgmt · random │ + │ general · slot_token · session · object · │ + per-call entry points, + │ crypto_ops · keymgmt · random │ argument and FSM validation └───────────────────────┬─────────────────────┘ │ ┌───────────────────────┴─────────────────────┐ - │ core state src/core/{state,lock}.zig │ global instance, init args, - │ │ C-boundary-safe locking + │ core state src/core/*.zig │ global instance behind a lock, + │ state · session · object_store · token │ sessions, objects, PIN, master key └───────────────────────┬─────────────────────┘ │ ┌───────────────────────┴─────────────────────┐ - │ store + crypto (built milestone by │ in-memory → encrypted file - │ milestone: sessions, objects, AES/EC/RSA) │ backend at rest + │ crypto src/crypto/*.zig │ pure-Zig std.crypto for AES/EC/ + │ digest · mac · cipher · ecdsa · rsa · │ hash/HMAC/ECDH, libcrypto for RSA, + │ keystore · pin · openssl │ Argon2id KDF, GCM envelope at rest └───────────────────────────────────────────────┘ ``` -**Design decisions:** non-RSA crypto is pure-Zig `std.crypto`; RSA links libcrypto (OpenSSL EVP) since `std.crypto` has no public RSA. RNG is sourced from `getrandom(2)` directly (there is no `std.Io` at the C boundary, and `std.crypto.random` was removed in Zig 0.16). The ABI is structured for v2.40 with room to add the v3.0 `C_GetInterface` surface later. +**Design decisions:** non-RSA crypto is pure-Zig `std.crypto`. RSA links libcrypto (OpenSSL EVP) because `std.crypto` has no public RSA. The RSA binding is hand-written `extern` declarations, not `@cImport`, so the production `.so` exports nothing but `C_GetFunctionList`. The RNG is `std.Io.randomSecure`, which draws fresh entropy from the OS on every call (`getrandom(2)` on Linux) and keeps no CSPRNG state in process memory. The ABI is structured for v2.40 with room to add the v3.0 `C_GetInterface` surface later. ## Build and Test ```bash -zig build # build the module → zig-out/lib/libhsm.so -zig build test # ABI cross-check vs OASIS headers + unit tests -zig build smoke # dlopen the built .so and exercise the ABI as a host would +zig build # build the module → zig-out/lib/libhsm.so (Debug) +zig build --release=safe # the shipped artifact: ReleaseSafe, UB checks as traps +zig build test # ABI cross-check vs OASIS headers + the unit suite +zig build smoke # dlopen the built .so and exercise the whole ABI as a host would just ci # fmt-check + test + smoke ``` -The smoke harness in `examples/smoke.zig` is not a unit test — it `dlopen`s the *actual built shared object* and calls through the function list exactly like an external host, so it catches export and ABI-shape bugs that in-process tests cannot. +The smoke harness in `examples/smoke.zig` is not a unit test. It `dlopen`s the *actual built shared object* and calls through the function list exactly like an external host, so it catches export and ABI-shape bugs that in-process tests cannot. It walks a full lifecycle: init, login, keygen, sign, encrypt, wrap, derive, GCM streaming, dual-function, recover, operation-state, and the conformance edges. + +> [!NOTE] +> Plain `zig build` produces a **Debug** binary. The shipped artifact is `--release=safe` (ReleaseSafe), which keeps every undefined-behavior check live and turns it into a fail-closed trap rather than silent corruption. Set both `ANGELAMOS_HSM_TOKEN` and `ANGELAMOS_HSM_OBJECTS` to a scratch path for tests and tool runs, or the module falls back to `$HOME/.angelamos-hsm-*`. ## Run in Docker -No Zig or OpenSC on the host? The container builds the module and drives it end-to-end through `pkcs11-tool` — token init, RSA + EC keygen and signing, AES-CBC round-trip — all inside the image. +No Zig or OpenSC on the host? The container builds the module and drives it end to end through `pkcs11-tool`: token init, RSA and EC keygen and signing, AES-CBC round-trip, all inside the image. ```bash just docker-demo # build the image, then run the full pkcs11-tool demo @@ -113,48 +177,69 @@ docker build -t angelamos-hsm:latest . docker run --rm angelamos-hsm:latest ``` -A multi-stage build compiles the module in ReleaseSafe in a `debian-slim` builder, then ships only the `.so` plus `opensc` and `libssl3` in a ~96 MB runtime image. The demo exits non-zero if any signature fails to verify. +A multi-stage build compiles the module in ReleaseSafe in a `debian-slim` builder, then ships only the `.so` plus `opensc` and `libssl3` in a roughly 96 MB runtime image. The demo exits non-zero if any signature fails to verify. ## Project Structure ``` hsm-emulator/ -├── build.zig # addLibrary(.dynamic), version script, test + smoke steps, translate-c +├── build.zig # addLibrary(.dynamic), version script, sanitize_c=.trap, +│ # libcrypto link, translate-c ABI cross-check, test + smoke ├── build.zig.zon # package manifest -├── pkcs11.map # version script — exports only C_GetFunctionList +├── pkcs11.map # version script: exports only C_GetFunctionList ├── src/ │ ├── ck.zig # the hand-written Cryptoki v2.40 ABI (types, constants, structs, list) │ ├── config.zig # identity strings, key-size bounds, mechanism list (no magic numbers) │ ├── util.zig # comptime helpers (space-padded fixed fields) │ ├── main.zig # exported C_GetFunctionList + the wired 68-slot table │ ├── core/ -│ │ ├── state.zig # global instance, init-args parsing, atomic init flag -│ │ └── lock.zig # spinlock wrapper (std.Thread.Mutex is gone in 0.16) -│ └── api/ -│ ├── general.zig # C_Initialize / Finalize / GetInfo (locking template) -│ ├── slot_token.zig # slot + token + mechanism queries -│ └── session.zig, object.zig, crypto_ops.zig, keymgmt.zig, random.zig -├── tests/abi_test.zig # @sizeOf/@offsetOf/constant asserts, incl. cross-check vs OASIS -├── examples/smoke.zig # loads the built .so via dlopen and drives it +│ │ ├── state.zig # global instance behind a lock, init-args parsing, generation counter +│ │ ├── lock.zig # spinlock wrapper over std.atomic.Mutex +│ │ ├── env.zig # reads std.c.environ for storage paths at the C boundary +│ │ ├── token.zig # token record: PIN slots, fail counters, wrapped master key +│ │ ├── session.zig # session table, op-state unions, the RUP-safe GCM buffer +│ │ └── object_store.zig# attribute-bag objects, the selective-sealing codec +│ ├── api/ +│ │ ├── general.zig # C_Initialize / Finalize / GetInfo / WaitForSlotEvent +│ │ ├── slot_token.zig # slot + token + mechanism queries, InitToken / InitPIN / SetPIN +│ │ ├── session.zig # OpenSession / Login / Logout / Get+SetOperationState +│ │ ├── object.zig # CreateObject / Find / GetAttributeValue +│ │ ├── crypto_ops.zig # the digest / sign / verify / encrypt / decrypt / dual surface +│ │ ├── keymgmt.zig # GenerateKey(Pair) / WrapKey / UnwrapKey / DeriveKey +│ │ └── random.zig # GenerateRandom / SeedRandom +│ └── crypto/ +│ ├── openssl.zig # hand-written extern EVP/BN/OSSL_PARAM declarations +│ ├── pin.zig # Argon2id derive / verify (constant-time) +│ ├── digest.zig # SHA-2 hasher union + serializable op-state +│ ├── mac.zig # HMAC-SHA-2 union +│ ├── cipher.zig # AES-CBC/CBC-PAD/GCM + RFC 3394 key wrap +│ ├── ecdsa.zig # P-256/384 keygen, sign, verify, ECDH, curve OID + EC point DER +│ ├── rsa.zig # the stateless libcrypto RSA bridge +│ └── keystore.zig # master-key gen, wrap/unwrap, seal/unseal envelope +├── tests/abi_test.zig # @sizeOf/@offsetOf/constant/signature cross-check vs OASIS +├── examples/smoke.zig # loads the built .so via dlopen and drives the whole lifecycle └── vendor/pkcs11/ # unmodified OASIS v2.40 headers (build-time cross-check only) ``` ## Roadmap -Each milestone ends with a proof from a real external tool — no feature is "done" until `pkcs11-tool` or OpenSSL exercises it. +Each milestone ends with a proof from a real external tool. No feature is "done" until `pkcs11-tool` or OpenSSL exercises it. | Milestone | Scope | Proof | |-----------|-------|-------| | **M0** ✅ | Scaffold + hand-written ABI + loadable `.so` | `pkcs11-tool -L/-M`, `objdump -T` | -| **M1** | Sessions + login + PIN (Argon2id, lockout) | `pkcs11-tool --init-token --init-pin --login --change-pin` | -| **M2** | Objects + find (in-memory), `CKA_PRIVATE` gating | `pkcs11-tool -O --read-object` | -| **M3** | RNG + SHA + HMAC + AES-GCM/CBC | `--hash --encrypt --decrypt --generate-random` | -| **M4** | ECDSA P-256/384 + keygen | `--keypairgen EC --sign`, cross-verify with OpenSSL | -| **M5** | RSA via libcrypto (v1.5 / PSS / OAEP) | OpenSSL pkcs11 provider signs through the module | -| **M6** | Encrypted file backend at rest (AES-256-GCM under Argon2id KEK) | persist across restart; tamper → fails closed | -| **M7** | Hardening (secret zeroization, fail-closed) + Docker | — | -| **M8** | Learn modules + mechanism reference + final docs | — | +| **M1** ✅ | Sessions + login + PIN (Argon2id, lockout) | `--init-token --init-pin --login --change-pin` | +| **M2** ✅ | Objects + find, `CKA_PRIVATE` gating | `-O --read-object`, two-call attribute fetch | +| **M3** ✅ | RNG + SHA-2 + HMAC + AES-CBC/CBC-PAD/GCM | `--hash --sign --encrypt --decrypt --generate-random` | +| **M4** ✅ | ECDSA P-256/384 + keygen | `--keypairgen EC --sign`, cross-verify with OpenSSL | +| **M5** ✅ | RSA via libcrypto (v1.5 / PSS / OAEP) | sign / verify / encrypt / decrypt through the module | +| **M6** ✅ | Encrypted store at rest (AES-256-GCM under Argon2id KEK) | survives restart; wrong PIN and tamper fail closed | +| **M7** ✅ | Hardening (secret zeroization, fail-closed relock) + Docker | leak-checked build, `just docker-demo` exits 0 | +| **M9** ✅ | Key management: wrap / unwrap, ECDH derive, digest-key | RFC 3394 KAT, ECDH matches `openssl pkeyutl -derive` | +| **M10** ✅ | Crypto surface: GCM streaming, dual-function, Sign/VerifyRecover, op-state | chunked GCM equals one-shot, recover round-trips | +| **M11** ✅ | Conformance pass + `CONFORMANCE.md` | every N/A boundary asserted against the built `.so` | +| **M12** ✅ | Learn modules + mechanism reference | this `learn/` track | ## License -[AGPL 3.0](LICENSE) +[AGPL 3.0](LICENSE). The vendored OASIS headers under `vendor/pkcs11/` keep their original copyright and are used only for the build-time cross-check. diff --git a/PROJECTS/advanced/hsm-emulator/learn/00-OVERVIEW.md b/PROJECTS/advanced/hsm-emulator/learn/00-OVERVIEW.md new file mode 100644 index 00000000..d94c94a8 --- /dev/null +++ b/PROJECTS/advanced/hsm-emulator/learn/00-OVERVIEW.md @@ -0,0 +1,143 @@ + + + +# HSM Emulator: Overview + +## What This Is + +A software **Hardware Security Module** written in Zig that compiles to a real PKCS#11 (Cryptoki) shared object. Any program that already knows how to talk to a smartcard, a YubiKey, or a cloud HSM (OpenSSL, `pkcs11-tool`, Java's SunPKCS11, p11-kit) can load this `.so` and use it to generate keys, sign, encrypt, and store secrets. It speaks the same C ABI a real HSM speaks, so to the host it is indistinguishable from hardware until you look at where the bytes actually live. + +The point of the project is to understand, by building it, what an HSM actually does and why the interface to one looks the way it does. You get a working module you can poke at with standard tools, and a codebase small enough to read in an afternoon. + +## Why This Matters + +A cryptographic key is the whole game. If an attacker reads your TLS private key, your code-signing key, or your CA root key out of a file or out of process memory, every signature and every certificate that key ever produced is now forgeable. The defense the industry settled on is to never let the application touch the key. The key lives inside a separate trust boundary (a chip, a card, a network appliance) and the application sends it *requests*: "sign this", "decrypt that". The key goes in once and never comes back out. + +The cost of getting this wrong is not hypothetical. + +- **RSA SecurID, 2011.** Attackers breached RSA and exfiltrated seed records tied to SecurID tokens, then used them to attack Lockheed Martin. The lesson the industry took from it: the secrets that authenticate everything else must sit behind a hardware boundary, not in a database an intruder can read. +- **Stuxnet, 2010.** The worm carried drivers signed with legitimate code-signing keys stolen from Realtek and JMicron. Valid signatures from stolen keys let malware sail past trust checks. Code-signing keys are exactly the kind of thing that belongs in an HSM, where the key signs but never leaves. +- **DigiNotar, 2011.** A certificate authority was compromised and issued rogue certificates for `*.google.com`, used to intercept traffic in Iran. The company did not survive it. CA root and issuing keys are the canonical HSM use case: the key must be usable for signing yet impossible to copy. + +**Real world scenarios where this applies:** +- **Certificate authorities and PKI.** A CA's signing keys live in an HSM. The CA software calls `C_Sign`; the key never exists in the CA server's memory in usable form. +- **Code signing.** Apple, Microsoft, and Linux distros sign release artifacts with keys held in HSMs so a server breach cannot mint signed malware. +- **Payment and KMS backends.** AWS KMS and CloudHSM, payment HSMs (PIN translation, card issuance), and database TDE all delegate the actual crypto to a module that exposes an interface very much like this one. + +## What You'll Learn + +This project teaches how key custody actually works under the hood. By building it yourself, you will understand: + +**Security concepts:** +- **Key custody and the trust boundary.** Why "the application never sees the key" is the entire design goal, and how an attribute like `CKA_SENSITIVE` turns into a hard refusal to ever hand `CKA_VALUE` back to the caller. +- **Authentication and lockout.** How a PIN is stretched with Argon2id so the value on disk is useless to a thief, and why three wrong tries must lock the token rather than let an attacker grind. +- **Encryption at rest with key wrapping.** The envelope pattern: a random master key encrypts your secrets, and the master key is itself wrapped under a key derived from your PIN. Change the PIN and you re-wrap one small key instead of re-encrypting everything. +- **Release of unverified plaintext (RUP).** Why a streaming AEAD decrypt that hands back bytes before the authentication tag is checked is dangerous, and how buffering until the tag verifies removes the problem. +- **Side channels.** Why comparing a PIN or a MAC with a normal byte-by-byte compare leaks information through timing, and what constant-time comparison fixes. + +**Technical skills:** +- **Writing a C ABI by hand.** Laying out `extern struct` types so they match a C header byte for byte, and proving it with compile-time `@sizeOf` / `@offsetOf` assertions instead of hoping. +- **Calling a C library safely from Zig.** Binding OpenSSL's `libcrypto` through hand-written `extern` declarations (RSA lives there because Zig's standard library has none), without leaking its symbols out of your module. +- **Concurrency at a C boundary.** Holding a global instance behind a lock, and using a generation counter so a slow operation that started before the token was reinitialized cannot commit stale results. +- **Secret hygiene in a systems language.** Zeroizing key material with `std.crypto.secureZero`, understanding what the `volatile` trick buys you, and freeing heap secrets without leaving copies behind. + +**Tools and techniques:** +- **`pkcs11-tool`** (from OpenSC), the standard command-line host for any PKCS#11 module. You will use it to init the token, log in, generate keys, and sign. +- **`pkcs11-spy`**, a shim that sits in front of any module and logs every call with its arguments and return code. It is the single best debugging tool for ABI work. +- **OpenSSL as an oracle.** Verifying that a signature this module produces validates under OpenSSL, and that an ECDH secret it derives matches `openssl pkeyutl -derive` byte for byte. + +## Prerequisites + +You do not need prior HSM experience. You do need some comfort with the following. + +**Required knowledge:** +- **C ABI basics.** What a struct's memory layout is, what a function pointer is, what "calling convention" means. The whole module is an exercise in matching a C interface exactly. +- **Symmetric versus asymmetric crypto.** The difference between AES (one shared key) and RSA/EC (a keypair). You do not need the math, but you should know which is which and what "sign" versus "encrypt" means. +- **Basic Zig or a willingness to read it.** The code uses tagged unions, `comptime`, error unions, and slices. If you know Rust, Go, or modern C++, you can follow it. + +**Tools you'll need:** +- **Zig 0.16.0** to build the module. The exact version matters; the `std.Io` interface and the build API changed in 0.16. +- **OpenSC** for `pkcs11-tool` and `pkcs11-spy`. On Debian or Ubuntu: `apt install opensc`. +- **OpenSSL development headers** (`libssl-dev`) so the module can link `libcrypto` for RSA. + +**Helpful but not required:** +- A reading of the [PKCS#11 v2.40 base specification](https://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html). You can also just read [`01-CONCEPTS.md`](./01-CONCEPTS.md) and pick up the spec when something is unclear. +- Familiarity with how SoftHSM2 (the reference open-source software HSM) is structured. This project borrows its three-layer split. + +## Quick Start + +```bash +cd PROJECTS/advanced/hsm-emulator + +# Build, cross-check the ABI, and confirm pkcs11-tool can load it +./install.sh + +# Keep all state in a scratch directory instead of $HOME +export ANGELAMOS_HSM_TOKEN=/tmp/hsm-token +export ANGELAMOS_HSM_OBJECTS=/tmp/hsm-objects +MOD=zig-out/lib/libhsm.so + +# Look at the empty token, then bring it to life +pkcs11-tool --module $MOD -L +pkcs11-tool --module $MOD --init-token --label demo --so-pin 12345678 +pkcs11-tool --module $MOD --init-pin --so-pin 12345678 --pin 1234 + +# Generate a keypair and sign with it +pkcs11-tool --module $MOD -l --pin 1234 --keypairgen --key-type EC:prime256v1 --label k1 +echo "hello hsm" > msg.bin +pkcs11-tool --module $MOD -l --pin 1234 --sign --mechanism ECDSA-SHA256 --label k1 \ + --input-file msg.bin --output-file sig.bin +``` + +Expected output: `-L` shows one slot whose token starts `uninitialized`. After `--init-token` the token reports `flags: ... token initialized`. The keypair generation prints the new public and private object handles, and `--sign` writes a 64-byte signature to `sig.bin`. If you want to *watch* every Cryptoki call the tool makes, run any command through `just spy` (for example `just spy --keypairgen ...`) and read the spy log. + +## Project Structure + +``` +hsm-emulator/ +├── src/ +│ ├── ck.zig # the hand-written Cryptoki ABI (the contract with the host) +│ ├── main.zig # the one exported symbol and the 68-entry function table +│ ├── config.zig # every tunable constant in one place +│ ├── core/ # global state, sessions, the object store, the token record +│ ├── api/ # one file per group of C_ functions (the entry points) +│ └── crypto/ # the actual cryptography (AES, EC, RSA, hashing, the keystore) +├── tests/abi_test.zig # proves ck.zig matches the OASIS headers at build time +├── examples/smoke.zig # loads the built .so and drives it like a real host +└── vendor/pkcs11/ # the official OASIS headers, used only for the cross-check +``` + +The single most important file to understand first is `src/ck.zig`. Everything else exists to fill in the function pointers that file declares. + +## Next Steps + +1. **Understand the ideas.** Read [01-CONCEPTS.md](./01-CONCEPTS.md) for key custody, the login model, encryption at rest, and the side-channel defenses, each grounded in a real incident. +2. **See the design.** Read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) for the three-layer split, the object model, and how concurrency is handled. +3. **Walk the code.** Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) to trace a full operation from the C call down to the crypto and back. +4. **Learn the crypto.** Read [MECHANICS.md](./MECHANICS.md) for how each mechanism (AES modes, GCM, key wrap, ECDSA, ECDH, RSA, Argon2id) works byte by byte. +5. **Check the contract.** Read [CONFORMANCE.md](./CONFORMANCE.md) for the exact return code at every boundary. +6. **Extend it.** Read [04-CHALLENGES.md](./04-CHALLENGES.md) for projects from "add a mechanism" to "implement the v3.0 interface". + +## Common Issues + +**`pkcs11-tool` writes files into my home directory** +``` +~/.angelamos-hsm-token +~/.angelamos-hsm-objects +``` +Solution: set `ANGELAMOS_HSM_TOKEN` and `ANGELAMOS_HSM_OBJECTS` before every command. The module falls back to `$HOME/.angelamos-hsm-*` only when those are unset. + +**`error: unable to find dynamic system library 'crypto'`** +Solution: install the OpenSSL development package (`libssl-dev` on Debian or Ubuntu, `openssl-devel` on Fedora). RSA is the one mechanism family that links a C library. + +**`C_Login` returns `CKR_PIN_LOCKED` and nothing works** +Solution: you tried the wrong PIN three times and the token locked. Re-run `--init-token` (with the Security Officer PIN) to reset it, which also clears all objects. This is the intended behavior; an attacker should not get unlimited tries. + +**A plain `zig build` behaves differently from the installed module** +Solution: `zig build` with no flags is a Debug build. The shipped module is `zig build --release=safe`. Debug and ReleaseSafe differ in how freed and nulled memory is poisoned, which matters when you are inspecting zeroized secrets. Use `--release=safe` to match the artifact. + +## Related Projects + +If you found this interesting, look at: +- **api-rate-limiter**: another advanced project where the security property (correctness under concurrency) lives in carefully designed state handling, here with Lua scripts instead of a Zig mutex. +- **bug-bounty-platform**: shows how key material and credentials are handled in a full application, the layer that would *call* an HSM like this one. diff --git a/PROJECTS/advanced/hsm-emulator/learn/01-CONCEPTS.md b/PROJECTS/advanced/hsm-emulator/learn/01-CONCEPTS.md new file mode 100644 index 00000000..ff603b74 --- /dev/null +++ b/PROJECTS/advanced/hsm-emulator/learn/01-CONCEPTS.md @@ -0,0 +1,351 @@ + + + +# Core Security Concepts + +This document explains the security ideas the project is built on. These are not just definitions. Each one is tied to a real failure and to the exact code in this module that defends against it. + +## Key Custody: the application must never see the key + +### What it is + +An HSM exists to enforce one rule: the secret key goes in once and never comes back out in usable form. The application that needs a signature does not get the key and compute the signature itself. It hands the HSM a request, and the HSM computes the signature inside its own boundary and returns only the result. The key material never crosses back over the line. + +In PKCS#11 this rule is expressed through two attributes on a key object: + +- `CKA_SENSITIVE = CK_TRUE` means the value of the key cannot be read back through the API. Asking for `CKA_VALUE` returns `CKR_ATTRIBUTE_SENSITIVE`, not the bytes. +- `CKA_EXTRACTABLE = CK_FALSE` means the key cannot be wrapped out (encrypted and exported) either. + +### Why it matters + +If the key is just a variable in your process, then any bug that reads process memory reads the key. The 2014 **Heartbleed** bug (CVE-2014-0160) was exactly this: a missing bounds check in OpenSSL let a remote attacker read up to 64 KB of server memory per request, and private TLS keys were among the things people pulled out. Servers whose keys lived in an HSM were not exposed the same way. The key was never in the web server's address space to leak. + +### How it works here + +When you generate a key, the module marks the secret material sensitive and unextractable by default, and it computes the two "always" flags the spec requires. From `keymgmt.zig`: + +```zig +if (!obj.has(ck.CKA_SENSITIVE)) try obj.set(allocator, ck.CKA_SENSITIVE, &[_]u8{ck.CK_TRUE}); +if (!obj.has(ck.CKA_EXTRACTABLE)) try obj.set(allocator, ck.CKA_EXTRACTABLE, &[_]u8{ck.CK_FALSE}); +const always_sensitive: u8 = if (obj.getBool(ck.CKA_SENSITIVE)) ck.CK_TRUE else ck.CK_FALSE; +const never_extractable: u8 = if (!obj.getBool(ck.CKA_EXTRACTABLE)) ck.CK_TRUE else ck.CK_FALSE; +``` + +When a host then tries to read that value, `object.zig`'s `sensitiveProtected` check fires before any bytes are copied, and `C_GetAttributeValue` reports the value length as `CK_UNAVAILABLE_INFORMATION` and returns `CKR_ATTRIBUTE_SENSITIVE`. The key is usable for signing (you pass its *handle* to `C_SignInit`) but its bytes are unreachable. + +You can watch this in the smoke harness: after generating an AES key, the harness asks for `CKA_VALUE` and asserts the answer is `CKR_ATTRIBUTE_SENSITIVE`, not the key. + +### Common attacks + +1. **Memory disclosure.** A buffer over-read (Heartbleed), an uninitialized-memory leak, or a crash dump that ships off-box. If the key was never in your memory in plaintext, the leak yields nothing useful. +2. **Key export through the API.** An attacker who gains a session tries to simply read the key out. `CKA_SENSITIVE` refuses. They try to wrap it out with `C_WrapKey`. `CKA_EXTRACTABLE = CK_FALSE` refuses with `CKR_KEY_UNEXTRACTABLE`. +3. **Cold disk theft.** Someone copies the token file. The sensitive values in it are sealed (see encryption at rest, below), so the file is ciphertext. + +### Defense strategies + +The whole module is the defense, but the core moves are: mark secrets sensitive and unextractable by default, gate every read through one chokepoint (`sensitiveProtected`), and never construct a code path that returns raw secret bytes to the caller. Operations take *handles*, not keys. + +## The Login Model: public objects, private objects, and roles + +### What it is + +A PKCS#11 token has two roles: the **Security Officer** (SO), who administers the token and sets up the User, and the **User**, who actually uses the keys. Objects are either *public* (visible to anyone with a session) or *private* (`CKA_PRIVATE = CK_TRUE`, visible only after the User logs in). + +### Why it matters + +The role split is what lets you hand someone a token without handing them the keys on it. The SO can initialize and reset, but in a correct design the SO cannot read the User's secrets. If the administrator role could read user key material, "administrator" would just be a second word for "attacker who got admin". + +### How it works here + +Visibility is one function, `object_store.visible`: + +```zig +pub fn visible(obj: *const Object, logged_in: ?ck.CK_USER_TYPE) bool { + if (!obj.isPrivate()) return true; + return logged_in == ck.CKU_USER; +} +``` + +Every object-facing entry point (`C_FindObjects`, `C_GetAttributeValue`, `C_DestroyObject`, the key fetches inside `crypto_ops.zig`) calls this first. A private object simply does not exist to a caller who has not logged in as User. The smoke harness proves it: it creates a private object, logs out, runs `C_FindObjects`, and asserts the private object does not appear and that fetching it returns `CKR_OBJECT_HANDLE_INVALID`. The object is not just hidden from listings; it is unreachable by handle. + +Critically, this module gives the SO **no keyslot for user secrets**. The master key that protects sensitive values is wrapped only under the User PIN (see below). An SO who resets the token can wipe it, but cannot read what was there. That is a deliberate design choice, documented in [CONFORMANCE.md](./CONFORMANCE.md) section 4. + +### Common attacks + +1. **Privilege confusion.** An attacker with SO access tries to read User keys. The absence of an SO keyslot means there is nothing for the SO to decrypt with. +2. **Pre-login enumeration.** An attacker without credentials lists objects hoping private keys leak into the listing. `visible` keeps them out. + +## Authentication: stretching the PIN, and locking the door + +### What it is + +The User and SO authenticate with a PIN. A PIN is short and low-entropy by nature (often four to eight digits), so two things must be true: the stored form must be expensive to attack offline, and online guessing must be rate-limited to a hard stop. + +### Why it matters + +If you store the PIN itself, or a fast hash of it, an attacker who steals the token file runs a dictionary in seconds. The 2012 **LinkedIn breach** leaked 6.5 million unsalted SHA-1 password hashes; because SHA-1 is fast and the hashes were unsalted, most were cracked almost immediately. A PIN protected by a fast hash is no better. + +### How it works here + +The PIN is stretched with **Argon2id**, the memory-hard KDF that won the Password Hashing Competition. From `config.zig` and `pin.zig`: + +```zig +const params: argon2.Params = .{ + .t = config.pin_kdf_t, // 3 iterations + .m = config.pin_kdf_m_kib, // 64 MiB of memory + .p = config.pin_kdf_p, // 1 lane +}; +``` + +Each guess costs 64 MiB of memory and three passes over it, which collapses the throughput of a brute-force rig. Only a random 16-byte salt and the 32-byte derived hash ever touch disk. The PIN is never stored in any form. Verification is constant-time: + +```zig +pub fn verify(io: std.Io, allocator: std.mem.Allocator, pin: []const u8, salt: *const Salt, expected: *const Hash) !bool { + var got: Hash = undefined; + defer std.crypto.secureZero(u8, &got); + try derive(io, allocator, pin, salt, &got); + return std.crypto.timing_safe.eql(Hash, got, expected.*); +} +``` + +For online guessing, the token counts failures and locks after three. From `slot_token.zig`, the token info reflects the count low / final try / locked flags, and `C_Login` refuses once the counter reaches the limit: + +```zig +if (inst.token.user_fail >= config.login_max_attempts) { + state.mutex.unlock(); + return ck.CKR_PIN_LOCKED; +} +``` + +The smoke harness drives three wrong PINs and asserts the fourth attempt returns `CKR_PIN_LOCKED` and that `CKF_USER_PIN_LOCKED` is set in the token flags. + +### Common pitfalls + +**Mistake: comparing the hash with a normal equality check** +```zig +// Bad: std.mem.eql returns as soon as it finds a mismatched byte. +// The time it takes leaks how many leading bytes matched. +return std.mem.eql(u8, &got, expected); + +// Good: timing_safe.eql looks at every byte before deciding. +return std.crypto.timing_safe.eql(Hash, got, expected.*); +``` + +**Mistake: a fast hash for the PIN** +```zig +// Bad: a thief with the file runs billions of SHA-256 guesses per second. +var h: [32]u8 = undefined; +std.crypto.hash.sha2.Sha256.hash(pin, &h, .{}); + +// Good: Argon2id makes each guess cost 64 MiB and three passes. +try argon2.kdf(allocator, &h, pin, &salt, params, .argon2id, io); +``` + +## Encryption at Rest and the Envelope Pattern + +### What it is + +The token persists to a file so keys survive a restart. The sensitive values in that file are encrypted. The clever part is *how* the encryption key is managed: a random per-token **master key** (MK) encrypts the secrets, and the MK is itself encrypted ("wrapped") under a key derived from the User PIN. This two-level scheme is the **envelope** pattern. + +### Why it matters + +Encrypting each secret directly under a PIN-derived key sounds simpler, but it means changing the PIN requires re-encrypting every secret. With the envelope, changing the PIN re-wraps one 32-byte master key and nothing else. It also means the expensive Argon2id derivation happens once per login, not once per object. AWS KMS, Google Tink, and every serious secrets system use envelope encryption for these reasons. + +The threat it answers is plaintext storage of secrets, CWE-312 (cleartext storage of sensitive information). A token file that is just key bytes on disk is a single `cat` away from total compromise. + +### How it works here + +``` + User PIN ──Argon2id──▶ KEK ──AES-256-GCM wrap──▶ wrapped MK (on disk, in the token record) + │ + C_Login unwraps with the KEK + ▼ + random master key (MK) ──AES-256-GCM seal──▶ sealed secret values (on disk, in the object file) +``` + +`keystore.zig` holds the two halves. `wrap` derives the KEK from the PIN and GCM-encrypts the MK; `unwrap` reverses it, and a wrong PIN fails the GCM tag and returns false rather than garbage: + +```zig +pub fn unwrap(io: std.Io, allocator: std.mem.Allocator, pin_bytes: []const u8, w: *const Wrapped, out: *MasterKey) !bool { + var kek: MasterKey = undefined; + defer std.crypto.secureZero(u8, &kek); + try deriveKek(io, allocator, pin_bytes, &w.salt, &kek); + gcm.decrypt(out, &w.ct, w.tag, "", w.nonce, kek) catch { + std.crypto.secureZero(u8, out); + return false; + }; + return true; +} +``` + +`seal` and `unseal` protect the individual attribute values, binding each one to its attribute type as associated data so a sealed `CKA_PRIVATE_EXPONENT` cannot be swapped in where a `CKA_VALUE` was expected. The object store seals *selectively*: only the sensitive values (the AES key bytes, the RSA private exponent and CRT factors, the EC scalar) are encrypted. Public material (the modulus, the EC point, labels) stays plaintext so it is visible before login, which is spec-correct. + +### Common attacks + +1. **Disk theft.** The file is sealed under a key the thief does not have. Without the PIN, the MK stays wrapped. +2. **Tampering.** Flipping a byte in a sealed value breaks the GCM tag on unseal, and the module fails closed. The store-level tests flip a byte and assert `error.AuthFailed`. +3. **Downgrade.** An attacker swaps in an old token file from before a PIN change. The MK in that file is wrapped under the old KEK, so the new PIN cannot unwrap it. The record version is also bumped, so an old plaintext-era file is rejected outright. + +## Release of Unverified Plaintext (RUP) + +### What it is + +AES-GCM is an authenticated cipher: decryption both decrypts and verifies a 128-bit tag, and the tag is what tells you the ciphertext was not tampered with. A *streaming* decrypt that returns plaintext chunk by chunk has a problem: it hands you bytes before it has seen the tag. If you act on those bytes and the tag later turns out to be wrong, you acted on attacker-controlled data. That is release of unverified plaintext. + +### Why it matters + +For an HSM this is unacceptable. The whole point is to be trustworthy about what it returns. Returning plaintext that has not been authenticated, even briefly, is a foothold for chosen-ciphertext attacks. The cryptographic community treats RUP resistance as a property a serious AEAD usage must have. + +### How it works here + +The module makes RUP impossible by construction. GCM is implemented as a **buffered** operation: `C_EncryptUpdate` and `C_DecryptUpdate` append to an internal buffer and emit zero bytes, and the real work happens once at `C_*Final`, where decrypt verifies the tag before any plaintext is released. The accumulator lives in `session.zig` as `GcmStream`, and it is bounded to 16 MiB so a host cannot exhaust memory by streaming forever: + +```zig +pub fn append(self: *GcmStream, allocator: std.mem.Allocator, bytes: []const u8) error{ OutOfMemory, TooLarge }!void { + if (bytes.len == 0) return; + const needed = self.len + bytes.len; + if (needed > config.max_gcm_stream_len) return error.TooLarge; + ... +} +``` + +The smoke harness streams a multi-block message through `C_DecryptUpdate` in 19-byte chunks, asserts that every update emits zero bytes, and only gets the plaintext at `C_DecryptFinal`. It then flips a byte and confirms decryption returns `CKR_ENCRYPTED_DATA_INVALID` with no plaintext released. + +## Side Channels: timing, and zeroizing memory + +### What it is + +A side channel is information that leaks through *how* a computation runs rather than its output: how long it took, what memory it touched, what the cache state is afterward. Two side channels matter most for a key-handling module: timing (a comparison that exits early reveals where it stopped) and residue (a secret left in freed memory after use). + +### Why it matters + +**Timing.** The 1998 **Bleichenbacher attack** and its 2017 revival **ROBOT** recovered RSA-encrypted secrets by measuring how a server responded to malformed PKCS#1 v1.5 ciphertexts. **Lucky Thirteen** (2013) did the same against TLS CBC padding using timing alone. The lesson: any branch whose timing depends on a secret is a leak. + +**Residue.** The 2008 **cold boot attack** (Halderman et al.) showed that DRAM retains its contents for seconds to minutes after power loss, long enough to dump and recover keys. If a key sits in freed memory, a crash dump, a swapped-out page, or a cold-boot read can recover it. + +### How it works here + +Comparisons of secrets are constant-time. PIN verification uses `std.crypto.timing_safe.eql`; MAC verification in `crypto_ops.zig` uses a hand-rolled `ctEql` that ORs every byte difference before deciding: + +```zig +fn ctEql(a: []const u8, b: []const u8) bool { + if (a.len != b.len) return false; + var diff: u8 = 0; + for (a, b) |x, y| diff |= x ^ y; + return diff == 0; +} +``` + +Secrets are zeroized when they go out of scope. Stack secrets use `defer std.crypto.secureZero(...)`; the session operation unions zeroize themselves on teardown; the object store frees every secret value through `secureFree`, which scrubs before releasing: + +```zig +fn secureFree(allocator: std.mem.Allocator, value: []u8) void { + std.crypto.secureZero(u8, value); + allocator.free(value); +} +``` + +`secureZero` takes a `[]volatile` slice precisely so the compiler is forbidden from optimizing the write away as dead (it has no subsequent read). On logout the whole store re-seals and the master key is wiped, so an idle, logged-out token holds no plaintext secrets in RAM. [MECHANICS.md](./MECHANICS.md) covers the constant-time and zeroization mechanics in more depth. + +### Common pitfalls + +**Mistake: zeroizing through a non-volatile slice** +```zig +// Bad: with no later read, the optimizer may delete this @memset entirely. +@memset(&key, 0); + +// Good: secureZero forces the store to happen. +std.crypto.secureZero(u8, &key); +``` + +**Mistake: a padding oracle through distinct error codes** +```zig +// Bad: returning a different error for "bad padding" vs "bad MAC" tells an +// attacker which step failed, which is what Bleichenbacher/Lucky13 exploit. + +// Good: this module maps RSA decrypt failures to one uniform CKR code and +// lets libcrypto handle the padding check in constant time. CBC-PAD padding +// verification ORs all the pad bytes before deciding, no early exit. +``` + +## How These Concepts Relate + +The concepts are layers of one system. Each depends on the one below it. + +``` + Key custody (sensitive, unextractable) + │ requires + ▼ + The login model (who may see what) + │ requires + ▼ + Authentication (Argon2id PIN + lockout) + │ unlocks + ▼ + Encryption at rest (envelope: PIN -> KEK -> MK -> sealed secrets) + │ relies on + ▼ + Authenticated encryption done safely (GCM, no RUP) + │ relies on + ▼ + Side-channel hygiene (constant-time compares, zeroized memory) +``` + +If the constant-time compare leaks, the PIN falls. If the PIN falls, the envelope opens. If the envelope opens, the login model is moot and key custody is broken. The strength of the chain is the strength of its weakest link, which is why the module is uniform about all of them. + +## Industry Standards and Frameworks + +### OWASP Top 10 (2021) + +- **A02:2021 Cryptographic Failures.** The headline category. This project is a study in not committing them: strong KDF for the PIN, AES-256-GCM for storage, fresh nonces, authenticated encryption, no home-grown primitives except where they are textbook (CBC mode assembly, RFC 3394 wrap) and tested against published vectors. +- **A04:2021 Insecure Design.** The envelope pattern, the SO-cannot-read-User-secrets decision, and RUP-safe GCM are design-level choices, not bolt-ons. +- **A07:2021 Identification and Authentication Failures.** The PIN lockout and the role model address brute force and privilege confusion. + +### MITRE ATT&CK + +- **T1552 Unsecured Credentials** and **T1555 Credentials from Password Stores.** An HSM is the countermeasure: credentials that authenticate everything else are not sitting in a readable store. +- **T1003 OS Credential Dumping.** Zeroization and never-in-plaintext-memory custody reduce what a memory dump yields. +- **T1588.004 / T1649 Obtain or Forge Certificates.** Keeping signing keys unextractable is what stops a breach from minting forged certificates, the DigiNotar failure mode. + +### CWE + +- **CWE-312 Cleartext Storage of Sensitive Information.** Defended by the encrypted-at-rest envelope. +- **CWE-316 Cleartext Storage in Memory.** Defended by zeroization and the logout relock. +- **CWE-208 Observable Timing Discrepancy.** Defended by constant-time comparison. +- **CWE-326 Inadequate Encryption Strength** and **CWE-327 Use of a Broken or Risky Cryptographic Algorithm.** Defended by AES-256-GCM, Argon2id, and standard signature schemes. +- **CWE-522 Insufficiently Protected Credentials.** Defended by Argon2id stretching and lockout. + +## Real World Examples + +### Case study: Heartbleed (CVE-2014-0160, 2014) + +A missing bounds check in OpenSSL's TLS heartbeat let a remote attacker read server memory in 64 KB chunks. Private keys, session cookies, and passwords all leaked. The defense that worked was custody: organizations whose private keys lived in an HSM did not leak those keys, because the keys were never in the vulnerable process's memory. This project's `CKA_SENSITIVE` enforcement and zeroization are the same principle in miniature. + +### Case study: the Bleichenbacher family (1998, ROBOT 2017) + +Daniel Bleichenbacher showed that an RSA decryption oracle which reveals whether PKCS#1 v1.5 padding was valid lets an attacker decrypt a ciphertext with about a million queries. Nineteen years later ROBOT found the same flaw still live in major TLS stacks because the "fixes" were not uniform. The takeaway baked into this module: never let the error you return depend on which internal check failed, and prefer OAEP, whose design resists the attack. RSA decrypt failures here collapse to one return code. + +## Testing Your Understanding + +Before moving on, make sure you can answer these. + +1. A host generates an AES key, then immediately calls `C_GetAttributeValue` for `CKA_VALUE`. What does it get back, and which function decided that? What would the answer be for `CKA_MODULUS_BITS` on an RSA public key, and why is that different? +2. Walk the envelope from a User PIN to a decrypted secret value. Name each key in the chain and what encrypts what. Why does changing the PIN not require re-encrypting every object? +3. Why does `C_DecryptUpdate` for AES-GCM return zero bytes every time, and where does the plaintext actually appear? What attack would a chunk-by-chunk release enable? +4. The SO can reset the token. Why can the SO not read the User's keys? Point to the design decision that makes that true. + +If any of these are fuzzy, re-read the matching section. The implementation will make far more sense once these click. + +## Further Reading + +**Essential:** +- [PKCS#11 v2.40 Base Specification (OASIS)](https://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html). The contract this module implements. Read the function semantics and the object model sections. +- [SoftHSMv2](https://github.com/softhsm/SoftHSMv2). The reference open-source software HSM. This project borrows its three-layer split and its object store idea. + +**Deep dives:** +- Bleichenbacher, "Chosen Ciphertext Attacks Against Protocols Based on the RSA Encryption Standard PKCS #1" (CRYPTO 1998), and the ROBOT writeup at robotattack.org for the modern recurrence. +- Halderman et al., "Lest We Remember: Cold Boot Attacks on Encryption Keys" (USENIX Security 2008), the motivation for `secureZero`. +- The Argon2 paper and RFC 9106 for why a memory-hard KDF beats a fast hash for low-entropy secrets. + +**Historical context:** +- The original RSA Security PKCS#11 documents (pre-OASIS) for how the standard came to look the way it does, and why "the function list is the API". diff --git a/PROJECTS/advanced/hsm-emulator/learn/02-ARCHITECTURE.md b/PROJECTS/advanced/hsm-emulator/learn/02-ARCHITECTURE.md new file mode 100644 index 00000000..761e9ead --- /dev/null +++ b/PROJECTS/advanced/hsm-emulator/learn/02-ARCHITECTURE.md @@ -0,0 +1,430 @@ + + + +# System Architecture + +This document explains how the module is built and why it is built that way. It assumes you have read [01-CONCEPTS.md](./01-CONCEPTS.md). + +## High Level Architecture + +The module is three layers. A thin C-ABI façade takes calls from the host and validates them. A core-state layer owns the live token, sessions, and objects behind a single lock. A crypto layer does the actual mathematics and the storage codec. Calls flow down, results flow back up, and the only thing the host ever sees is the function table at the top. + +``` + ┌──────────────────────────────────────────────────────────────┐ + │ HOST: pkcs11-tool, OpenSSL, p11-kit, Java SunPKCS11 │ + └───────────────────────────────┬──────────────────────────────┘ + │ C ABI, through a function-pointer table + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ LAYER 1 ABI façade │ + │ main.zig one exported symbol, the 68-slot table │ + │ ck.zig every CK_ type, constant, struct │ + │ api/general Initialize / Finalize / GetInfo │ + │ api/slot_token slot + token + mechanism queries, PIN admin │ + │ api/session OpenSession / Login / Logout / op-state │ + │ api/object CreateObject / Find / GetAttributeValue │ + │ api/crypto_ops digest / sign / verify / encrypt / decrypt │ + │ api/keymgmt GenerateKey(Pair) / Wrap / Unwrap / Derive │ + │ api/random GenerateRandom / SeedRandom │ + └───────────────────────────────┬──────────────────────────────┘ + │ acquire() the instance under the lock + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ LAYER 2 core state (one global Instance, one mutex) │ + │ state the Instance, init-args, generation counter │ + │ lock the mutex (std.atomic.Mutex + spin/yield) │ + │ session session Table, op-state unions, GCM buffer │ + │ object_store attribute-bag objects, the persist codec │ + │ token the token record: PINs, fail counts, MK │ + │ env reads storage paths from std.c.environ │ + └───────────────────────────────┬──────────────────────────────┘ + │ call the primitives + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ LAYER 3 crypto │ + │ digest / mac SHA-2, HMAC (pure Zig std.crypto) │ + │ cipher AES-CBC/CBC-PAD/GCM, RFC 3394 wrap (pure) │ + │ ecdsa P-256/384 sign/verify/keygen, ECDH (pure) │ + │ rsa + openssl RSA via libcrypto (hand-written extern) │ + │ pin Argon2id derive/verify (pure) │ + │ keystore master-key wrap, value seal/unseal (pure) │ + └──────────────────────────────────────────────────────────────┘ +``` + +### Component breakdown + +**`main.zig` (the entry point).** Exports exactly one symbol, `C_GetFunctionList`, which hands the host a pointer to a static `CK_FUNCTION_LIST`. That table's 68 function pointers are wired to the `api/*` functions in canonical order. A `comptime` block asserts the table is `69 * @sizeOf(usize)` bytes (version plus 68 pointers) and that `CK_ATTRIBUTE` is 24 bytes, so an ABI regression cannot compile. + +**`ck.zig` (the contract).** The hand-written Cryptoki v2.40 ABI: scalar typedefs, 200+ constants, every struct laid out for the Linux LP64 ABI with natural alignment, the function-pointer typedefs, and the `CK_FUNCTION_LIST` struct. Nothing in here executes; it is pure shape, and that shape is the agreement with the host. + +**`api/*` (the entry points).** One file per group of `C_*` functions. Every entry point follows the same skeleton: acquire the instance under the lock, validate the session handle and arguments, do the work or dispatch into the crypto layer, set the in/out length, return the precise `CKR_*` code. These functions are where the spec's rules live (two-call length queries, the operation state machine, login gating). + +**`core/state.zig` (the spine).** Owns the single global `Instance`: the allocator, the `std.Io` backend, the locking mode, the token, the session table, the object store, the current login role, and the unwrapped master key. It exposes `acquire()` (the only way to touch the instance), the init-args parser, and the generation counter used to make slow operations safe. + +**`core/session.zig`, `core/object_store.zig`, `core/token.zig`.** The three data stores. Sessions hold per-session operation state. The object store holds objects and the persistence codec. The token holds authentication state and the wrapped master key. + +**`crypto/*` (the math).** Stateless or self-contained primitives. Everything except RSA is pure-Zig `std.crypto`. RSA is `libcrypto` reached through hand-written `extern` declarations in `openssl.zig`. + +## Data Flow + +### Signing with an RSA key, end to end + +Here is what happens when a host calls `C_Sign` after a `C_SignInit` with an RSA key. It shows every layer. + +``` +1. host: C_SignInit(session, {CKM_SHA256_RSA_PKCS}, hRsaPriv) + api/crypto_ops C_SignInit + state.acquire() -> lock, get the *Instance + sessions.get(hSession) -> the Session, or CKR_SESSION_HANDLE_INVALID + signInitOp(...) -> classify the mechanism as RSA-sign, + parse the scheme/digest params, + fetch+validate the private key object + sess.sign_op = .{ .rsa = { key, params, sig_len } } (just the handle + params) + return CKR_OK (unlock via defer) + +2. host: C_Sign(session, data, dataLen, NULL, &sigLen) (length query) + api/crypto_ops C_Sign + signLen(&sess.sign_op.?) -> modulus length + *pulSignatureLen = that; return CKR_OK + +3. host: C_Sign(session, data, dataLen, sigBuf, &sigLen) (real call) + api/crypto_ops C_Sign, the .rsa arm + rsaPrivateComponents(inst, op.key, CKA_SIGN) -> re-fetch the components, + refuse if sealed or usage-denied + rsa.sign(components, params, data, out) -> crypto layer + openssl.zig: rebuild EVP_PKEY from the components, + EVP_DigestSign (hash-then-sign) for SHA256-RSA-PKCS + *pulSignatureLen = n + sess.endSign() -> zeroize the op union, clear it + return CKR_OK (unlock via defer) +``` + +Two design points stand out. First, the operation state stores only the key *handle* and the parsed parameters, never the key material; the components are re-fetched under the lock for the actual sign. Second, the two-call pattern (NULL buffer to learn the length, then the real buffer) is handled at the top, and a buffer-too-small answer does not consume the operation, exactly as the spec requires. + +### Logging in, end to end + +`C_Login` is the most interesting flow because it does an expensive Argon2id derivation that must not be done while holding the lock, and it has to defend against the token being reinitialized underneath it. + +``` +1. acquire(), validate the session, read the salt/hash and (for User) the wrapped MK +2. gen = cryptoBegin(); grab io + allocator; UNLOCK <- release the lock for the slow part +3. pin.verify(...) -> Argon2id, the expensive part, no lock held + keystore.unwrap(...) -> if User, derive the KEK and unwrap the MK +4. LOCK again; cryptoEnd() + if generation changed since step 2 -> someone reinitialized the token: abort + commit: set logged_in, install the MK, object_store.unlock() to unseal secrets + token.save(...); return CKR_OK +``` + +The `generation` counter is the safety net. `C_InitToken` and `C_InitPIN` bump it. If a second thread reinitialized the token during the long Argon2id call, the committing thread sees the generation has changed and refuses to apply its now-stale result. This is the classic time-of-check-to-time-of-use defense, made explicit. + +## Design Patterns + +### The acquire-and-defer template + +Every fast entry point uses the same shape: + +```zig +pub fn C_SomeOp(hSession: ck.CK_SESSION_HANDLE, ...) callconv(.c) ck.CK_RV { + const inst = state.acquire() orelse return ck.CKR_CRYPTOKI_NOT_INITIALIZED; + defer state.mutex.unlock(); + const sess = inst.sessions.get(hSession) orelse return ck.CKR_SESSION_HANDLE_INVALID; + // ... do the work, return a precise CKR_* ... +} +``` + +`acquire()` locks the mutex and returns the instance only if the library is initialized; if not, it unlocks and returns null so the caller can answer `CKR_CRYPTOKI_NOT_INITIALIZED`. The `defer` guarantees the unlock on every return path. This pattern is used at roughly sixty entry points and is the reason the locking is uniform and hard to get wrong. + +**Why this and not a lock-free instance accessor?** An earlier version exposed a lock-free `current()` that read the instance pointer without the lock. That had a window: `C_Finalize` could tear down the allocator and session table between the present-check and the actual lock. The fix was to make `acquire()` the only accessor and to verify `present` *under* the lock. There is now no way to touch the instance without holding the mutex. + +### The snapshot-unlock-recheck template (for slow operations) + +Argon2id takes real time (it is supposed to). Holding the global lock across it would serialize the whole module behind one login. So slow entry points (`C_Login`, `C_InitToken`, `C_InitPIN`, `C_SetPIN`) snapshot what they need, record the generation, release the lock, do the slow work, then re-lock and verify the generation before committing. This is the only safe way to run a long computation against shared state without freezing the world or risking a stale write. + +### Tagged unions for operation state + +A session can have one active operation of each kind. Each is a Zig tagged union that holds exactly the state that operation needs and nothing more: + +```zig +pub const SignOp = union(enum) { + mac: mac.Mac, // HMAC: the running hash state + ec: ecdsa.SignState, // ECDSA: the curve, scalar, and accumulator + rsa: RsaSig, // RSA: just the key handle + parsed params +}; +``` + +This keeps the operation state inline in the `Session` (no heap allocation for sign/verify/digest), makes "wrong mechanism for this call" a simple `switch` arm, and lets teardown zeroize the whole union in one `secureZero`. The one exception is GCM, which needs a growable buffer and so carries a heap slice (`GcmStream`). + +## Layer Separation + +``` +┌────────────────────────────────────────────────────────┐ +│ Layer 1: ABI façade (ck.zig, api/*) │ +│ - Validates handles, arguments, and the operation FSM │ +│ - Translates between the C ABI and Zig types │ +│ - Does NOT do cryptography or own state │ +└────────────────────────────────────────────────────────┘ + ↓ may call into +┌────────────────────────────────────────────────────────┐ +│ Layer 2: core state (state, session, object_store, │ +│ token, lock, env) │ +│ - Owns the single Instance behind the mutex │ +│ - Knows objects, sessions, login, persistence │ +│ - Does NOT speak the C ABI or pick return codes │ +└────────────────────────────────────────────────────────┘ + ↓ may call into +┌────────────────────────────────────────────────────────┐ +│ Layer 3: crypto (digest, mac, cipher, ecdsa, rsa, │ +│ pin, keystore, openssl) │ +│ - Pure primitives: takes bytes, returns bytes/errors │ +│ - Does NOT know about sessions, handles, or the ABI │ +└────────────────────────────────────────────────────────┘ +``` + +### Why layers + +- **The crypto layer is testable in isolation.** Every file in `crypto/` has unit tests that run without a session or a token, against published vectors (NIST SP800-38A for AES-CBC, RFC 4231 for HMAC, RFC 3394 for key wrap, RFC 6979 for ECDSA). A bug in the math is caught there, not in an integration test. +- **The ABI layer can be cross-checked mechanically.** Because `ck.zig` is pure shape with no logic, `tests/abi_test.zig` can compare it field by field against the translated OASIS headers. +- **Return codes live in exactly one layer.** The crypto layer returns Zig errors; the ABI layer maps them to `CKR_*`. There is one `mapCipherErr`, one `mapSetErr`. A reader looking for "where does `CKR_DATA_LEN_RANGE` come from" has one place to look. + +### What lives where + +- **Layer 1** may import Layer 2 and Layer 3. It owns no mutable state of its own. +- **Layer 2** may import Layer 3. It never imports an `api/*` file (no upward dependency). +- **Layer 3** imports only `std`, `ck` (for constants and types), `config`, and for RSA the `openssl` bindings. It never reaches up. + +## Data Models + +### Object + +An object is a bag of attributes. Each attribute is a type tag, a byte value, and a flag that says whether the value is currently sealed. + +```zig +pub const Attribute = struct { + type: ck.CK_ATTRIBUTE_TYPE, // CKA_* + value: []u8, // owned bytes + sealed: bool = false, // true when the value is GCM ciphertext at rest +}; + +pub const Object = struct { + attrs: std.ArrayList(Attribute), + // get / set / has / getBool / isToken / isPrivate / shouldSeal / clone / deinit +}; +``` + +The `sealed` flag is the in-memory analogue of the at-rest envelope. While the User is logged in, sensitive values are plaintext and `sealed = false`. On logout they are re-encrypted in place and `sealed = true`. A crypto operation that finds a sealed value it needs returns `CKR_USER_NOT_LOGGED_IN`, which covers the case where a sensitive token key was loaded sealed and the user has not logged in to unseal it. + +### Object store + +A fixed array of 256 slots, each optionally holding an entry, with a monotonic handle counter that never reuses a handle. + +```zig +pub const Store = struct { + slots: [config.max_objects]?Entry = @splat(null), + next_handle: ck.CK_OBJECT_HANDLE = 1, // never reused, so stale handles stay invalid +}; +``` + +Handle `0` is reserved as `CK_INVALID_HANDLE`. A host that holds a handle after `C_DestroyObject` and passes it back gets `CKR_OBJECT_HANDLE_INVALID`, not a recycled object, because the counter only ever moves forward. + +### Session + +```zig +pub const Session = struct { + slot: ck.CK_SLOT_ID, + flags: ck.CK_FLAGS, // CKF_RW_SESSION, CKF_SERIAL_SESSION + find: Find, // the active C_FindObjects cursor + digest_op: ?digest.Hasher, // one active op of each kind, at most + sign_op: ?SignOp, + verify_op: ?VerifyOp, + encrypt_op: ?EncryptOp, + decrypt_op: ?DecryptOp, + sign_recover_op: ?RsaRecover, + verify_recover_op: ?RsaRecover, +}; +``` + +The session table is a fixed array of 64 slots. Opening a session scrubs the slot before reuse; closing it frees any heap (the GCM buffer) and zeroizes the slot. The whole table is wiped on `C_Finalize`. + +### Token record + +The token's authentication state is a small fixed struct, serialized to a fixed-size `extern struct` record on disk: + +```zig +pub const Token = struct { + initialized: bool, + label: [32]u8, + so: PinSlot, // SO salt + Argon2id hash + user: ?PinSlot, // User salt + hash, null until C_InitPIN + so_fail: u32, // failure counters for lockout + user_fail: u32, + user_mk: ?keystore.Wrapped, // the master key, wrapped under the User KEK +}; +``` + +`PinSlot` is a salt and a hash; the PIN itself is never present. `user_mk` is the wrapped master key: a salt, a nonce, the ciphertext, and a GCM tag. This is the entire authentication and at-rest-key state, and it fits in a few hundred bytes. + +## Security Architecture + +### Threat model + +What the module is built to resist: + +1. **Disk theft.** An attacker copies the token and object files. Sensitive values are sealed under a master key that is wrapped under the User PIN; the files are ciphertext without the PIN. +2. **Offline PIN attack.** The attacker has the files and runs a dictionary against the PIN. Argon2id (64 MiB, t=3) makes each guess expensive. +3. **Online PIN attack.** The attacker guesses against the live module. Three failures lock the token. +4. **Key export through the API.** A logged-in attacker tries to read or wrap out the key. Sensitive and unextractable refuse. +5. **Tampering at rest.** The attacker flips bytes in a sealed value. The GCM tag fails on unseal and the module fails closed. +6. **Privilege confusion.** An SO tries to read User secrets. There is no SO keyslot for them. +7. **Stale-result race.** A slow login races a token reinit. The generation counter rejects the stale commit. +8. **Memory residue.** A crash dump or cold-boot read tries to recover keys from freed or idle memory. Zeroization and the logout relock shrink that window. + +What is explicitly out of scope: + +- **Physical side channels.** Power analysis, electromagnetic emanation, and microarchitectural attacks like cachebleed are hardware problems. The software constant-time work addresses timing, not power. A real HSM adds physical countermeasures this software cannot. +- **A malicious host while the User is logged in.** PKCS#11's model trusts the calling process during a logged-in session. If the host is compromised while logged in, it can ask the module to sign anything. The module protects the *key*, not the host's intentions. +- **The Windows ABI.** The struct layout is the Linux/macOS LP64 natural-alignment ABI. Windows uses 1-byte packing and a 4-byte `CK_ULONG`, a separate build that is not targeted here. + +### Defense in depth + +``` + On disk: sealed values (AES-256-GCM) + wrapped MK (AES-256-GCM under Argon2id KEK) + ↓ C_Login unseals into RAM + In RAM: plaintext only while User-logged-in; relocked + MK wiped on logout/close/finalize + ↓ a sensitive value needed by an op while sealed + At the API: CKR_USER_NOT_LOGGED_IN; sensitive reads -> CKR_ATTRIBUTE_SENSITIVE + ↓ every comparison of a secret + In compute: constant-time PIN and MAC checks; uniform error codes on RSA decrypt +``` + +## Storage Strategy + +There are two files, both paths resolved from the environment. + +- **The token file** (`ANGELAMOS_HSM_TOKEN`, default `$HOME/.angelamos-hsm-token`). A single fixed-size record holding the auth state and the wrapped master key. Written with a temp-file-then-rename so a crash mid-write cannot leave a half-record. +- **The object file** (`ANGELAMOS_HSM_OBJECTS`, default `$HOME/.angelamos-hsm-objects`). A variable-length record holding only *token* objects (`CKA_TOKEN = CK_TRUE`); session objects are never persisted. Sensitive attribute values are sealed; public values are stored plaintext. + +Both records carry a magic number and a version. The version was bumped when sealing was introduced, so an old plaintext-era file is rejected rather than misread. A corrupt object file parses to empty rather than crashing, so a damaged store degrades to "no objects" instead of undefined behavior. + +This is the one place the design diverges from SoftHSM2, which uses a directory per token and a file per object (or a SQLite database). A single file is simpler and sufficient for an emulator; the per-object directory layout is noted as an extension in [04-CHALLENGES.md](./04-CHALLENGES.md). + +## Configuration + +Every tunable lives in `config.zig`. There are no magic numbers scattered through the code; a value like the GCM buffer cap or the Argon2id memory cost is named once and referenced everywhere. + +```bash +ANGELAMOS_HSM_TOKEN # path to the token record (default: $HOME/.angelamos-hsm-token) +ANGELAMOS_HSM_OBJECTS # path to the object store (default: $HOME/.angelamos-hsm-objects) +``` + +Reading the environment at a C boundary is its own small problem: there is no `std.process` arena to lean on, so `env.zig` walks `std.c.environ` directly. Notable constants in `config.zig`: PIN length 4 to 255, Argon2id t=3 / m=64 MiB / p=1, three login attempts, AES 16 or 32-byte keys, RSA 2048 to 4096 bits, the 16 MiB GCM stream cap, 256 objects, 64 sessions. + +## Performance Considerations + +This is an emulator, so correctness and clarity are weighted above throughput. That said, the design avoids the obvious traps: + +- **The global lock is released across Argon2id.** A login does not freeze every other session for the duration of the KDF. The lock is held only for the quick state reads and the final commit. +- **AES-NI when present.** `std.crypto.core.aes` selects the hardware AES path at compile time when the target has it, so CBC and GCM run on the CPU's AES unit, which is also constant-time by design. +- **Operation state is inline.** Sign, verify, and digest state live in the `Session` struct with no per-operation heap allocation. Only GCM, which must buffer the whole message, touches the heap. + +The bottleneck under load is the single global mutex: this is a serial design, which matches PKCS#11's serial execution model (`C_GetFunctionStatus` and `C_CancelFunction` exist only for the long-dead parallel model and return `CKR_FUNCTION_NOT_PARALLEL`). Sharding by slot is possible but pointless here, since there is one slot. + +## Design Decisions + +### libcrypto for RSA, pure Zig for everything else + +**What we chose:** AES, SHA-2, HMAC, ECDSA, ECDH, and Argon2id come from `std.crypto`. RSA links OpenSSL's `libcrypto`. + +**Alternatives considered:** A pure-Zig RSA. Rejected because `std.crypto` has no public RSA (it exists privately inside the TLS code), and the available third-party Zig RSA libraries are either blind-signature-focused or not safe to trust. Hand-rolling RSA, with its padding schemes and constant-time requirements, is exactly the kind of thing you should not hand-roll. + +**Trade-offs:** A C dependency and a slightly larger build. Mitigated by binding `libcrypto` through hand-written `extern` declarations rather than `@cImport`, and by a version script that exports only `C_GetFunctionList`, so none of OpenSSL's symbols leak out of the module. + +### Selective sealing, not whole-file encryption + +**What we chose:** Only sensitive attribute *values* are encrypted at rest. Public material stays plaintext. + +**Alternatives considered:** Encrypting the entire object file under the master key. Rejected because public objects and public key material are meant to be visible before login (you can read a public key's modulus without a PIN). Encrypting everything would break that and force a login just to list the token. + +**Trade-offs:** The codec is more complex (it has to decide per attribute), but the behavior is spec-correct and the public surface stays usable pre-login. + +### A single User keyslot, no SO keyslot for secrets + +**What we chose:** The master key is wrapped only under the User PIN. The SO can administer and reset, but has no path to the User's secret values. + +**Alternatives considered:** A second keyslot wrapping the MK under the SO PIN, so the SO could recover. Rejected on principle: an SO who can read User keys is a backdoor. The cost is that there is no key recovery if the User PIN is lost, which is the correct trade for a security module. + +### GCM buffered, not streamed through libcrypto + +**What we chose:** AES-GCM accumulates the whole message and runs the authenticated operation once at `*Final`. Covered in [01-CONCEPTS.md](./01-CONCEPTS.md) and [MECHANICS.md](./MECHANICS.md). + +**Alternatives considered:** Using libcrypto's EVP streaming-decrypt. Rejected because it releases unverified plaintext before the tag check. Buffering removes the RUP class of bug entirely, at the cost of a 16 MiB per-message bound. + +## Error Handling Strategy + +Errors come in two flavors and are handled in two places. + +1. **Zig errors in the crypto layer.** A cipher failure, an out-of-memory, a malformed record. These are Zig `error` values. The crypto layer never picks a `CKR_*` code; it returns the error. +2. **`CKR_*` codes at the ABI layer.** Each entry point maps the Zig error to the precise return code the spec wants, through one of a handful of mappers (`mapCipherErr`, `mapSetErr`, the per-call switches). RSA decryption failures collapse to a single code so they cannot be used as a padding oracle. + +The guiding rule is **fail closed**. If a re-seal at logout fails (say, out of memory), the plaintext is scrubbed in place anyway (`scrubUnsealed`) so a failed cleanup never leaves a secret in the clear. If a slow operation cannot prove its result is still current, it returns `CKR_FUNCTION_FAILED` rather than commit stale state. + +## Extensibility + +### Adding a new mechanism + +The mechanism list is data, not code. To add one: + +1. Add the `CKM_*` constant to `ck.zig` (the ABI test will cross-check it against the OASIS header automatically). +2. Add it to `config.supported_mechanisms` and give it a `CK_MECHANISM_INFO` arm in `slot_token.zig`'s `C_GetMechanismInfo`. +3. Implement the primitive in a `crypto/*` file with unit tests against a published vector. +4. Wire it into the relevant `crypto_ops.zig` or `keymgmt.zig` dispatch (the `modeOf` / `hashModeOf` / `isRsa*Mech` helpers). + +The layered design means each step is local: the math lands in Layer 3 with its own tests, the dispatch in Layer 1. + +### Where v3.0 would go + +PKCS#11 v3.0 adds a second entry point, `C_GetInterface`, and 24 functions including the message-based AEAD family. The ABI is laid out so a `CK_FUNCTION_LIST_3_0` (the same prefix plus the new pointers) can be added beside the existing table, and `C_GetInterface` exported alongside `C_GetFunctionList`. This is the largest of the [04-CHALLENGES.md](./04-CHALLENGES.md) extensions. + +## Limitations + +These are conscious trade-offs, not bugs. + +1. **One slot, one token.** A real HSM exposes many slots. Here there is exactly one, always present. Multi-slot would mean sharding the global state per slot. +2. **Single-file storage.** No per-object files, no database. Fine for an emulator; a directory backend is the SoftHSM2 approach and a listed extension. +3. **AES-128 and AES-256 only.** Zig's standard library has no 192-bit AES. Supporting it would mean routing AES-192 through libcrypto like RSA. +4. **No v3.0 surface.** v2.40 only, by scope. The structure leaves room for it. +5. **Linux/macOS ABI.** The Windows packed layout is a separate build. + +## Comparison to Similar Systems + +### SoftHSM2 + +The reference open-source software HSM, and the model this project follows. + +- **Same:** the three-layer façade / store / crypto split, the two-call attribute handling, the per-token PIN with encrypted private-key material, the OpenSSL crypto backend for RSA. +- **Different:** SoftHSM2 stores each object as a file under a per-token directory (or in SQLite); this uses one object file. SoftHSM2 supports multiple tokens and slots; this has one. SoftHSM2 is C++; this is Zig with a machine-checked ABI, which SoftHSM2 does not have. + +### A real hardware HSM (Thales Luna, AWS CloudHSM, YubiHSM) + +- **Same interface.** A host cannot tell the difference at the API level; that is the point of PKCS#11. +- **Different boundary.** A hardware HSM enforces custody with a physical chip and tamper response. This enforces it with process boundaries and software discipline. The software version resists the threats in the model above; it does not resist an attacker with physical access and an oscilloscope. + +## Key Files Reference + +Quick map of where to find things: + +- `src/ck.zig`: the ABI. Start here. +- `src/main.zig`: the exported symbol and the wired table. +- `src/core/state.zig`: the instance, `acquire()`, the generation counter. +- `src/api/session.zig`: login, the snapshot-unlock-recheck pattern in `C_Login`. +- `src/api/crypto_ops.zig`: the whole sign/verify/encrypt/decrypt surface. +- `src/core/object_store.zig`: objects and the sealing codec. +- `src/crypto/keystore.zig`: the envelope (wrap/unwrap, seal/unseal). +- `src/crypto/cipher.zig`: AES modes and RFC 3394 key wrap. + +## Next Steps + +Now that you understand the design: +1. Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) to walk the actual code of the ABI, an end-to-end operation, and the secret-handling patterns. +2. Read [MECHANICS.md](./MECHANICS.md) for the cryptographic details of each mechanism. +3. Try modifying `config.supported_mechanisms` and watch the ABI test and `C_GetMechanismInfo` respond. diff --git a/PROJECTS/advanced/hsm-emulator/learn/03-IMPLEMENTATION.md b/PROJECTS/advanced/hsm-emulator/learn/03-IMPLEMENTATION.md new file mode 100644 index 00000000..68ffcd0b --- /dev/null +++ b/PROJECTS/advanced/hsm-emulator/learn/03-IMPLEMENTATION.md @@ -0,0 +1,510 @@ + + + +# Implementation Guide + +This document walks the actual code. It assumes you have read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md). Code is referenced by file and function name so you can open the file and search for it. Snippets are real, lightly trimmed for focus. + +## File Structure Walkthrough + +``` +src/ +├── ck.zig # the hand-written ABI: types, constants, structs, the function list +├── main.zig # exports C_GetFunctionList, wires the 68-slot table, comptime asserts +├── config.zig # every constant: sizes, KDF params, the mechanism list +├── util.zig # padded(): space-pad a fixed C string field at comptime +├── core/ +│ ├── state.zig # the global Instance, acquire(), the generation counter, finalize() +│ ├── lock.zig # the mutex +│ ├── env.zig # storage-path resolution from std.c.environ +│ ├── token.zig # the token record + atomic save/load +│ ├── session.zig # the session Table, op-state unions, the GcmStream buffer +│ └── object_store.zig# the Object/Store types + the persistence codec with sealing +├── api/ +│ ├── general.zig # Initialize / Finalize / GetInfo / WaitForSlotEvent +│ ├── slot_token.zig # slot/token/mechanism queries + InitToken / InitPIN / SetPIN +│ ├── session.zig # OpenSession / Login / Logout / Get+SetOperationState +│ ├── object.zig # CreateObject / Find / GetAttributeValue / SetAttributeValue +│ ├── crypto_ops.zig # the digest/sign/verify/encrypt/decrypt/dual surface (the big one) +│ ├── keymgmt.zig # GenerateKey(Pair) / WrapKey / UnwrapKey / DeriveKey +│ └── random.zig # GenerateRandom / SeedRandom +└── crypto/ + ├── openssl.zig # hand-written extern EVP/BN/OSSL_PARAM declarations + ├── pin.zig # Argon2id derive / verify + ├── digest.zig # the SHA-2 Hasher union + serializable state + ├── mac.zig # the HMAC-SHA-2 Mac union + ├── cipher.zig # AES-CBC/CBC-PAD/GCM + RFC 3394 key wrap + ├── ecdsa.zig # P-256/384 keygen, sign, verify, ECDH + ├── rsa.zig # the stateless libcrypto RSA bridge + └── keystore.zig # the envelope: master-key wrap/unwrap, value seal/unseal +``` + +## Building the ABI: the contract with the host + +### The hand-written types + +`ck.zig` is the whole agreement with the host, and it is pure shape. The scalar types map to the Linux LP64 ABI: + +```zig +pub const CK_BYTE = u8; +pub const CK_ULONG = c_ulong; // 8 bytes on LP64 Linux/macOS; this is the #1 ABI hazard +pub const CK_RV = CK_ULONG; +pub const CK_SESSION_HANDLE = CK_ULONG; +pub const CK_OBJECT_HANDLE = CK_ULONG; +``` + +Using `c_ulong` (not `u64`) is deliberate: it is whatever the platform's C `unsigned long` is, which is exactly what the host's headers use. Get this wrong and every struct field after the first `CK_ULONG` is at the wrong offset, and the host silently reads garbage. + +The structs are plain `extern struct` with natural alignment. No `packed`, no `align(1)`: + +```zig +pub const CK_ATTRIBUTE = extern struct { + type: CK_ATTRIBUTE_TYPE, + pValue: ?*anyopaque, + ulValueLen: CK_ULONG, +}; +``` + +The spec text says structures are 1-byte packed, but on Linux and macOS the real headers set no packing pragma and use natural alignment. That is the de-facto ABI every Linux host actually uses, so `extern struct` is correct and `packed` would be wrong. + +### The function list and the one exported symbol + +The heart of the ABI is `CK_FUNCTION_LIST`: a version followed by 68 function pointers in a fixed order that you cannot reorder. `main.zig` builds one static instance of it and exports the single symbol that hands out its address: + +```zig +export fn C_GetFunctionList(ppFunctionList: *?*ck.CK_FUNCTION_LIST) callconv(.c) ck.CK_RV { + ppFunctionList.* = &function_list; + return ck.CKR_OK; +} + +var function_list: ck.CK_FUNCTION_LIST = .{ + .version = ck.CK_VERSION{ .major = 2, .minor = 40 }, + .C_Initialize = general.C_Initialize, + .C_Finalize = general.C_Finalize, + // ... every one of the 68 slots, in canonical order ... + .C_WaitForSlotEvent = general.C_WaitForSlotEvent, +}; +``` + +Every slot points at a real `callconv(.c)` function. None is null, because a host calls through these pointers without checking, and a null slot segfaults inside the host. Operations that are deliberately unsupported point at a real function that returns the right `CKR_*` code, never at null. + +### Proving the layout at compile time + +`main.zig` will not compile if the table is the wrong size: + +```zig +comptime { + std.debug.assert(@sizeOf(ck.CK_FUNCTION_LIST) == 69 * @sizeOf(usize)); + std.debug.assert(@sizeOf(ck.CK_ATTRIBUTE) == 24); +} +``` + +The table is the version plus 68 pointers, so 69 pointer-widths. `CK_ATTRIBUTE` is three eight-byte fields, so 24. These are cheap, but the real proof is the build-time cross-check against the OASIS headers, covered later. + +### The version script + +`pkcs11.map` is what keeps everything else hidden: + +``` +PKCS11_2_40 { + global: + C_GetFunctionList; + local: + *; +}; +``` + +`build.zig` applies it with `lib.setVersionScript`. Everything except `C_GetFunctionList` is `local`, so `objdump -T zig-out/lib/libhsm.so` shows exactly one exported symbol. The other `C_*` functions are reachable only through the table, and none of libcrypto's symbols leak. + +## Building an entry point: the acquire-and-defer template + +Every fast entry point has the same skeleton. `C_GetSessionInfo` from `api/session.zig` is a clean example: + +```zig +pub fn C_GetSessionInfo(hSession: ck.CK_SESSION_HANDLE, pInfo: *ck.CK_SESSION_INFO) callconv(.c) ck.CK_RV { + const inst = state.acquire() orelse return ck.CKR_CRYPTOKI_NOT_INITIALIZED; + defer state.mutex.unlock(); + const s = inst.sessions.get(hSession) orelse return ck.CKR_SESSION_HANDLE_INVALID; + pInfo.* = .{ + .slotID = s.slot, + .state = sessionState(s.flags, inst.logged_in), + .flags = s.flags, + .ulDeviceError = 0, + }; + return ck.CKR_OK; +} +``` + +`state.acquire()` is the only way to touch the instance. Its implementation locks the mutex, verifies the library is initialized *under the lock*, and unlocks-and-returns-null if not: + +```zig +pub fn acquire() ?*Instance { + mutex.lock(); + if (!@atomicLoad(bool, &present, .acquire)) { + mutex.unlock(); + return null; + } + return &storage; +} +``` + +The `defer state.mutex.unlock()` in the caller guarantees the lock is released on every path, including the early `CKR_SESSION_HANDLE_INVALID` return. This is why there is no lock-leak: you cannot forget the unlock because `defer` runs it for you. + +### The two-call length pattern + +Variable-length outputs use the spec's two-call dance: pass a null buffer to learn the size, then pass a real buffer. Here is the encrypt path from `crypto_ops.zig`, the AES arm: + +```zig +const need: ck.CK_ULONG = @intCast(cipher.encryptOutLen(c.mode, in.len)); +if (pEncryptedData == null) { + pulEncryptedDataLen.* = need; // first call: report the size + return ck.CKR_OK; +} +if (pulEncryptedDataLen.* < need) { + pulEncryptedDataLen.* = need; // buffer too small: report size again + return ck.CKR_BUFFER_TOO_SMALL; // and do NOT consume the operation +} +``` + +The important subtlety is that a `CKR_BUFFER_TOO_SMALL` does not tear down the operation. The host is expected to retry with a bigger buffer, and the operation state must still be there when it does. A successful single-shot call, on the other hand, *does* end the operation (you will see `sess.endEncrypt(...)` after the bytes are written). + +## Logging in: the snapshot-unlock-recheck pattern + +`C_Login` in `api/session.zig` is the most careful function in the codebase, because it does an expensive Argon2id derivation that must not run while holding the global lock, and it must not commit a stale result if the token was reinitialized during that derivation. + +Step one: acquire, validate, and read out what the slow part needs. Then record the generation, grab the io and allocator, and release the lock: + +```zig +const gen = state.cryptoBegin(); // returns the current generation, marks an op in flight +const io = inst.io(); +const allocator = inst.allocator(); +state.mutex.unlock(); // release the lock for the slow part +defer std.crypto.secureZero(u8, &hash); +``` + +Step two: the slow work, with no lock held. Verify the PIN, and if this is a User login, unwrap the master key: + +```zig +const ok = pin.verify(io, allocator, pinSlice(pPin, ulPinLen), &salt, &hash) catch { + state.cryptoAbort(); + return ck.CKR_FUNCTION_FAILED; +}; +// ... if ok and userType == CKU_USER, keystore.unwrap(...) the master key ... +``` + +Step three: re-lock, and refuse if the world changed underneath us: + +```zig +state.mutex.lock(); +defer state.mutex.unlock(); +state.cryptoEnd(); +if (state.currentGeneration() != gen) return ck.CKR_FUNCTION_FAILED; // token was reinitialized +if (inst.logged_in != null) return ck.CKR_USER_ALREADY_LOGGED_IN; +if (ok) { + inst.logged_in = userType; + if (have_mk) { + inst.mk = mk; + object_store.unlock(allocator, &inst.objects, mk) catch { + inst.relock(); + inst.logged_in = null; + return ck.CKR_FUNCTION_FAILED; + }; + } + token.save(inst.io(), inst.token) catch {}; + return ck.CKR_OK; +} +``` + +The `generation` check is the time-of-check-to-time-of-use defense made explicit. `C_InitToken` and `C_InitPIN` call `state.bumpGeneration()`. If one of them ran while this login was deriving Argon2id, the committing thread sees `currentGeneration() != gen` and throws its stale result away rather than logging into a token that no longer exists in the form it checked. + +On success, `object_store.unlock` walks every sealed sensitive value and decrypts it in place with the now-unwrapped master key. If that fails (a corrupt sealed value), the function relocks and refuses the login, so a damaged store cannot leave you half-unsealed. + +## The object store and the sealing codec + +### Objects are attribute bags + +`object_store.zig` models an object as a list of attributes, each with a `sealed` flag. `set` replaces or appends, securely freeing the old value and clearing any stale sealed flag: + +```zig +pub fn set(self: *Object, allocator: std.mem.Allocator, t: ck.CK_ATTRIBUTE_TYPE, bytes: []const u8) !void { + if (bytes.len > config.max_attr_value_len) return error.AttrTooLarge; + if (self.findPtr(t)) |a| { + const dup = try allocator.dupe(u8, bytes); + secureFree(allocator, a.value); // scrub the old value before freeing + a.value = dup; + a.sealed = false; // a fresh plaintext value is not sealed + return; + } + // ... append a new attribute ... +} +``` + +### What gets sealed + +`shouldSeal` decides whether a given attribute on a given object is secret material that must be encrypted at rest. It is secret if the type is one of the key-material attributes (the AES/HMAC value, or the RSA private exponent and CRT factors) and the object is marked sensitive or unextractable: + +```zig +pub fn shouldSeal(self: *const Object, t: ck.CK_ATTRIBUTE_TYPE) bool { + if (!isSecretMaterial(t)) return false; + if (self.getBool(ck.CKA_SENSITIVE)) return true; + return self.has(ck.CKA_EXTRACTABLE) and !self.getBool(ck.CKA_EXTRACTABLE); +} +``` + +A public value (a modulus, an EC point, a label) returns false and is stored plaintext, so it stays readable before login. + +### The codec: seal on the way out, mark sealed on the way in + +`serialize` writes only token objects, and seals each sensitive value as it writes it: + +```zig +if (!a.sealed and e.obj.shouldSeal(a.type)) { + const key = mk orelse return error.NoMasterKey; + const scratch = try allocator.alloc(u8, keystore.sealedLen(a.value.len)); + defer allocator.free(scratch); + const wrote = try keystore.seal(io, &key, std.mem.asBytes(&a.type), a.value, scratch); + // append the sealed bytes +} else { + // append the plaintext value +} +``` + +Note `std.mem.asBytes(&a.type)` as the associated data: the attribute's type is bound into the GCM tag, so a sealed `CKA_PRIVATE_EXPONENT` cannot be moved into a slot expecting a `CKA_VALUE` without breaking authentication. + +`parse` reads the values back and, after loading, marks the sensitive ones sealed so they are not treated as usable plaintext until `C_Login` unseals them. The whole load is fail-safe: a bad magic, a bad version, a truncated record, or too many attributes all cause the store to clear to empty rather than crash. + +## The RUP-safe GCM buffer + +`GcmStream` in `session.zig` is the accumulator that makes streaming GCM safe. `append` grows a heap buffer, secure-zeroing the old backing on every realloc, and enforces the 16 MiB cap: + +```zig +pub fn append(self: *GcmStream, allocator: std.mem.Allocator, bytes: []const u8) error{ OutOfMemory, TooLarge }!void { + if (bytes.len == 0) return; + const needed = self.len + bytes.len; + if (needed > config.max_gcm_stream_len) return error.TooLarge; + if (self.buf == null or self.buf.?.len < needed) { + var new_cap: usize = if (self.buf) |b| b.len else 256; + while (new_cap < needed) new_cap *|= 2; + if (new_cap > config.max_gcm_stream_len) new_cap = config.max_gcm_stream_len; + const fresh = try allocator.alloc(u8, new_cap); + if (self.buf) |old| { + @memcpy(fresh[0..self.len], old[0..self.len]); + std.crypto.secureZero(u8, old); // scrub the old buffer, it held plaintext + allocator.free(old); + } + self.buf = fresh; + } + @memcpy(self.buf.?[self.len..][0..bytes.len], bytes); + self.len += bytes.len; +} +``` + +In `crypto_ops.zig`, `C_EncryptUpdate` and `C_DecryptUpdate` for the GCM arm append and report zero bytes written. The real GCM call happens in `C_*Final`, where for decrypt the tag is verified before any plaintext is returned. The smoke harness drives a multi-block message through `C_DecryptUpdate` in 19-byte chunks and asserts every update emits zero, then gets the plaintext only at `C_DecryptFinal`. + +## The RSA bridge to libcrypto + +`rsa.zig` is stateless. It never holds an `EVP_PKEY` across calls; it rebuilds one from the PKCS#11-native components each time. `generate` extracts the components out of a freshly generated key: + +```zig +pub fn generate(bits: u32) Error!Generated { + const ctx = ossl.EVP_PKEY_CTX_new_id(ossl.pkey_rsa, null) orelse return Error.Crypto; + defer ossl.EVP_PKEY_CTX_free(ctx); + if (ossl.EVP_PKEY_keygen_init(ctx) <= 0) return Error.Crypto; + if (ossl.EVP_PKEY_CTX_set_rsa_keygen_bits(ctx, @intCast(bits)) <= 0) return Error.Crypto; + var pkey: ?*ossl.EVP_PKEY = null; + if (ossl.EVP_PKEY_generate(ctx, &pkey) <= 0) return Error.Crypto; + defer ossl.EVP_PKEY_free(pkey); + // extract n, e, d, p, q, dmp1, dmq1, iqmp into a Generated struct +} +``` + +`sign` rebuilds the key from the stored components, then either hash-then-signs (for `CKM_SHA256_RSA_PKCS` and the PSS variants) or signs a pre-hashed value directly (`CKM_RSA_PKCS` raw). The components are passed in fresh each call; the operation state in the session held only the handle. + +The bindings in `openssl.zig` are hand-written `extern fn` declarations, not `@cImport`. This is what keeps the production `.so` clean: there is no translated header pulling symbols in, and the version script exports nothing but `C_GetFunctionList`. One detail worth seeing is the use of `BN_clear_free` rather than `BN_free` for the private BIGNUMs: + +```zig +pub extern fn BN_clear_free(a: ?*BIGNUM) void; // zeroes the BIGNUM before freeing +``` + +`BN_free` does not zero the memory; `BN_clear_free` does. The private components (d, p, q, the CRT values) are freed with `BN_clear_free` so they are not left in freed heap. + +## Secret-handling patterns + +The same handful of patterns appear everywhere a secret lives. + +**Stack secrets: defer secureZero.** A derived hash, a master key on the stack, a decrypted buffer: + +```zig +var hash: pin.Hash = undefined; +defer std.crypto.secureZero(u8, &hash); +``` + +`secureZero` takes a `[]volatile` slice, which forbids the compiler from deleting the write as dead store. A plain `@memset(&hash, 0)` with no later read can be optimized away; `secureZero` cannot. + +**Heap secrets: secureFree.** Every attribute value is freed through one function that scrubs first: + +```zig +fn secureFree(allocator: std.mem.Allocator, value: []u8) void { + std.crypto.secureZero(u8, value); + allocator.free(value); +} +``` + +**Operation state: zeroize on teardown.** The session op unions zero themselves when an operation ends. `endSign`, `endEncrypt`, and friends call into the union's `zeroize`/`deinit` (which `secureZero`s the whole union and frees any heap), then null the field. The session `Table` scrubs slots on open, close, and a full `wipeAll` at `C_Finalize`. + +**Logout: relock and wipe.** `inst.relock()` re-seals every sensitive value and wipes the master key: + +```zig +pub fn relock(self: *Instance) void { + if (self.mk) |*mk| object_store.lock(self.io(), self.allocator(), &self.objects, mk.*) catch { + object_store.scrubUnsealed(&self.objects); // if re-seal fails, scrub in place: fail closed + }; + self.wipeMasterKey(); +} +``` + +If the re-seal fails (for example, out of memory), `scrubUnsealed` zeroes the plaintext values in place and marks them sealed, so a failed cleanup never leaves a secret in the clear. This is the fail-closed principle in code. + +## The ABI cross-check + +`tests/abi_test.zig` is what turns "the ABI matches the spec" from a hope into a build-time invariant. `build.zig` runs `addTranslateC` on a shim that includes the vendored OASIS headers, producing a `p11c` module. The test then compares the hand-written `ck.zig` against it. + +Layout equality, field by field: + +```zig +test "hand-coded structs match OASIS-translated layout byte-for-byte" { + try expectSameLayout(ck.CK_ATTRIBUTE, p11c.CK_ATTRIBUTE); + try expectSameLayout(ck.CK_FUNCTION_LIST, p11c.CK_FUNCTION_LIST); + // ... every struct ... +} +``` + +Constant equality, every constant that exists in both: + +```zig +test "every hand-coded constant equals its OASIS value" { + inline for (@typeInfo(ck).@"struct".decls) |d| { + if (@hasDecl(p11c, d.name)) { + // assert ck. == p11c., error.ConstantMismatch if not + } + } + try std.testing.expect(checked >= 100); // and we checked at least 100 of them +} +``` + +And the per-function C-ABI signatures: every entry in `CK_FUNCTION_LIST` is compared against the translated one for parameter count and parameter sizes/alignments. Add a constant to `ck.zig` that also exists in the OASIS headers and this loop picks it up automatically; get an offset wrong and the build fails with the exact field that diverged. + +## The smoke harness + +`examples/smoke.zig` is not a unit test. It `dlopen`s the *built* `.so` and calls through the function list exactly like an external host, which is the only way to catch export bugs and ABI-shape bugs that in-process tests miss: + +```zig +var lib = try std.DynLib.open(default_module); +const getFunctionList = lib.lookup(GetFunctionList, "C_GetFunctionList") orelse return error.SymbolNotFound; +var list_ptr: ?*ck.CK_FUNCTION_LIST = null; +try check("C_GetFunctionList", getFunctionList(&list_ptr)); +const f = list_ptr orelse return error.NullFunctionList; +``` + +From there it walks a full lifecycle: init the token, log in as SO, init the User PIN, set the PIN, create and find objects, prove a private object is hidden after logout, generate AES/EC/RSA keys, sign and verify (and tamper), encrypt and decrypt, derive an ECDH secret on both sides, wrap and unwrap, run GCM streaming and assert it equals the one-shot, exercise the dual functions, recover with RSA, round-trip operation state, and finally assert every conformance edge (`C_WaitForSlotEvent`, `C_GetFunctionStatus`, `C_SeedRandom`). It prints a summary block at the end so you can see at a glance what passed. + +## Common Implementation Pitfalls + +### Pitfall: a plain `zig build` does not match the shipped module + +**Symptom:** you inspect a zeroized secret in a debugger and see `0xAA` bytes, not zeros, and conclude the zeroization is broken. + +**Cause:** `zig build` with no flags is a Debug build. In Debug and ReleaseSafe, Zig poisons an optional's payload to `0xAA` when you set it to null, and `Allocator.free` memsets freed memory to `0xAA`. So `secureZero`-then-null leaves `0xAA` (the secret is gone, just not zero). + +**Fix:** test for "the secret pattern is gone", not "all bytes are zero". The session tests assert the secret byte is absent, not that the buffer is zero. The shipped artifact is `zig build --release=safe`, where the explicit `secureZero` is what protects you (there is no poison in release builds without the safety check, so the explicit zero matters). + +### Pitfall: comparing secrets with `std.mem.eql` + +**Symptom:** a PIN or MAC check that works functionally but leaks timing. + +**Cause:** `std.mem.eql` returns on the first mismatched byte. + +**Fix:** `std.crypto.timing_safe.eql` for fixed-size arrays (the PIN hash), or the `ctEql` helper in `crypto_ops.zig` for variable-length MACs. Both look at every byte before deciding. + +### Pitfall: leaving a function-list slot null + +**Symptom:** the host segfaults the moment it calls an unsupported function. + +**Cause:** a null pointer in the table. + +**Fix:** every slot points at a real `callconv(.c)` function. Unsupported operations return a `CKR_*` code; they are never null. The `comptime` size assert in `main.zig` and the ABI test guard the shape. + +## Debugging Tips + +### Watch every call with pkcs11-spy + +When a host call misbehaves, the fastest way to see what it actually sent is `pkcs11-spy`, which sits in front of your module and logs every call: + +```bash +export PKCS11SPY=$PWD/zig-out/lib/libhsm.so +export PKCS11SPY_OUTPUT=/tmp/spy.log +pkcs11-tool --module /usr/lib/x86_64-linux-gnu/pkcs11/pkcs11-spy.so -T +``` + +The `just spy` recipe wraps this. The log shows each `C_*` call, its arguments, and its return code, so you can find the exact call that returned the wrong thing and with what inputs. + +### Read the return code, then read the spec + +PKCS#11 return codes are specific. `CKR_OPERATION_NOT_INITIALIZED` means a `C_Sign` came without a `C_SignInit`. `CKR_OPERATION_ACTIVE` means a second `C_SignInit` while one was live. `CKR_ATTRIBUTE_SENSITIVE` means you asked for a sealed value. When a host fails, the code it got usually names the bug. + +### Confirm the export surface + +If a host cannot load the module at all, check that the one symbol is there and nothing else leaked: + +```bash +objdump -T zig-out/lib/libhsm.so | grep ' g ' # should show only C_GetFunctionList +``` + +## Code Organization Principles + +### Why one file per function group + +The `api/*` split mirrors the spec's own grouping (slot/token, session, object, crypto, key management). A reader looking for "how does login work" opens `api/session.zig`; "how does signing work" opens `api/crypto_ops.zig`. Each file imports the core and crypto layers it needs and nothing it does not. + +### Why the crypto layer never picks a return code + +Crypto functions return Zig errors (`error.Crypto`, `cipher.Error.DataLenRange`). The ABI layer maps them to `CKR_*`. This keeps the crypto layer independently testable (a test asserts `error.AuthFailed`, not `CKR_*`) and keeps return-code policy in one layer. There is one `mapCipherErr`, one `mapSetErr`. + +## Extending the Code + +### Adding a mechanism, concretely + +Say you want to add HMAC-SHA-512 (it is already there, but the steps are the same for any new one): + +1. **Constant.** Add `CKM_FOO` to `ck.zig`. If the OASIS headers define it, the ABI test cross-checks the value for free. +2. **Advertise it.** Add `CKM_FOO` to `config.supported_mechanisms` and give it an arm in `slot_token.zig`'s `C_GetMechanismInfo` with the right key-size bounds and flags. +3. **Implement.** Add the primitive in the relevant `crypto/*` file. Write a unit test against a published vector and add the file to `test_all.zig`. +4. **Dispatch.** Wire it into the classifier helper it belongs to (`mac.macLenOf`, `cipher.modeOf`, `ecdsa.hashModeOf`, or the RSA `isRsa*Mech` checks) so the `*Init` functions route to it. + +### Adding an entry point + +Copy the acquire-and-defer skeleton from any `api/*` function. Validate the session handle first, then the arguments, then do the work, then set the in/out lengths and return the precise code. If the operation is slow (a KDF, a keygen that should not hold the lock), use the snapshot-unlock-recheck pattern from `C_Login`. + +## Code Style + +The project follows the repository conventions: a file header comment and no inline comments elsewhere, every constant in `config.zig` (no magic numbers in the logic), and `zig fmt` clean. Run the formatter check and the full suite together: + +```bash +just ci # zig fmt --check + zig build test + zig build smoke +``` + +## Build and Test + +```bash +zig build --release=safe # the shipped artifact +zig build test # ABI cross-check + the crypto/core unit suite +zig build smoke # dlopen the built .so and run the full lifecycle +``` + +`zig build test` runs two test binaries: `tests/abi_test.zig` (the OASIS cross-check) and `src/test_all.zig` (which pulls in every `crypto/*` and `core/*` test file). The crypto tests run against published vectors, so a regression in the math is caught at the unit level, and the ABI test catches any layout drift at the same step. + +## Next Steps + +1. Read [MECHANICS.md](./MECHANICS.md) for the cryptographic detail behind the dispatch you just saw: how AES-CBC chains, how PKCS#7 padding is verified, how RFC 3394 wrapping works, how ECDSA and ECDH compute, and how RSA's schemes differ. +2. Read [CONFORMANCE.md](./CONFORMANCE.md) for the exact return code at every deliberate boundary. +3. Open `examples/smoke.zig` and trace one operation from the `f.C_...` call down through the layers you now know. diff --git a/PROJECTS/advanced/hsm-emulator/learn/04-CHALLENGES.md b/PROJECTS/advanced/hsm-emulator/learn/04-CHALLENGES.md new file mode 100644 index 00000000..b0deb179 --- /dev/null +++ b/PROJECTS/advanced/hsm-emulator/learn/04-CHALLENGES.md @@ -0,0 +1,330 @@ + + + +# Extension Challenges + +You have a working software HSM. Now make it yours. These challenges are ordered by difficulty, and each one names the real files and functions you will touch so you are not hunting. Start easy to learn the codebase's shape, then go deeper. + +The golden rule of this project applies to every challenge: nothing is done until an external tool exercises it. Add a unit test against a published vector, then prove it through `pkcs11-tool` or the smoke harness. + +## Easy Challenges + +### Challenge 1: Add SHA-224 as a digest mechanism + +**What to build:** Support `CKM_SHA224` alongside the existing SHA-256/384/512 digests. + +**Why it's useful:** SHA-224 is still required by some compliance profiles, and adding it teaches you the exact path a new mechanism takes through the layers. + +**What you'll learn:** +- The four-step "add a mechanism" path described in [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) +- How the ABI test cross-checks a new constant for free + +**Hints:** +- Add `CKM_SHA224` to `ck.zig`. The "every hand-coded constant equals its OASIS value" test in `tests/abi_test.zig` will check the value against the header automatically. +- Add a `sha224` arm to the `Hasher` union in `crypto/digest.zig` (Zig's std has `sha2.Sha224`), and give it a new state tag for operation-state serialization. +- Add it to `config.supported_mechanisms` and to the digest arm of `C_GetMechanismInfo` in `slot_token.zig`. + +**Test it works:** +```bash +pkcs11-tool --module zig-out/lib/libhsm.so --hash --mechanism SHA224 --input-file msg.bin +``` +Compare against `sha224sum msg.bin`. Add a unit test in `digest.zig` against the `SHA-224("abc")` vector. + +### Challenge 2: Expose a token-info detail you control + +**What to build:** Make the token model string or serial number configurable through an environment variable instead of a fixed constant. + +**Why it's useful:** Real deployments label tokens. It is a gentle introduction to the config and slot-token layers. + +**What you'll learn:** +- How `config.zig` centralizes every constant +- How `env.zig` reads the environment at the C boundary (there is no `std.process` arena here) + +**Hints:** +- `C_GetTokenInfo` in `slot_token.zig` fills the `model` and `serialNumber` fields from `config.token_model` and `config.token_serial` through `util.padded`. +- Add an env lookup in `env.zig` and fall back to the constant when it is unset, mirroring how the storage paths work. +- Remember the fields are space-padded fixed arrays, not NUL-terminated strings. + +**Test it works:** `pkcs11-tool -T` should show your value, and an unset variable should show the default. + +### Challenge 3: Add a `just` recipe that signs and then verifies with OpenSSL + +**What to build:** A recipe that generates an EC key in the module, signs a file, exports the public key, and verifies the signature with the `openssl` command line. + +**Why it's useful:** Cross-verifying against an independent implementation is exactly how this project proves correctness. You are turning that into a one-liner. + +**What you'll learn:** +- How `pkcs11-tool` and `openssl` interoperate through the module +- Why an independent oracle is stronger proof than a self-test + +**Hints:** +- Look at the existing `justfile` recipes and the smoke flow for the command sequence. +- `pkcs11-tool --read-object --type pubkey` exports the public key; `openssl dgst -verify` checks the signature. + +**Test it works:** The recipe exits 0 and prints `Verified OK`. + +## Intermediate Challenges + +### Challenge 4: AES-192 + +**What to build:** Support 192-bit AES keys for `CKM_AES_KEY_GEN`, `CKM_AES_CBC`, `CKM_AES_CBC_PAD`, and `CKM_AES_GCM`. + +**Why it's useful:** It is currently unsupported because Zig's standard library exposes no 192-bit AES (see [CONFORMANCE.md](./CONFORMANCE.md) section 3.3). Solving it teaches you the same libcrypto-bridge pattern RSA uses. + +**Real world application:** Some FIPS profiles and legacy systems require AES-192 specifically. + +**Implementation approach:** +1. Decide the backend. Zig std has no Aes192, so route 192-bit keys through libcrypto's EVP AES, the way `rsa.zig` routes RSA. +2. Add the extern declarations you need to `openssl.zig` (the EVP cipher functions). +3. Branch on key length in `cipher.zig`'s block and GCM functions, calling libcrypto for 24-byte keys and keeping std for 16 and 32. +4. Update `cipher.validKeyLen` and the `CKR_KEY_SIZE_RANGE` checks to accept 24. + +**Hints:** +- The `encBlockRaw` / `decBlockRaw` functions currently `switch (key.len)` over 16 and 32 with `unreachable` otherwise. That switch is where the 24-byte arm goes. +- Keep the constant-time and zeroization patterns identical to the existing arms. + +**Test it works:** A NIST AES-192-CBC vector in `cipher.zig`, plus a `pkcs11-tool` round-trip with a 24-byte key. + +### Challenge 5: A per-object file backend + +**What to build:** Replace the single object file with a directory where each token object is its own file, the way SoftHSM2 does it. + +**Why it's useful:** A single file means rewriting everything on every change. Per-object files let you write only what changed and make backup a per-object copy. + +**What you'll learn:** +- Storage design trade-offs (one file versus many) +- How the persistence codec in `object_store.zig` separates serialization from I/O + +**Implementation approach:** +1. Keep the existing `serialize` / `parse` for a single object's attributes; you are changing only how they are grouped on disk. +2. Give each object a stable on-disk id (the handle is monotonic and never reused, which helps). +3. Write the sealing logic unchanged; each object file still seals its sensitive values under the master key. + +**Hints:** +- The seal-on-write and mark-sealed-on-read logic in `object_store.zig` does not care how many files there are. Keep it. +- `env.zig` resolves a path today; you will resolve a directory. + +**Test it works:** Create several objects, restart the process, and confirm they all reload. Corrupt one object file and confirm only that object is lost, not the whole store. + +## Advanced Challenges + +### Challenge 6: Implement the PKCS#11 v3.0 interface discovery + +**What to build:** Export `C_GetInterface` alongside `C_GetFunctionList`, returning a `CK_FUNCTION_LIST_3_0` for hosts that ask for v3.0. + +**Why this is hard:** You are extending the ABI itself, the part that must match the host byte for byte. A mistake here means hosts crash, not just misbehave. + +**What you'll learn:** +- How v3.0 interface discovery works and why hosts fall back to `C_GetFunctionList` when `C_GetInterface` is absent (the OpenJDK fallback path) +- How `CK_FUNCTION_LIST_3_0` shares a prefix with `CK_FUNCTION_LIST` and appends the new pointers + +**Architecture changes needed:** +``` + C_GetInterface(name, version, &iface, flags) + │ + ▼ + CK_INTERFACE { pInterfaceName, pFunctionList -> CK_FUNCTION_LIST_3_0, flags } + │ + (C_GetFunctionList still returns the v2.40 CK_FUNCTION_LIST prefix) +``` + +**Implementation steps:** +1. Add `CK_FUNCTION_LIST_3_0` and `CK_INTERFACE` to `ck.zig`, with the 24 new v3.0 function-pointer typedefs. +2. Build one larger table and hand out a pointer typed either way, setting the version field correctly per caller. +3. Export `C_GetInterface` in `main.zig` and add it to the version script in `pkcs11.map`. +4. Extend `tests/abi_test.zig` to cross-check the new structs against the v3.0 OASIS headers (vendor them alongside the v2.40 ones). + +**Gotchas:** +- The first N members of `CK_FUNCTION_LIST_3_0` must be byte-identical to `CK_FUNCTION_LIST`. The cross-check test is your safety net. +- New v3.0 functions you do not implement still need real, correctly-typed stub functions in the table. Never null. + +**Resources:** The PKCS#11 v3.1 specification, and the `08-pkcs11.md` research note in `docs/zig/reference/` which covers the v3.0 surface. + +### Challenge 7: The v3.0 message-based AEAD API + +**What to build:** Implement `C_EncryptMessage` / `C_DecryptMessage` (and the begin/next/final variants) for AES-GCM. + +**Why this is hard:** The message API is how the OpenSSL 3.x provider increasingly prefers to do GCM, and it has a different nonce-management contract than the classic `C_Encrypt`. + +**What you'll learn:** +- The v3.0 message API shape and how it differs from the v2.40 single-shot and update/final flows +- How to keep your RUP-safe stance under the new API + +**Implementation approach:** +1. Requires Challenge 6 first (the v3.0 table). +2. The message API passes the nonce and AAD through `CK_MESSAGE`-family parameters rather than the mechanism parameter. +3. Keep the buffered, verify-before-release behavior for decrypt that the classic path already has in `GcmStream`. + +**Hints:** Your existing GCM accumulator and the strict parameter validation in `buildCipher` are most of the work; the new part is the message-API parameter parsing. + +### Challenge 8: Fork safety + +**What to build:** Detect a `fork()` and refuse to operate in the child with stale state inherited from the parent, or re-initialize cleanly. + +**Why this is hard:** A child process inherits the parent's initialized state, including an unwrapped master key, which is a real hazard. The PKCS#11 answer is the `CKF_INTERFACE_FORK_SAFE` flag, but only if you actually handle it. + +**What you'll learn:** +- Why inherited crypto state across a fork is dangerous +- How to track the process id and invalidate state on change + +**Implementation approach:** +1. Record the pid in the `Instance` at `C_Initialize`. +2. In `state.acquire()`, compare the current pid; on a mismatch, treat the library as not initialized in the child (force a re-init) rather than operating on the parent's secrets. +3. Only then advertise `CKF_INTERFACE_FORK_SAFE` (a v3.0 interface flag, so this pairs with Challenge 6). + +**Gotchas:** The master key inherited by a child must be wiped, not used. Be careful that the detection happens before any operation can touch `inst.mk`. + +## Expert Challenges + +### Challenge 9: A SQLite storage backend with multi-token support + +**What to build:** Replace the file backend with SQLite, and lift the one-token limit so the module can host several tokens across several slots. + +**Estimated time:** A week or more. + +**Prerequisites:** You should have done Challenge 5 (per-object storage) first, because this builds on separating the codec from the I/O. + +**What you'll learn:** +- How a real software HSM (SoftHSM2 has exactly this) structures multi-token storage +- How to shard the global state per slot without losing the single-lock simplicity, or how to move to per-slot locks + +**Planning this feature:** + +Before you code, think through: +- How does multi-slot change the global `Instance`? Today it is one token, one session table, one object store. You need a collection keyed by slot. +- Does the single global mutex still serve, or do you want per-slot locks? Per-slot locks add real concurrency but also real deadlock risk. +- How do you migrate an existing single-file token into the database on first run? + +**High level architecture:** +``` + slots: [N] of Token + each Token: its own PIN state, master key, object store + storage: SQLite (one table for tokens, one for objects, sealed values as BLOBs) + locking: either the one global mutex (simple) or per-slot (concurrent) +``` + +**Implementation phases:** + +**Phase 1: Multi-token in memory.** Turn the single token/store/sessions in `Instance` into slot-indexed collections. Keep the file backend. Get `C_GetSlotList` returning several slots and the per-slot operations routing correctly. + +**Phase 2: The SQLite backend.** Link SQLite (the libcrypto bridge in `openssl.zig` is your template for binding a C library cleanly). Move serialize/parse to read and write rows. Keep the sealing logic untouched; values are still GCM-sealed BLOBs. + +**Phase 3: Migration and concurrency.** Migrate an existing file token on first open. Decide and implement the locking model. + +**Phase 4: Polish.** Error handling for a locked or corrupt database, and the full test pass through `pkcs11-tool` against multiple tokens. + +**Success criteria:** +- [ ] `C_GetSlotList` returns more than one slot +- [ ] Two tokens hold independent keys and PINs +- [ ] Keys survive a restart, loaded from SQLite +- [ ] A corrupt or locked database fails closed, never returns a wrong key +- [ ] The ABI test and smoke harness still pass unchanged + +## Mix and Match + +Combine challenges into bigger projects: + +**A v3.0-native module.** Challenge 6 (interface) plus Challenge 7 (message API) plus Challenge 8 (fork safety) gives you a module a modern OpenSSL 3.x provider talks to natively. + +**A production-shaped store.** Challenge 5 (per-object) plus Challenge 9 (SQLite, multi-token) gives you something with SoftHSM2's storage capabilities. + +## Real World Integration Challenges + +### Drive the module from the OpenSSL pkcs11 provider + +**The goal:** Use this module as the key store behind OpenSSL itself, so `openssl` commands sign and decrypt through it. + +**What you'll need:** +- The `pkcs11-provider` (the modern OpenSSL 3.x provider) configured to point at `libhsm.so` +- An OpenSSL config that loads the provider + +**Watch out for:** The provider exercises corners `pkcs11-tool` does not, especially around RSA-PSS parameters and GCM. This is a great way to find conformance gaps. Anything it trips is a real bug or a documented narrowing in [CONFORMANCE.md](./CONFORMANCE.md). + +### Use the module from Java via SunPKCS11 + +**The goal:** Have a Java program generate a key and sign through the module using `java.security` and the SunPKCS11 provider. + +**What you'll learn:** SunPKCS11 is the classic `C_GetInterface`-fallback consumer, so this also validates that your `C_GetFunctionList` path is solid (and motivates Challenge 6 if you want the v3.0 path). + +## Performance Challenges + +### Reduce lock contention under concurrent sessions + +**The goal:** Let many sessions do crypto in parallel instead of serializing on the one global mutex. + +**Current bottleneck:** Every entry point takes `state.mutex`. Two sessions signing at once serialize, even though their operations are independent. + +**Approaches:** +- **Per-session operation state, lock only for shared state.** The op-state already lives in the `Session`; you could do the crypto outside the lock and only lock to fetch and commit, the way `C_Login` already does for Argon2id. +- **Per-slot locks.** Pairs with the multi-slot work in Challenge 9. + +**Benchmark it:** Drive N threads each signing in a loop and measure throughput before and after. Watch for races; the generation-counter pattern in `state.zig` is your model for safe lock-release-relock. + +## Security Challenges + +### Prove constant-time behavior with Valgrind + +**The goal:** Use Valgrind's memcheck and Zig's `classify` / `declassify` to detect any branch or memory access that depends on a secret. + +**What you'll learn:** The closest thing to a constant-time verifier Zig has. `classify` marks secret memory as uninitialized to Valgrind, which then flags any conditional that depends on it. + +**Implementation:** Add a debug build that classifies the PIN hash and key material before the crypto, run it under `valgrind --tool=memcheck --track-origins=yes`, and chase down any "conditional jump depends on uninitialised value" reports. The `16-constant-time-security.md` research note in `docs/zig/reference/` walks through the API. + +**Testing the security:** A clean run with no conditional-on-secret warnings through a full sign and decrypt is the goal. + +### Fuzz the persistence codec + +**The goal:** Throw malformed token and object files at `parse` and confirm it always fails closed (clears to empty) and never crashes or reads out of bounds. + +**Threat model:** This protects against a malicious or corrupted store file. The codec already has length and bounds checks; fuzzing proves they are complete. + +**Implementation:** Generate random and mutated record bytes, feed them to `object_store.parse` and `token.loadFrom`, and assert the result is either a valid parse or a clean empty store, never a panic. The existing "rejects a bad magic" and "fails safe on a truncated record" tests are your starting points. + +## Contribution Ideas + +Finished something? Share it back: + +1. Fork the repo. +2. Implement your extension in a branch, with unit tests against a published vector and a smoke-harness or `pkcs11-tool` proof. +3. Document it: update [CONFORMANCE.md](./CONFORMANCE.md) if you changed a boundary, and add a note to the relevant learn doc. +4. Open a PR with the implementation, the tests, and the external-tool proof. + +Good extensions (a new mechanism with vectors, a real storage backend, the v3.0 surface) are exactly the kind of thing that makes the project more useful to the next person. + +## Challenge Yourself Further + +### Build something new + +Use what you learned here to build: +- A **PKCS#11 provider for a different key type**, like Ed25519 or ML-DSA (both are in Zig's std), following the ECDSA path. +- A **minimal host** that loads any PKCS#11 module and exercises it, the consumer side of the interface you just implemented. + +### Study real implementations + +Read these and steal their good ideas: +- **SoftHSMv2** for the per-token directory store and the OpenSSL/Botan crypto abstraction. +- **OpenSC `pkcs11-spy`** for how a transparent shim logs every call, which is a clean piece of ABI engineering in its own right. + +## Getting Help + +Stuck on a challenge? + +1. **Debug systematically.** What did you expect, what happened, what is the smallest input that reproduces it? Run the failing case through `just spy` and read the exact call and return code. +2. **Read the existing code.** The mechanism you are adding almost certainly has a sibling already implemented. The path is the same. +3. **Lean on the cross-check.** If you touched `ck.zig`, run `zig build test`; the ABI test will tell you precisely which field or constant diverged from the OASIS headers. + +## Challenge Completion + +Track your progress: + +- [ ] Easy 1: SHA-224 digest +- [ ] Easy 2: configurable token info +- [ ] Easy 3: OpenSSL cross-verify recipe +- [ ] Intermediate 4: AES-192 via libcrypto +- [ ] Intermediate 5: per-object file backend +- [ ] Advanced 6: v3.0 `C_GetInterface` +- [ ] Advanced 7: v3.0 message API +- [ ] Advanced 8: fork safety +- [ ] Expert 9: SQLite, multi-token + +Finished them all? You have built a multi-token, v3.0-capable software HSM with a real database backend and proven constant-time behavior. At that point you understand key custody at a level most working engineers never reach. Go read a real HSM vendor's PKCS#11 docs; you will recognize every design decision they made and why. diff --git a/PROJECTS/advanced/hsm-emulator/learn/MECHANICS.md b/PROJECTS/advanced/hsm-emulator/learn/MECHANICS.md new file mode 100644 index 00000000..e7f9c13b --- /dev/null +++ b/PROJECTS/advanced/hsm-emulator/learn/MECHANICS.md @@ -0,0 +1,348 @@ + + + +# How the Mechanisms Actually Work + +The [conformance statement](./CONFORMANCE.md) tells you *what* each mechanism does and which return code it gives. This document tells you *how* each one works under the hood, byte by byte, and why the implementation looks the way it does. If [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) is the tour of the plumbing, this is the tour of the water. + +Everything here except RSA is pure-Zig `std.crypto`, tested against published vectors. RSA is libcrypto. Each section names the file you can open to read the real thing. + +## The Short Version + +| Family | Mechanisms | Where the math is | Backed by | +|--------|-----------|-------------------|-----------| +| Digest | SHA-256 / 384 / 512 | `digest.zig` | `std.crypto.hash.sha2` | +| MAC | HMAC-SHA-256 / 384 / 512 | `mac.zig` | `std.crypto.auth.hmac` | +| AES | CBC, CBC-PAD, GCM | `cipher.zig` | `std.crypto.core.aes`, `aead.aes_gcm` | +| Key wrap | AES-KEY-WRAP (RFC 3394) | `cipher.zig` | the AES block on top | +| ECDSA | P-256, P-384, raw and SHA-256 | `ecdsa.zig` | `std.crypto.sign.ecdsa` | +| ECDH | P-256, P-384, `CKD_NULL` | `ecdsa.zig` | `std.crypto.ecc` | +| RSA | PKCS#1 v1.5, PSS, OAEP | `rsa.zig` + `openssl.zig` | OpenSSL `libcrypto` | +| KDF | Argon2id (PIN stretching) | `pin.zig` | `std.crypto.pwhash.argon2` | +| Envelope | AES-256-GCM seal/wrap | `keystore.zig` | `std.crypto.aead.aes_gcm` | + +A theme runs through all of them: the dangerous part of each primitive is handled by a tested standard-library function, and the module's own code is the wiring, the validation, and the constant-time and memory hygiene around it. + +## AES-CBC: chaining blocks + +CBC (Cipher Block Chaining) turns a block cipher, which only encrypts one 16-byte block, into something that encrypts a whole message. The trick is that each plaintext block is XORed with the previous ciphertext block before encryption, so identical plaintext blocks do not produce identical ciphertext. + +``` + P0 P1 P2 + │ │ │ +IV─XOR ┌─►XOR ┌─►XOR (IV seeds the very first XOR) + │ │ │ │ │ + AES │ AES │ AES + │ │ │ │ │ + ▼ │ ▼ │ ▼ + C0─────┘ C1─────┘ C2 +``` + +In `cipher.zig`, one encrypt step is exactly that XOR-then-encrypt, with the chain value updated to the new ciphertext: + +```zig +fn cbcEncStep(self: *Cipher, in16: *const [block]u8, out16: *[block]u8) void { + var x: [block]u8 = undefined; + defer std.crypto.secureZero(u8, &x); // the XOR scratch held plaintext, scrub it + for (0..block) |j| x[j] = in16[j] ^ self.chain[j]; + encBlockRaw(self.key(), &x, out16); + self.chain = out16.*; // chain forward +} +``` + +Decryption reverses it: decrypt the block, then XOR with the previous ciphertext (held in `chain`). The chain advances to the input block, not the output. + +Plain `CKM_AES_CBC` has no padding, so the input must be a whole number of blocks. A partial trailing block is an error (`CKR_DATA_LEN_RANGE`), because there is nothing to do with 5 leftover bytes. The IV arrives in the mechanism parameter and seeds `chain`. + +## AES-CBC-PAD: padding, and the held-back block + +`CKM_AES_CBC_PAD` adds PKCS#7 padding so any length encrypts. PKCS#7 is simple: if you need `n` bytes of padding, append `n` copies of the byte `n`. If the message is already block-aligned, you add a whole block of `0x10` (16) so there is always padding to remove. + +``` + message ...... | pad + "ABCDE" (5) | 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B (11 bytes of 0x0B) + └──────────── 11 = pad length ────────────┘ +``` + +`encryptFinal` writes the padding into the last partial block and encrypts it: + +```zig +const padlen: u8 = @intCast(block - self.partial_len); +for (self.partial_len..block) |j| self.partial[j] = padlen; +self.cbcEncStep(&self.partial, out[0..block]); +``` + +Decryption is where it gets subtle. The decryptor cannot strip padding until it knows it has the *last* block, but in streaming mode it does not know which block is last until the stream ends. So CBC-PAD decrypt **holds back one block**: each time a full block arrives, it emits the *previous* held block and holds the new one. `decryptFinal` decrypts the final held block and strips the padding. This held-back behavior is exactly why CBC-PAD cannot participate in the decrypt side of a dual function (the digest would miss the last block); see [CONFORMANCE.md](./CONFORMANCE.md) section 2.2. + +The padding check is constant-time. A naive check returns as soon as a pad byte is wrong, which leaks where it failed (the Vaudenay padding-oracle attack, and Lucky Thirteen against TLS). This module ORs all the differences first, then decides once: + +```zig +const padlen = pt[block - 1]; +if (padlen == 0 or padlen > block) return Error.EncryptedDataInvalid; +var bad: u8 = 0; +for (0..block) |j| { + const is_pad = j >= block - padlen; + if (is_pad) bad |= pt[j] ^ padlen; // accumulate, do not branch out early +} +if (bad != 0) return Error.EncryptedDataInvalid; +``` + +## AES-GCM: authenticated, and why it is buffered here + +GCM (Galois/Counter Mode) does two things at once: it encrypts with AES in counter mode, and it computes an authentication tag over the ciphertext and the associated data using multiplication in a Galois field. The tag is the whole value of GCM: decryption verifies it, and a wrong tag means the ciphertext was tampered with (or you used the wrong key or nonce). + +``` + nonce(12) + counter ──► AES ──► keystream ──XOR──► ciphertext + │ + AAD + ciphertext ──► GHASH (GF(2^128) mult) ──► tag(16) +``` + +This module uses the standard library's one-shot GCM, with a 12-byte (96-bit) nonce and a 128-bit tag, and up to 256 bytes of associated data: + +```zig +aesgcm.Aes256Gcm.encrypt(out[0..input.len], tag, input, ad, self.iv, self.key_buf[0..32].*); +``` + +The 96-bit nonce is the standard choice: it is used directly as the initial counter without the extra hashing a different length would require. The fixed 128-bit tag is the full-strength tag; truncated tags weaken authentication. + +**Why buffered.** A streaming GCM decrypt that releases plaintext block by block would hand back unverified bytes before it has seen the tag. That is release of unverified plaintext, and it is an anti-pattern for a security module (see [01-CONCEPTS.md](./01-CONCEPTS.md)). This module accumulates the whole message in `GcmStream` and runs the authenticated operation once at `*Final`, where decrypt verifies the tag before returning a single byte. The buffer is capped at 16 MiB so a host cannot exhaust memory. The cost is that GCM is single-message-bounded; the benefit is that RUP is impossible by construction. + +On a bad tag, decryption returns `error.EncryptedDataInvalid` and no plaintext. The standard library zeroes the output buffer internally before returning the error, so failed decryption leaves nothing usable behind. + +## AES Key Wrap (RFC 3394): encrypting a key with a key + +You cannot just AES-encrypt a key and call it wrapped; you want the result to be tamper-evident, so a corrupted wrapped key is *rejected* rather than unwrapped into garbage. RFC 3394 AES Key Wrap does this with a fixed initial value (the integrity check value, ICV) and six passes over the data. + +The ICV is the constant `A6 A6 A6 A6 A6 A6 A6 A6`. Wrapping prepends it, then runs 6 rounds where each 64-bit block of the key is mixed with a running `A` register through the AES block cipher and a counter: + +```zig +var a: [8]u8 = key_wrap_iv; // A6 A6 ... A6 +// for j in 0..6, for each 64-bit block i: +@memcpy(blk[0..8], &a); +@memcpy(blk[8..16], r[i * 8 ..][0..8]); +encBlockRaw(kek, &blk, &enc); // B = AES(A | R[i]) +@memcpy(&a, enc[0..8]); // A = MSB64(B) +xorCounter(&a, n * j + i + 1); // A ^= t (the round counter) +@memcpy(r[i * 8 ..][0..8], enc[8..16]); // R[i] = LSB64(B) +``` + +Unwrapping runs the rounds backward and then checks that the recovered `A` equals the ICV. If a single bit of the wrapped key was flipped, `A` will not match `A6...A6`, and the unwrap fails. The check is done by ORing the differences (not an early-exit compare), and on failure the partial output is scrubbed: + +```zig +var diff: u8 = 0; +for (a, key_wrap_iv) |x, y| diff |= x ^ y; +if (diff != 0) { + std.crypto.secureZero(u8, out[0..plain_len]); + return WrapError.Integrity; // -> CKR_WRAPPED_KEY_INVALID at the API +} +``` + +The implementation is verified against the RFC 3394 section 4.1 known-answer test: wrapping the test key under the test KEK produces the exact published ciphertext, and unwrapping round-trips. The 8-byte ICV is why a wrapped key is 8 bytes longer than the key it wraps. + +## SHA-2 and HMAC: hashing and keyed hashing + +SHA-256, SHA-384, and SHA-512 are the digest mechanisms. The `Hasher` in `digest.zig` is a tagged union over the three, and multi-part hashing (`C_DigestUpdate` repeatedly) just feeds the running state: + +```zig +pub const Hasher = union(enum) { + sha256: sha2.Sha256, + sha384: sha2.Sha384, + sha512: sha2.Sha512, +}; +``` + +A digest is verified against the classic `SHA-256("abc")` vector in the unit tests and again in the smoke harness through the real `.so`. + +The interesting part is **operation-state serialization**. `C_GetOperationState` lets a host snapshot a digest mid-stream and resume it later with `C_SetOperationState`. The module serializes the raw hasher state with a version byte and a type tag: + +``` + [version=1][tag: 1=sha256 2=sha384 3=sha512][raw hasher state bytes] +``` + +```zig +out[0] = config.op_state_version; +out[1] = d.stateTag(); +d.writeState(out[config.op_state_header_len..]); +``` + +This blob is **same-build only**: it carries the in-memory layout of the standard library's hasher, which is not promised stable across builds or implementations. The spec allows exactly this (operation state is not portable), and restore validates the version, the tag, and the exact length, failing closed on anything else. Only digest state is saveable; trying to save a sign or encrypt operation returns `CKR_STATE_UNSAVEABLE`. + +HMAC (`mac.zig`) is the keyed-hash construction `H(key ⊕ opad || H(key ⊕ ipad || message))`, but you do not implement that by hand; the standard library's `HmacSha256` and friends do. The module's job is to verify the tag in constant time, which `crypto_ops.zig` does with `ctEql`. HMAC-SHA-256 is checked against RFC 4231 test case 2. + +## ECDSA: signing on a curve + +ECDSA signs over an elliptic curve. This module supports NIST P-256 and P-384, with two mechanisms: `CKM_ECDSA` (you supply an already-hashed value) and `CKM_ECDSA_SHA256` (the module hashes the message with SHA-256 first). + +### The curve and the key encoding + +The curve is identified by an OID in DER, stored in `CKA_EC_PARAMS`. P-256 is `06 08 2A 86 48 CE 3D 03 01 07`, P-384 is `06 05 2B 81 04 00 22`. `curveFromParams` matches the bytes: + +```zig +pub fn curveFromParams(ec_params: []const u8) ?Curve { + if (std.mem.eql(u8, ec_params, &oid_p256)) return .p256; + if (std.mem.eql(u8, ec_params, &oid_p384)) return .p384; + return null; +} +``` + +The public key is a point in SEC1 uncompressed form: a `0x04` byte followed by the X and Y coordinates. For P-256 that is `1 + 32 + 32 = 65` bytes; the `CKA_EC_POINT` attribute wraps it in a DER `OCTET STRING`, so the stored value is 67 bytes (the `0x04 0x41` DER header plus the 65-byte point). `wrapEcPoint` and `unwrapEcPoint` handle that wrapping, including the long-form DER length encoding for P-384's larger point. + +### Prehash, reduce, sign + +The message (or its hash) is reduced to a scalar by taking the leftmost bytes equal to the curve's order size, which is what ECDSA does with a hash that may be a different width than the curve: + +```zig +fn reduce(curve: Curve, dgst: []const u8, out: *[max_scalar]u8) []const u8 { + const n = curve.scalarLen(); + if (dgst.len >= n) @memcpy(out[0..n], dgst[0..n]) // take the leftmost n bytes + else // left-pad with zeros +} +``` + +The actual signing uses `signPrehashed`. One subtlety worth knowing: Zig's deterministic ECDSA is not plain RFC 6979. It mixes in a noise field, so the signature is not a fixed function of the message and key. That means you cannot pin a sign-output known-answer test (two signatures of the same message differ), so the test suite uses an RFC 6979 vector as a *verify* known-answer test instead, and round-trips its own signatures. The noise comes from `randomSecure`, with a fallback to deterministic if entropy is unavailable: + +```zig +const nz: ?[slen]u8 = if (io.randomSecure(&noise)) |_| noise else |_| null; +const sig = kp.signPrehashed(ph, nz) catch return Error.Crypto; +``` + +A P-256 signature is 64 bytes (r and s, 32 each); P-384 is 96. Verification reconstructs the public key from the SEC1 point, validates the point is on the curve, and checks the signature. The smoke harness signs over the module, verifies, then flips a byte and confirms `CKR_SIGNATURE_INVALID`. + +## ECDH: agreeing on a shared secret + +ECDH (`CKM_ECDH1_DERIVE`) lets two parties with EC keypairs compute the same shared secret without ever transmitting it. Each multiplies their own private scalar by the other's public point; the math guarantees both arrive at the same point, and the secret is that point's X coordinate. + +``` + Alice: secret = X-coordinate of (alice_private * bob_public) + Bob: secret = X-coordinate of (bob_private * alice_public) + alice_private * bob_public == bob_private * alice_public (same point) +``` + +The implementation multiplies the peer point by the scalar and takes the affine X coordinate: + +```zig +const peer = Pt.fromSec1(peer_point_sec1) catch return Error.Crypto; // validates on-curve +const shared = peer.mul(s, .big) catch return Error.Crypto; +const xb = shared.affineCoordinates().x.toBytes(.big); +@memcpy(out[0..n], xb[0..n]); +``` + +`fromSec1` rejects a point that is not on the curve, which is the defense against invalid-curve attacks (feeding a carefully chosen off-curve point to leak the private scalar). The module supports only the `CKD_NULL` key-derivation function, meaning the raw shared X coordinate is the derived key with no further KDF, and it accepts the peer point either as raw SEC1 or DER-wrapped. The derived value is correct to the bit: the smoke harness derives on both sides and asserts they match, and a cross-process test confirms it equals `openssl pkeyutl -derive` byte for byte. The smoke harness even derives one side with a raw peer point and the other with a DER-wrapped one and confirms both agree. + +## RSA: three schemes, one modular exponentiation + +RSA is the one family this module does not implement itself. Zig's standard library has no public RSA, so `rsa.zig` calls OpenSSL's `libcrypto` through the hand-written `extern` declarations in `openssl.zig`. Every RSA operation is a single modular exponentiation over the whole input, so RSA has no multi-part form (no `C_SignUpdate`); you use the one-shot calls. + +Keys are stored PKCS#11-native: the modulus and public exponent are public, and the private side keeps the private exponent plus the four CRT (Chinese Remainder Theorem) values that make private operations about four times faster. The public exponent is fixed at 65537 (the F4 prime), the universal default. `rsa.zig` is stateless: it rebuilds an `EVP_PKEY` from these components on every call rather than holding one. + +The three schemes differ in how they pad before the exponentiation: + +**PKCS#1 v1.5** (`CKM_RSA_PKCS`, `CKM_SHA256_RSA_PKCS`). The classic padding. For signing, the message (or its hash, wrapped in a DigestInfo) is padded with `00 01 FF FF ... FF 00` up to the modulus size. Simple and everywhere, but its decryption form is the one vulnerable to the Bleichenbacher oracle, which is why decryption failures here collapse to one uniform return code. + +**PSS** (`CKM_RSA_PKCS_PSS`, `CKM_SHA256_RSA_PKCS_PSS`). Probabilistic Signature Scheme. It mixes a random salt into the padding, so signing the same message twice gives different signatures, and it has a security proof PKCS#1 v1.5 lacks. The module requires the MGF hash to equal the content hash and rejects mismatches: + +```zig +if (mgfHash(pp.mgf) != h) return .{ .err = ck.CKR_MECHANISM_PARAM_INVALID }; +``` + +**OAEP** (`CKM_RSA_PKCS_OAEP`). Optimal Asymmetric Encryption Padding, for encryption and key wrapping. It is the encryption counterpart to PSS: randomized, with a proof, and the recommended replacement for PKCS#1 v1.5 encryption precisely because it resists Bleichenbacher. + +The split between hash-then-sign and raw is visible in `sign`. When a digest is specified, it uses `EVP_DigestSign` (which hashes then signs); when the input is already a hash, it uses `EVP_PKEY_sign` directly: + +```zig +if (p.digest != .none) { + // EVP_DigestSignInit + EVP_DigestSign: hash the message, then sign +} else { + // EVP_PKEY_sign: sign the pre-hashed value as-is +} +``` + +Sign/VerifyRecover (`C_SignRecover` / `C_VerifyRecover`) use PKCS#1 v1.5's property that the message is recoverable from the signature. The module offers recover only for `CKM_RSA_PKCS`, and it checks the data fits the modulus with the 11-byte v1.5 overhead before signing. + +## Argon2id: turning a weak PIN into a strong key + +A PIN is low-entropy, so the stored form has to be expensive to attack and the derived KEK has to be strong. Argon2id, the winner of the Password Hashing Competition, is memory-hard: each guess must allocate and traverse a large block of memory, which defeats the massive parallelism a GPU or ASIC brings to a fast hash. + +The parameters in `config.zig` are `t = 3` iterations, `m = 64 MiB`, `p = 1` lane. Each PIN guess costs 64 MiB of memory and three passes over it. `pin.zig` derives the 32-byte output from the PIN and a random 16-byte salt: + +```zig +pub fn derive(io: std.Io, allocator: std.mem.Allocator, pin: []const u8, salt: *const Salt, out: *Hash) !void { + try argon2.kdf(allocator, out, pin, salt, params, .argon2id, io); +} +``` + +The same derivation does double duty. For authentication, the output is compared (constant-time) against the stored hash. For the envelope, the output is the key-encryption key that wraps the master key. The `id` variant blends the data-independent and data-dependent modes, giving resistance to both side-channel and time-memory-tradeoff attacks. + +## The Envelope: sealing values and wrapping the master key + +The at-rest scheme has two GCM operations, both in `keystore.zig`. + +**Wrapping the master key.** The master key (32 random bytes) is encrypted under the KEK derived from the PIN. The wrapped form is salt, nonce, ciphertext, and tag, stored in the token record. A wrong PIN derives a wrong KEK, the GCM tag fails, and `unwrap` returns false rather than a bogus key: + +```zig +gcm.decrypt(out, &w.ct, w.tag, "", w.nonce, kek) catch { + std.crypto.secureZero(u8, out); + return false; +}; +``` + +**Sealing a value.** Each sensitive attribute value is GCM-encrypted under the master key, with the attribute *type* as associated data, and a fresh nonce per seal: + +```zig +pub fn seal(io: std.Io, mk: *const MasterKey, ad: []const u8, plain: []const u8, out: []u8) !usize { + var nonce: [nonce_len]u8 = undefined; + try io.randomSecure(&nonce); // fresh nonce every time + @memcpy(out[0..nonce_len], &nonce); + const ct = out[nonce_len..][0..plain.len]; + const tag = out[nonce_len + plain.len ..][0..tag_len]; + gcm.encrypt(ct, tag, plain, ad, nonce, mk.*); + return sealedLen(plain.len); +} +``` + +The associated data binds the ciphertext to its attribute type. Unseal of a value whose `ad` does not match the type it is being loaded into fails the tag, so a sealed `CKA_PRIVATE_EXPONENT` cannot be swapped into a `CKA_VALUE` slot. The fresh nonce per seal is what keeps GCM safe; reusing a nonce under the same key is GCM's one catastrophic foot-gun, and the design never does. + +A sealed value is `nonce_len + plaintext_len + tag_len` bytes, which is `12 + n + 16`. The on-disk record stores that length so unseal knows where the ciphertext ends and the tag begins. + +## Constant-Time and Zeroization: the cross-cutting mechanics + +These are not mechanisms a host calls, but they are the machinery that makes the mechanisms safe. + +**Constant-time comparison.** Any compare of a secret has to look at every byte regardless of where the first mismatch is, or the time it takes leaks the position. Fixed-size secrets (the PIN hash) use `std.crypto.timing_safe.eql`, which XORs all elements into one accumulator and tests it branchlessly. Variable-length MACs use `ctEql`, the same idea by hand. The CBC-PAD and RFC 3394 checks fold all their differences before deciding, for the same reason. + +**Zeroization.** A secret left in freed or idle memory can be recovered from a crash dump, a swapped page, or a cold-boot read. The module zeroes secrets with `std.crypto.secureZero`, which takes a `[]volatile` slice: + +```zig +pub fn secureZero(comptime T: type, s: []volatile T) void { + @memset(s, 0); +} +``` + +The `volatile` is the whole point. A plain `@memset(&key, 0)` with no subsequent read is a dead store the optimizer is free to delete. Marking the slice volatile tells the compiler the write is an observable side effect that must happen. This is the same technique every serious C crypto library uses, and Zig builds it into one auditable function. Stack secrets pair it with `defer`; heap secrets go through `secureFree`; session operation state zeroes itself on teardown; logout re-seals and wipes the master key. The net effect is that an idle, logged-out token holds no plaintext key material anywhere. + +One Zig-specific wrinkle: in Debug and ReleaseSafe builds, setting an optional to null poisons its payload to `0xAA`, and `Allocator.free` memsets freed memory to `0xAA`. So after a `secureZero`-then-null you see `0xAA`, not zeros. The secret is destroyed (it is `0xAA`, not the key), it is simply not zero. The tests therefore assert "the secret pattern is gone", not "all bytes are zero". In a release build without those safety memsets, the explicit `secureZero` is what does the work. + +## Picking a Mechanism + +Most of the time the choice is dictated by what you are interoperating with, but here is the guidance the design encodes: + +- **Symmetric encryption:** prefer GCM (`CKM_AES_GCM`). It authenticates, so tampering is detected. Use CBC-PAD only when a peer requires unauthenticated CBC, and pair it with a separate MAC if you do. +- **RSA encryption / key transport:** prefer OAEP (`CKM_RSA_PKCS_OAEP`). PKCS#1 v1.5 encryption is the Bleichenbacher target; OAEP is the modern replacement. +- **RSA signatures:** prefer PSS (`CKM_RSA_PKCS_PSS`). It has a security proof v1.5 lacks. Use v1.5 (`CKM_SHA256_RSA_PKCS`) for compatibility with systems that expect it. +- **Signatures in general:** ECDSA gives you the same security as RSA with far smaller keys and signatures (a P-256 signature is 64 bytes versus 256 for RSA-2048). Prefer it when both ends support it. +- **Key wrapping:** AES-KEY-WRAP (RFC 3394) for wrapping a symmetric key under another symmetric key; RSA-OAEP for wrapping under a public key. +- **Key agreement:** ECDH (`CKM_ECDH1_DERIVE`) to derive a shared secret from two EC keypairs. + +## The Detail Behind ECDSA Determinism + +If you are curious why the test suite verifies an external vector but cannot assert its own sign output: + +Plain RFC 6979 ECDSA is fully deterministic: the nonce is a function of the message and the private key, so signing the same message twice yields the same signature, and you can pin a known-answer test on the output. Zig's `signPrehashed`, however, mixes an optional noise field into the nonce derivation (a hedged-signature design that adds defense against fault attacks). That makes the output non-deterministic, so two signatures of the same message differ. + +The consequence for testing is that you verify against RFC 6979 (a known signature must validate under the known key) but you cannot assert that your own signing reproduces the RFC 6979 signature byte for byte. The suite does both of the things you *can* do: it uses the RFC 6979 vector as a verify known-answer test, and it round-trips its own signatures (sign, then verify, then tamper and confirm rejection). This is the correct way to test a hedged signature scheme, and it is worth understanding so the test design does not look like a gap. + +## Next Steps + +1. Read [CONFORMANCE.md](./CONFORMANCE.md) for the exact parameter rules and return codes at every boundary of these mechanisms. +2. Open `src/crypto/cipher.zig` and read the RFC 3394 test, then `src/crypto/ecdsa.zig` and read the ECDH both-sides test. The tests are short and show each mechanism end to end. +3. Try the extensions in [04-CHALLENGES.md](./04-CHALLENGES.md), several of which add a mechanism and walk you through the same wiring you have now seen for the existing ones. diff --git a/README.md b/README.md index 4bce21d2..3aa68579 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@

View Complete Projects:

- Projects + Projects
@@ -151,7 +151,7 @@ Tools, courses, certifications, communities, and frameworks for cybersecurity pr | **[Blockchain Smart Contract Auditor](./SYNOPSES/advanced/Blockchain.Smart.Contract.Auditor.md)**
Solidity vulnerability analysis | ![3-4w](https://img.shields.io/badge/⏱️_3--4w-blue) ![Python](https://img.shields.io/badge/Python-3776AB?logo=python&logoColor=white) ![Solidity](https://img.shields.io/badge/Solidity-363636?logo=solidity) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | Smart contracts • Static analysis • Solidity security
[Learn More](./SYNOPSES/advanced/Blockchain.Smart.Contract.Auditor.md) | | **[Adversarial ML Attacker](./SYNOPSES/advanced/Adversarial.ML.Attacker.md)**
Generate adversarial examples | ![3-4w](https://img.shields.io/badge/⏱️_3--4w-blue) ![Python](https://img.shields.io/badge/Python-3776AB?logo=python&logoColor=white) ![TensorFlow](https://img.shields.io/badge/TensorFlow-FF6F00?logo=tensorflow&logoColor=white) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | Adversarial ML • FGSM/DeepFool • Model robustness
[Learn More](./SYNOPSES/advanced/Adversarial.ML.Attacker.md) | | **[Advanced Persistent Threat Simulator](./SYNOPSES/advanced/Advanced.Persistent.Threat.Simulator.md)**
Multi-stage APT simulation | ![3-4w](https://img.shields.io/badge/⏱️_3--4w-blue) ![Go](https://img.shields.io/badge/Go-00ADD8?logo=go&logoColor=white) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | APT techniques • C2 infrastructure • Lateral movement
[Learn More](./SYNOPSES/advanced/Advanced.Persistent.Threat.Simulator.md) | -| **[Hardware Security Module Emulator](./SYNOPSES/advanced/Hardware.Security.Module.Emulator.md)**
Software HSM with PKCS#11 | ![2-3w](https://img.shields.io/badge/⏱️_2--3w-blue) ![C](https://img.shields.io/badge/C-A8B9CC?logo=c&logoColor=black) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | HSM concepts • PKCS#11 interface • Cryptographic operations
[Learn More](./SYNOPSES/advanced/Hardware.Security.Module.Emulator.md) | +| **[Hardware Security Module Emulator](./PROJECTS/advanced/hsm-emulator)**
Software HSM that compiles to a real PKCS#11 `.so` | ![2-3w](https://img.shields.io/badge/⏱️_2--3w-blue) ![Zig](https://img.shields.io/badge/Zig-F7A41D?logo=zig&logoColor=white) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | PKCS#11/Cryptoki C ABI (machine-checked vs OASIS headers) • AES-GCM/CBC • RSA/ECDSA/ECDH • Argon2id + encrypted-at-rest • driven by `pkcs11-tool`
[Source Code](./PROJECTS/advanced/hsm-emulator) \| [Docs](./PROJECTS/advanced/hsm-emulator/learn) | | **[Network Covert Channel](./SYNOPSES/advanced/Network.Covert.Channel.md)**
Data exfiltration techniques | ![3-4w](https://img.shields.io/badge/⏱️_3--4w-blue) ![Rust](https://img.shields.io/badge/Rust-000000?logo=rust&logoColor=white) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | Covert channels • Data exfiltration • Steganography
[Learn More](./SYNOPSES/advanced/Network.Covert.Channel.md) | | **[Automated Penetration Testing](./SYNOPSES/advanced/Automated.Penetration.Testing.md)**
Full pentest automation | ![3-4w](https://img.shields.io/badge/⏱️_3--4w-blue) ![Python](https://img.shields.io/badge/Python-3776AB?logo=python&logoColor=white) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | Pentest automation • Recon to exploitation • Report generation
[Learn More](./SYNOPSES/advanced/Automated.Penetration.Testing.md) | | **[Haskell Reverse Proxy](./PROJECTS/advanced/haskell-reverse-proxy)**
Functional reverse proxy with security middleware | ![2-3w](https://img.shields.io/badge/⏱️_2--3w-blue) ![Haskell](https://img.shields.io/badge/Haskell-5D4F85?logo=haskell&logoColor=white) ![Advanced](https://img.shields.io/badge/●_Advanced-red) | Functional programming • Reverse proxy design • Security middleware • Haskell
[Source Code](./PROJECTS/advanced/haskell-reverse-proxy) |