M3 React web app over the existing FastAPI: hex viewer, disassembly pane, section map, strings, challenge runner with reveal-after-solve, and progress. Scaffolded from the open no-auth react-scss template; components kept self-contained for CertGames extraction.
M3.5 static-analysis depth, all still no-execution and KAT-traced to the gate binary: PLT/GOT import resolution (plt.py), RIP-relative cross-references and callers (xref.py), basic-block control-flow graphs (cfg.py), and prologue-scan function discovery for stripped binaries (discover.py). Adds the 06-stripped-gate challenge. The API now resolves call names, exposes rip targets, and serves /cfg, /xrefs, and discovered functions.
M2 API hardening from a read-only audit: challenge-loader error isolation, a request body-size middleware, session length bounds, CORS, and the disasm gate annotation gated behind solved state so it no longer pre-solves the challenge.
96 backend tests green; frontend typecheck, build, and lint clean.
Group the same story across outlets into clusters, the velocity signal for
ranking. Runs automatically after 'scrape' (recompute-from-scratch, idempotent).
- internal/cluster: connected-components engine (union-find). Two articles join
when, within a time window (Cluster.WindowHours, default 72h), EITHER their
normalized-title token-set Jaccard >= threshold (default 0.6) AND they are from
different outlets, OR they share >= 1 CVE (source-agnostic). The cross-outlet
guard on the title edge kills a real false positive where two distinct CISA
advisories with identical boilerplate titles were merging. Cluster key = the
earliest article by time; SourceCount tracks distinct outlets so the reported
'multi-source' metric is truthful.
- Shared-CVE join reads article_cves, which is empty until M3 CVE extraction, so
that path is a natural no-op today and lights up in M3 with no rework.
- store: ClusterCandidates (effective time = published_at else fetched_at, bounded
by lookback), ArticleCVEMap, ReplaceClusters (single atomic tx: wipe + rebuild).
- config: Cluster.LookbackHours (default 168h) bounds the corpus for O(n^2);
validated as >= window_hours so window edges are never silently dropped.
- Proven live: 215 real articles -> 137 clusters, 2 genuinely cross-outlet
(FortiBleed x3, Scattered Spider x2). 17 cluster tests; suite offline + -race.
One read-only audit agent run; findings (multi-source stat, lookback guard) fixed in-phase.
Clustering is recompute-from-scratch each scrape; safe to re-run.
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.
- internal/fetch: per-host token-bucket rate limiting, conditional GET
(ETag/Last-Modified with 304 handling), retry+backoff that honors
Retry-After and treats timeouts/cancellation as non-retryable, per-source
deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
Client.Allowed() for the future HTML article-scrape path, not on subscribed
feed fetches (user-directed retrieval; several publishers blanket-disallow
generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
-> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
suite is offline and passes under -race. Proven live: cold run 215 new across
7 sources, warm run 0 new (304s + dedup).
Ctrl-C aborts scrape cleanly via signal-aware context.
Go engine that aggregates cybersecurity news and enriches CVEs. This
first commit lays the M0 foundation:
- config: fetch/enrich/cluster/rank/AI settings, YAML + validation,
defaults as named constants, AI opt-in and off by default
- source: Source registry with embedded default feeds (7 verified RSS
sources), external-file override, dedup/URL validation
- store: pure-Go SQLite (modernc), WAL + foreign_keys, forward-only
embedded migrations with schema_migrations tracking and a loud
newer-than-binary guard; ErrDuplicate sentinel for dedup
- cmd: cobra skeleton, version + sources wired, remaining commands
stubbed to their milestones
go vet / gofmt / go test all clean.
Framework-free analysis engine (hex dump, hand-rolled ELF64 parser,
capstone x86-64 disassembly, string extraction, byte patching, challenge
grading) behind a thin read-only FastAPI layer over curated pre-compiled
challenge binaries.
No-execution security posture: the backend only reads and decodes bytes,
never runs a target; patch challenges are graded by static byte diff.
Reveal-after-solve pedagogy: challenge source is returned only on a
correct submission. Input size-caps enforced at the API boundary.
M0 research (ELF format, x86-64/capstone, pedagogy, no-execution) is
cross-checked against a real compiled binary. 63 tests green, including
an ELF parser cross-check against pyelftools and disassembly that matches
objdump instruction-for-instruction.
connect-scan: raw non-blocking connect() + poll(POLLOUT) + SO_ERROR via std.os.linux (std.Io.net connect-with-timeout is unimplemented in 0.16); own sized std.Io.Threaded with N io.async workers off a mutex+token-bucket dispenser; --backend connect / --connect, --concurrency, --connect-timeout.
cloud/VM detection: two-socket AF_PACKET self-probe (send on one, observe the tagged frame on a second - single-socket cannot see its own send); --backend auto checks CAP_NET_RAW + egress and auto-falls-back to connect with a notice; forced raw backend disables fallback and hard-fails; zero-response raw scan hints --connect.
raw IPv6 SYN: packet.Addr union result + RFC5952 v6 render; Ipv6Hdr + pseudoChecksum6/tcpChecksum6; 128-bit SipHash cookie (generate6/seq6); parseIpv6, resolveSrcIp6 (/proc/net/if_inet6), defaultGateway6 (/proc/net/ipv6_route); Engine6 (bounded prefix, RFC6890 reserved floor, ::/0 reject, host cap); SynTemplate6; classifyTcp6 + ICMPv6 type1; ndp.zig NDP neighbor resolution; runV6Scan dispatch; connect-path v6 rides the connect engine. v6 scope = TCP SYN.
240 tests Debug+ReleaseSafe (-Dxdp on/off); KAT + two-namespace netns e2e proven on the wire; two read-only audits, 0 Critical/High/Medium, all findings fixed in-phase.
OS-realistic SYN templates (Linux/Windows/macOS/masscan JA4T option chains
plus varying IP-id), Poisson jitter, source-port rotation (RX recomputes off
the reply, zero classify changes), scoped RST-suppression (iptables plus
ambient CAP_NET_ADMIN, self-healing delete-before-insert), decoys (bogon-free
RND, real probe always sent), and FIN/NULL/Xmas/Maimon/ACK/Window flag scans
with per-mode cookie matching plus State.unfiltered.
Dead theater (idle scan, fragmentation, TTL, MAC/source-route spoof, badsum)
omitted and documented as obsolete with citations.
Cursor-based TX emission with token refund plus cold-start pacing; non-stealth
path byte-identical to M7.
Turn the scan from sequential TX-then-RX into two io.concurrent engines on
std.Io.Threaded, fixing the M4 bug where replies arriving during transmit were
dropped and the RX deadline (anchored at socket-open) expired before draining.
The main thread is the non-blocking Io.Queue consumer and dashboard renderer.
- rx: Receiver keyed off a shared tx_done atomic; new pure planDrain anchors the
drain window at TX-completion, not socket-open, killing the quiet-gap early exit
during a slow TX. Hard-cap safety backstop anchored at drain start.
- output (new): cache-line-padded atomic Stats, truecolor/256/none palette
(violet gradient, neon-green, muted-gray), in-place multi-line live dashboard,
Unicode results table, NDJSON to stdout with visuals on stderr, full
NO_COLOR/CLICOLOR_FORCE/COLORTERM/--color chain plus a narrow-terminal fallback.
- tx: wall-clock deadline so a stalled TX ring can no longer hang the scan.
- cli: banner recolored to the violet gradient and routed through the color chain.
- targets: add 192.88.99.0/24 (6to4 relay anycast) to the RFC 6890 exclude floor.
78/78 unit tests pass under Debug and ReleaseSafe; proven end-to-end (open/closed,
dedup, dashboard, NDJSON) in an unshare -r -n netns under ReleaseSafe.
classify.zig pure frame parser: SYN-ACK -> open, RST/ACK -> closed,
ICMP type-3 {1,2,3,9,10,13} -> filtered, every verdict gated on a
wrapping SipHash cookie re-check (ack == seq+1 for replies, inner seq
== cookie for ICMP) so spoofed/stale frames are dropped state-free;
byte-offset readInt parsing, fully length-guarded. dedup.zig power-of-2
open-addressed (ip,port) set, single-owned allocator, grows at load
factor with a bounded probe (never wedges). rx.zig receive engine
generic over an injected Source + Sink (proven over hand-built frames),
Io.Queue found-host handoff across producer/consumer fibers, and a
privileged AF_PACKET Receiver (poll-bounded drain with a hard deadline,
ignore-outgoing, EINTR-retry, errno-gated, fail-closed on missing
CAP_NET_RAW). scan subcommand shares one per-run cookie across the TX
template + RX validation; consumer uses a dedicated allocator to keep
the producer/consumer fibers race-free. netutil.zig factors the shared
arg-parse/iface-resolve helpers out of txcmd. Proven end-to-end with a
live SYN scan inside a unshare -r -n netns over a veth pair
(open/closed against a real listener, dedup verified). Green Debug +
ReleaseSafe, leak-free, 61 unit tests.
PACKET_TX_RING (PACKET_MMAP, TPACKET_V2) over AF_PACKET with
PACKET_QDISC_BYPASS: socket + version + TX_RING setsockopt + mmap'd ring,
errno-gated, fail-closed on missing CAP_NET_RAW. ratelimit.zig integer
token bucket (banked-nanoseconds, no float, saturating, exact KATs).
template.zig SYN frame template: stamp dst IP/port + SipHash seq, recompute
IP + TCP checksums (reuses the M1 RFC 1071 + cookie code, both self-verify).
afpacket.zig pure Ring slot bookkeeping (inline tpacket2_hdr, comptime size
assert, reserve/fill/atomic-status, self-defending bounds, kernel-drain
reuse) split from the privileged Backend so the hot-path accounting is
unit-tested over an in-process buffer. tx.zig transmit engine generic over
an injected Sink + Clock: the M2 cyclic-group bijection is proven to survive
stamp -> ratelimit -> submit end-to-end via a fake sink + fake clock.
Privileged tx subcommand for on-hardware benchmarking (setcap + tcpdump/perf).
Green Debug + ReleaseSafe, leak-free, 45/45 tests.
SIMD @Vector internet checksum proven byte-equivalent to scalar across
every length 0..256 plus the 0xb861 IPv4 KAT. RFC 1624 incremental
update proven against the RFC section 4 worked example
(0xDD2F/0x5555/0x3285 -> 0x0000) plus 4096 random full-recompute trials.
Reusable TCP pseudo-header checksum helper; smoke.zig now reuses it.
New cookie.zig: stateless SipHash64(2,4) SYN-cookie over the 4-tuple,
full-128-bit randomSecure key, u64 generate with write-site u32
truncation, ack == cookie +% 1 wrapping validation. Reproduces the
published SipHash reference vector plus a pinned golden cookie KAT.
All green Debug + ReleaseSafe (14/14). Version bumped to 0.0.0-m1.
Zig 0.16 module graph (build.zig DAG with per-module addTest, run/smoke
steps), Juicy Main entry, truecolor tty-gated banner, --version/--help.
Wire-exact extern eth/ip/tcp headers with comptime @sizeOf asserts and
an RFC 1071 checksum proven by the canonical 0xb861 IPv4 KAT. AF_PACKET
raw socket that hand-builds and sends one SYN, checking errno not fd<0.
4/4 tests green under Debug and ReleaseSafe.
- add 1200x630 og-image.jpg plus Open Graph and Twitter Card meta tags so
the live demo renders a rich preview when shared (LinkedIn, etc.)
- omit og:url to avoid crawler cache canonicalizing share URLs back to a
stale entry
- justfile: add redeploy recipe (tunnel down, cached build, up -d with the
backend profile and remove-orphans); strip the exported dev .env overrides
so it deploys under the prod project name and ports
- install.sh: clearer live-capture hint (setcap, then run without sudo)
Add a --report flag to the pcap command that prints one forensic summary of
a whole capture (endpoint inventory, fingerprint distribution, intel verdicts,
detection alerts, and a coverage section with the TLS miss rate and throughput)
instead of the per-event stream, folding in intel and detection automatically
when a seeded database is present. Add a tls_handshakes_fingerprinted counter
and Counters::tls_miss_rate so truncated and multi-segment ClientHellos the
capture clipped are reported rather than mistaken for absence. Add criterion
benches over the vendored captures and the parse/hash hot path. Plus frontend
landing and asset polish.
materializeDefaults now sets CKA_ENCRYPT/DECRYPT/SIGN/VERIFY/SIGN_RECOVER/VERIFY_RECOVER/WRAP/UNWRAP/DERIVE (plus ALWAYS_AUTHENTICATE for private keys) to CK_FALSE when absent, per CKO_PUBLIC_KEY/PRIVATE_KEY/SECRET_KEY; smoke sets the RSA wrap and recover usage flags.
Passive TLS client and server fingerprinting from packet captures. Two-crate
workspace: tlsfp-core (parsing and fingerprint logic, no I/O, unsafe forbidden)
and tlsfp (clap CLI). Edition 2024, MSRV 1.85.
Core (M0-M4): a hand-rolled bounds-checked TLS parser for ClientHello,
ServerHello, and the TLS 1.2 Certificate chain, plus the full fingerprint
family computed byte-exact against published vectors: JA3/JA3S (MD5), JA4/JA4_r,
JA4S, JA4H, JA4X, and JA4T. GREASE is stripped, wire order and unknown cipher
and extension values are preserved.
Pipeline (M5): a pcap and pcapng file reader behind a PacketSource trait,
etherparse L2-L4 decode (Ethernet, raw IP, BSD loopback, Linux SLL and SLL2,
VLAN and QinQ), and a hand-rolled per-flow TCP reassembler with wrapping serial
arithmetic, out-of-order parking, first-write-wins overlap resolution, and
bounded memory. The tlsfp pcap subcommand fingerprints every handshake in a
capture, as readable lines or NDJSON.
Verified three ways: 94 tests including unit KATs, nine real-pcap full-pipeline
KATs against the FoxIO sample corpus, and property tests that reassemble
arbitrarily segmented and reordered streams. Builds clean on stable and MSRV
1.85; clippy -D warnings and rustfmt clean.
Test captures under testdata/pcap are vendored unmodified from the FoxIO ja4
repository with per-file SHA-256 provenance.
build_module printed (ReleaseSafe) but ran plain 'zig build', which is Debug here (preferred_optimize_mode only ships ReleaseSafe with --release). It left a 20M Debug .so labeled as the shipped build, and 'zig build smoke' would reinstall Debug over any ReleaseSafe artifact via its install-step dependency.
Run build + test + smoke with --release=safe so the script produces and validates the real 5.9M shipped artifact end to end. Kept check-only (no dependency auto-install): missing deps still hard-fail/warn with the exact apt command.
Verified: ./install.sh exits 0 on a clean tree, final zig-out/lib/libhsm.so is 5.9M ReleaseSafe, tests + smoke pass, pkcs11-tool loads it.
learn/: add 00-OVERVIEW, 01-CONCEPTS, 02-ARCHITECTURE, 03-IMPLEMENTATION, 04-CHALLENGES, and MECHANICS (the byte-by-byte crypto deep-dive). CONFORMANCE shipped in M11. Real-breach grounding, function/file code refs (no line numbers), no AI voice.
Project README: flip M0-stale to real M0-M11 (21 mechanisms, std.Io.randomSecure, roadmap done through M12, learn/ link table).
Root README: graduate HSM to a completed project (Full Source Code 32->33, C badge -> Zig, title + footer now link to source and learn/).
Verified docs-only: zig build test and zig build smoke both exit 0, smoke mechanisms=21, objdump -T shows only C_GetFunctionList. No source changed.
CarterPerez-dev
surya
dependabot[bot]