This is a security-news tool; CVE enrichment is optional garnish, never a
setup tax. Drop the NVD API key from the default path entirely.
- internal/cve/cvelist: keyless CVE Program cvelistV5 client (raw GitHub,
carries CISA-ADP / vulnrichment CVSS). Parses cna and adp metrics with
version precedence v4.0 > v3.1 > v3.0 > v2.0, plus CWE and description;
404 maps to not-found; v2 severity is capped at HIGH. Introduces a
neutral CoreResult and a CVESource interface (NVDResult renamed to
CoreResult; NVDClient still satisfies it).
- enrich: Clients.Core is the CVESource. buildCoreSource picks the keyless
cvelist client by default and NVD only when NVD_API_KEY is set (booster).
- scrape auto-enriches referenced CVEs best-effort after clustering
(time-bounded, non-fatal, --no-enrich to skip); enrich kept as an
optional manual refresh. Default flow is scrape -> tui.
- ranking default weights rebalanced news-first (recency / velocity /
source / keyword 70 percent, cvss / kev / epss 30 percent; was 55 percent CVE).
- KAT fixture testdata/cvelist/CVE-2021-44228.json.
Interactive terminal UI over the ranked story clusters. Rave Maximal
cybercore theming: cyan-to-magenta spectrum score bars, severity-reactive
rows, hot KEV chips, cool-ramp outlet dots, and a scrollable detail
dossier with per-outlet links and full CVE cards (CVSS meter, KEV /
ransomware, EPSS, CWE, vector, description). Reuses store.DigestClusters
and rank.Rank verbatim and preloads full store.CVE records for detail.
- internal/tui: model/update/view state machine with an injected Loader
and opener, so Update is fully unit-testable without a terminal or DB
(nav / clamp / open / back / quit / resize / empty-store / too-small).
- o opens the selected story in the default browser, keyboard-only,
cross-platform, and refuses non-http urls.
- tui command replaces the M5 stub; supports --since.
- deps: charmbracelet bubbletea v1.3.10 + bubbles v1.0.0 + lipgloss v1.1.0.
Turn the aggregated store into a ranked, exportable digest of story clusters.
- internal/rank: pure deterministic Score(Signals, cfg.Rank) — recency
(exp half-life decay), cvss/10, kev, epss, velocity (cluster size/age), source
weight, watchlist keyword. Every signal clamped to [0,1]; all weights +
half_life + velocity_norm from config, no literals. recency/velocity are total
(guard half_life<1 and norm<=0). Rank() scores each cluster and stable-sorts
desc, tiebreak freshest. Research 06's two worked examples are the golden-order
test (A ~0.99 strictly above B ~0.13).
- internal/export: per-cluster (story) digest. Headline = freshest member, distinct
sorted outlets, CVEs sorted KEV-first then CVSS. Markdown + deterministic JSON.
- store.DigestClusters: 3-query per-cluster aggregation (clusters by since, member
articles, union of CVE signals) with explicit ORDER BY; per-cluster CVE dedup
via SELECT DISTINCT.
- digest command: --top/--since/--format md|json/--out. Off the stub list.
- Proven live: scrape -> digest --format md and json render correctly, per-cluster
(FortiBleed shows 3 outlets). Suite offline + -race.
One read-only audit agent; 0 Crit/High/Med, golden-order math verified; Low/Nit
fixed in-phase (top<=0 count, pure-fn guards, ORDER BY, headline sanitize, 2-cluster
attach test).
Digest ranks per-cluster (Carter's choice), not per-article.
The intelligence layer. scrape now extracts CVE IDs (cheap, no network) into
article_cves, which activates M2's shared-CVE clustering. A separate 'enrich'
command does the slow, cached API work (Carter's split: fast scrape, deliberate
enrich).
- internal/cve: regex CVE-ID extraction + three clients (stdlib net/http):
- nvd: API 2.0, CVSS precedence v4.0>v3.1>v3.0>v2 with nullable metrics and v2
metric-level severity fallback, totalResults==0 => not-found, apiKey header
(exact case), token-bucket rate limit (5/30s anon, 50/30s keyed) + retry on
429/5xx, timeouts non-retryable.
- kev: one catalog download -> membership map; knownRansomwareCampaignUse is the
STRING 'Known'/'Unknown' mapped explicitly to bool.
- epss: batched; epss/percentile are QUOTED STRINGS parsed with ParseFloat (a
parse failure is skipped, a legitimate 0.0 is kept); partial batches survive
a transient error.
- internal/enrich: enriches only unenriched/stale CVEs (positive + negative TTL),
KEV once per run, EPSS batched, NVD per-CVE. NVD/EPSS soft per-item (resumable),
KEV fatal by design (enriching without it would persist wrong is_kev for a TTL).
- store: UpsertCVEStub/LinkArticleCVE, CVEsNeedingEnrichment (TTL), UpdateCVEEnrichment
(nullable *float64 -> SQL NULL), GetCVE, ArticlesForCVE, ListArticles (parameterized
--source/--since/--min-cvss/--kev/--keyword via EXISTS subqueries).
- commands: enrich, cve, list (cve/list off the stub list). Trimmed kev-sample.json
(7KB) fixture for KAT tests; nvd/epss fixtures committed.
- KAT proven offline AND live: CVE-2021-44228 -> CVSS 10.0 CRITICAL (3.1), CWE-20,
KEV added 2021-12-10 ransomware yes, EPSS 0.99999. 215 articles -> 82 CVEs, and
shared-CVE clustering lit up (multi-source clusters 2 -> 5). Suite offline + -race.
One read-only audit agent run; 0 Crit/High/Med, Low/Nit fixes applied in-phase.
NVD apiKey header set with exact case to bypass Go header canonicalization.
Group the same story across outlets into clusters, the velocity signal for
ranking. Runs automatically after 'scrape' (recompute-from-scratch, idempotent).
- internal/cluster: connected-components engine (union-find). Two articles join
when, within a time window (Cluster.WindowHours, default 72h), EITHER their
normalized-title token-set Jaccard >= threshold (default 0.6) AND they are from
different outlets, OR they share >= 1 CVE (source-agnostic). The cross-outlet
guard on the title edge kills a real false positive where two distinct CISA
advisories with identical boilerplate titles were merging. Cluster key = the
earliest article by time; SourceCount tracks distinct outlets so the reported
'multi-source' metric is truthful.
- Shared-CVE join reads article_cves, which is empty until M3 CVE extraction, so
that path is a natural no-op today and lights up in M3 with no rework.
- store: ClusterCandidates (effective time = published_at else fetched_at, bounded
by lookback), ArticleCVEMap, ReplaceClusters (single atomic tx: wipe + rebuild).
- config: Cluster.LookbackHours (default 168h) bounds the corpus for O(n^2);
validated as >= window_hours so window edges are never silently dropped.
- Proven live: 215 real articles -> 137 clusters, 2 genuinely cross-outlet
(FortiBleed x3, Scattered Spider x2). 17 cluster tests; suite offline + -race.
One read-only audit agent run; findings (multi-source stat, lookback guard) fixed in-phase.
Clustering is recompute-from-scratch each scrape; safe to re-run.
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.
- internal/fetch: per-host token-bucket rate limiting, conditional GET
(ETag/Last-Modified with 304 handling), retry+backoff that honors
Retry-After and treats timeouts/cancellation as non-retryable, per-source
deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
Client.Allowed() for the future HTML article-scrape path, not on subscribed
feed fetches (user-directed retrieval; several publishers blanket-disallow
generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
-> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
suite is offline and passes under -race. Proven live: cold run 215 new across
7 sources, warm run 0 new (304s + dedup).
Ctrl-C aborts scrape cleanly via signal-aware context.
Go engine that aggregates cybersecurity news and enriches CVEs. This
first commit lays the M0 foundation:
- config: fetch/enrich/cluster/rank/AI settings, YAML + validation,
defaults as named constants, AI opt-in and off by default
- source: Source registry with embedded default feeds (7 verified RSS
sources), external-file override, dedup/URL validation
- store: pure-Go SQLite (modernc), WAL + foreign_keys, forward-only
embedded migrations with schema_migrations tracking and a loud
newer-than-binary guard; ErrDuplicate sentinel for dedup
- cmd: cobra skeleton, version + sources wired, remaining commands
stubbed to their milestones
go vet / gofmt / go test all clean.
connect-scan: raw non-blocking connect() + poll(POLLOUT) + SO_ERROR via std.os.linux (std.Io.net connect-with-timeout is unimplemented in 0.16); own sized std.Io.Threaded with N io.async workers off a mutex+token-bucket dispenser; --backend connect / --connect, --concurrency, --connect-timeout.
cloud/VM detection: two-socket AF_PACKET self-probe (send on one, observe the tagged frame on a second - single-socket cannot see its own send); --backend auto checks CAP_NET_RAW + egress and auto-falls-back to connect with a notice; forced raw backend disables fallback and hard-fails; zero-response raw scan hints --connect.
raw IPv6 SYN: packet.Addr union result + RFC5952 v6 render; Ipv6Hdr + pseudoChecksum6/tcpChecksum6; 128-bit SipHash cookie (generate6/seq6); parseIpv6, resolveSrcIp6 (/proc/net/if_inet6), defaultGateway6 (/proc/net/ipv6_route); Engine6 (bounded prefix, RFC6890 reserved floor, ::/0 reject, host cap); SynTemplate6; classifyTcp6 + ICMPv6 type1; ndp.zig NDP neighbor resolution; runV6Scan dispatch; connect-path v6 rides the connect engine. v6 scope = TCP SYN.
240 tests Debug+ReleaseSafe (-Dxdp on/off); KAT + two-namespace netns e2e proven on the wire; two read-only audits, 0 Critical/High/Medium, all findings fixed in-phase.
OS-realistic SYN templates (Linux/Windows/macOS/masscan JA4T option chains
plus varying IP-id), Poisson jitter, source-port rotation (RX recomputes off
the reply, zero classify changes), scoped RST-suppression (iptables plus
ambient CAP_NET_ADMIN, self-healing delete-before-insert), decoys (bogon-free
RND, real probe always sent), and FIN/NULL/Xmas/Maimon/ACK/Window flag scans
with per-mode cookie matching plus State.unfiltered.
Dead theater (idle scan, fragmentation, TTL, MAC/source-route spoof, badsum)
omitted and documented as obsolete with citations.
Cursor-based TX emission with token refund plus cold-start pacing; non-stealth
path byte-identical to M7.
Turn the scan from sequential TX-then-RX into two io.concurrent engines on
std.Io.Threaded, fixing the M4 bug where replies arriving during transmit were
dropped and the RX deadline (anchored at socket-open) expired before draining.
The main thread is the non-blocking Io.Queue consumer and dashboard renderer.
- rx: Receiver keyed off a shared tx_done atomic; new pure planDrain anchors the
drain window at TX-completion, not socket-open, killing the quiet-gap early exit
during a slow TX. Hard-cap safety backstop anchored at drain start.
- output (new): cache-line-padded atomic Stats, truecolor/256/none palette
(violet gradient, neon-green, muted-gray), in-place multi-line live dashboard,
Unicode results table, NDJSON to stdout with visuals on stderr, full
NO_COLOR/CLICOLOR_FORCE/COLORTERM/--color chain plus a narrow-terminal fallback.
- tx: wall-clock deadline so a stalled TX ring can no longer hang the scan.
- cli: banner recolored to the violet gradient and routed through the color chain.
- targets: add 192.88.99.0/24 (6to4 relay anycast) to the RFC 6890 exclude floor.
78/78 unit tests pass under Debug and ReleaseSafe; proven end-to-end (open/closed,
dedup, dashboard, NDJSON) in an unshare -r -n netns under ReleaseSafe.
classify.zig pure frame parser: SYN-ACK -> open, RST/ACK -> closed,
ICMP type-3 {1,2,3,9,10,13} -> filtered, every verdict gated on a
wrapping SipHash cookie re-check (ack == seq+1 for replies, inner seq
== cookie for ICMP) so spoofed/stale frames are dropped state-free;
byte-offset readInt parsing, fully length-guarded. dedup.zig power-of-2
open-addressed (ip,port) set, single-owned allocator, grows at load
factor with a bounded probe (never wedges). rx.zig receive engine
generic over an injected Source + Sink (proven over hand-built frames),
Io.Queue found-host handoff across producer/consumer fibers, and a
privileged AF_PACKET Receiver (poll-bounded drain with a hard deadline,
ignore-outgoing, EINTR-retry, errno-gated, fail-closed on missing
CAP_NET_RAW). scan subcommand shares one per-run cookie across the TX
template + RX validation; consumer uses a dedicated allocator to keep
the producer/consumer fibers race-free. netutil.zig factors the shared
arg-parse/iface-resolve helpers out of txcmd. Proven end-to-end with a
live SYN scan inside a unshare -r -n netns over a veth pair
(open/closed against a real listener, dedup verified). Green Debug +
ReleaseSafe, leak-free, 61 unit tests.
PACKET_TX_RING (PACKET_MMAP, TPACKET_V2) over AF_PACKET with
PACKET_QDISC_BYPASS: socket + version + TX_RING setsockopt + mmap'd ring,
errno-gated, fail-closed on missing CAP_NET_RAW. ratelimit.zig integer
token bucket (banked-nanoseconds, no float, saturating, exact KATs).
template.zig SYN frame template: stamp dst IP/port + SipHash seq, recompute
IP + TCP checksums (reuses the M1 RFC 1071 + cookie code, both self-verify).
afpacket.zig pure Ring slot bookkeeping (inline tpacket2_hdr, comptime size
assert, reserve/fill/atomic-status, self-defending bounds, kernel-drain
reuse) split from the privileged Backend so the hot-path accounting is
unit-tested over an in-process buffer. tx.zig transmit engine generic over
an injected Sink + Clock: the M2 cyclic-group bijection is proven to survive
stamp -> ratelimit -> submit end-to-end via a fake sink + fake clock.
Privileged tx subcommand for on-hardware benchmarking (setcap + tcpdump/perf).
Green Debug + ReleaseSafe, leak-free, 45/45 tests.
SIMD @Vector internet checksum proven byte-equivalent to scalar across
every length 0..256 plus the 0xb861 IPv4 KAT. RFC 1624 incremental
update proven against the RFC section 4 worked example
(0xDD2F/0x5555/0x3285 -> 0x0000) plus 4096 random full-recompute trials.
Reusable TCP pseudo-header checksum helper; smoke.zig now reuses it.
New cookie.zig: stateless SipHash64(2,4) SYN-cookie over the 4-tuple,
full-128-bit randomSecure key, u64 generate with write-site u32
truncation, ack == cookie +% 1 wrapping validation. Reproduces the
published SipHash reference vector plus a pinned golden cookie KAT.
All green Debug + ReleaseSafe (14/14). Version bumped to 0.0.0-m1.
Zig 0.16 module graph (build.zig DAG with per-module addTest, run/smoke
steps), Juicy Main entry, truecolor tty-gated banner, --version/--help.
Wire-exact extern eth/ip/tcp headers with comptime @sizeOf asserts and
an RFC 1071 checksum proven by the canonical 0xb861 IPv4 KAT. AF_PACKET
raw socket that hand-builds and sends one SYN, checking errno not fd<0.
4/4 tests green under Debug and ReleaseSafe.
- add 1200x630 og-image.jpg plus Open Graph and Twitter Card meta tags so
the live demo renders a rich preview when shared (LinkedIn, etc.)
- omit og:url to avoid crawler cache canonicalizing share URLs back to a
stale entry
- justfile: add redeploy recipe (tunnel down, cached build, up -d with the
backend profile and remove-orphans); strip the exported dev .env overrides
so it deploys under the prod project name and ports
- install.sh: clearer live-capture hint (setcap, then run without sudo)
CarterPerez-dev
dependabot[bot]