RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Cybersecurity-Projects/PROJECTS/intermediate/security-news-scraper
History
CarterPerez-dev 312b13b348 feat(nadezhda): M2 cross-source dedup + clustering 
Group the same story across outlets into clusters, the velocity signal for
ranking. Runs automatically after 'scrape' (recompute-from-scratch, idempotent).

- internal/cluster: connected-components engine (union-find). Two articles join
  when, within a time window (Cluster.WindowHours, default 72h), EITHER their
  normalized-title token-set Jaccard >= threshold (default 0.6) AND they are from
  different outlets, OR they share >= 1 CVE (source-agnostic). The cross-outlet
  guard on the title edge kills a real false positive where two distinct CISA
  advisories with identical boilerplate titles were merging. Cluster key = the
  earliest article by time; SourceCount tracks distinct outlets so the reported
  'multi-source' metric is truthful.
- Shared-CVE join reads article_cves, which is empty until M3 CVE extraction, so
  that path is a natural no-op today and lights up in M3 with no rework.
- store: ClusterCandidates (effective time = published_at else fetched_at, bounded
  by lookback), ArticleCVEMap, ReplaceClusters (single atomic tx: wipe + rebuild).
- config: Cluster.LookbackHours (default 168h) bounds the corpus for O(n^2);
  validated as >= window_hours so window edges are never silently dropped.
- Proven live: 215 real articles -> 137 clusters, 2 genuinely cross-outlet
  (FortiBleed x3, Scattered Spider x2). 17 cluster tests; suite offline + -race.

One read-only audit agent run; findings (multi-source stat, lookback guard) fixed in-phase.

Clustering is recompute-from-scratch each scrape; safe to re-run.
 		2026-07-05 15:11:29 -04:00
..
cmd/nadezhda feat(nadezhda): M2 cross-source dedup + clustering 2026-07-05 15:11:29 -04:00
internal feat(nadezhda): M2 cross-source dedup + clustering 2026-07-05 15:11:29 -04:00
testdata/feeds feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
.gitignore feat(nadezhda): M0 scaffold for security news + CVE aggregator 2026-07-05 13:22:08 -04:00
README.md feat(nadezhda): M0 scaffold for security news + CVE aggregator 2026-07-05 13:22:08 -04:00
go.mod feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
go.sum feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
justfile feat(nadezhda): M0 scaffold for security news + CVE aggregator 2026-07-05 13:22:08 -04:00

README.md

Nadezhda

A concurrent security-news and CVE aggregation engine, written in Go.

Nadezhda ingests cybersecurity news from reliable RSS feeds, enriches every referenced CVE with authoritative exploit intelligence (NVD, CISA KEV, FIRST EPSS), clusters the same story across outlets, ranks items by real-world significance, and surfaces content angles. It ships as a single static binary with a local SQLite store, a colorful terminal UI, and Markdown/JSON export.

Status

Early development. The scaffold is in place: configuration, source registry, SQLite store with forward-only migrations, and the command skeleton.

nadezhda version     # print version
nadezhda sources     # list configured feeds and persist them to the store

Ingestion, CVE enrichment, ranking, the TUI, and the AI ideation layer land in subsequent milestones.

Build

just build     # -> ./nadezhda
just test

Requires Go 1.25+.

Full documentation lands in learn/ as the project matures.