This is a security-news tool; CVE enrichment is optional garnish, never a
setup tax. Drop the NVD API key from the default path entirely.
- internal/cve/cvelist: keyless CVE Program cvelistV5 client (raw GitHub,
carries CISA-ADP / vulnrichment CVSS). Parses cna and adp metrics with
version precedence v4.0 > v3.1 > v3.0 > v2.0, plus CWE and description;
404 maps to not-found; v2 severity is capped at HIGH. Introduces a
neutral CoreResult and a CVESource interface (NVDResult renamed to
CoreResult; NVDClient still satisfies it).
- enrich: Clients.Core is the CVESource. buildCoreSource picks the keyless
cvelist client by default and NVD only when NVD_API_KEY is set (booster).
- scrape auto-enriches referenced CVEs best-effort after clustering
(time-bounded, non-fatal, --no-enrich to skip); enrich kept as an
optional manual refresh. Default flow is scrape -> tui.
- ranking default weights rebalanced news-first (recency / velocity /
source / keyword 70 percent, cvss / kev / epss 30 percent; was 55 percent CVE).
- KAT fixture testdata/cvelist/CVE-2021-44228.json.
The intelligence layer. scrape now extracts CVE IDs (cheap, no network) into
article_cves, which activates M2's shared-CVE clustering. A separate 'enrich'
command does the slow, cached API work (Carter's split: fast scrape, deliberate
enrich).
- internal/cve: regex CVE-ID extraction + three clients (stdlib net/http):
- nvd: API 2.0, CVSS precedence v4.0>v3.1>v3.0>v2 with nullable metrics and v2
metric-level severity fallback, totalResults==0 => not-found, apiKey header
(exact case), token-bucket rate limit (5/30s anon, 50/30s keyed) + retry on
429/5xx, timeouts non-retryable.
- kev: one catalog download -> membership map; knownRansomwareCampaignUse is the
STRING 'Known'/'Unknown' mapped explicitly to bool.
- epss: batched; epss/percentile are QUOTED STRINGS parsed with ParseFloat (a
parse failure is skipped, a legitimate 0.0 is kept); partial batches survive
a transient error.
- internal/enrich: enriches only unenriched/stale CVEs (positive + negative TTL),
KEV once per run, EPSS batched, NVD per-CVE. NVD/EPSS soft per-item (resumable),
KEV fatal by design (enriching without it would persist wrong is_kev for a TTL).
- store: UpsertCVEStub/LinkArticleCVE, CVEsNeedingEnrichment (TTL), UpdateCVEEnrichment
(nullable *float64 -> SQL NULL), GetCVE, ArticlesForCVE, ListArticles (parameterized
--source/--since/--min-cvss/--kev/--keyword via EXISTS subqueries).
- commands: enrich, cve, list (cve/list off the stub list). Trimmed kev-sample.json
(7KB) fixture for KAT tests; nvd/epss fixtures committed.
- KAT proven offline AND live: CVE-2021-44228 -> CVSS 10.0 CRITICAL (3.1), CWE-20,
KEV added 2021-12-10 ransomware yes, EPSS 0.99999. 215 articles -> 82 CVEs, and
shared-CVE clustering lit up (multi-source clusters 2 -> 5). Suite offline + -race.
One read-only audit agent run; 0 Crit/High/Med, Low/Nit fixes applied in-phase.
NVD apiKey header set with exact case to bypass Go header canonicalization.
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.
- internal/fetch: per-host token-bucket rate limiting, conditional GET
(ETag/Last-Modified with 304 handling), retry+backoff that honors
Retry-After and treats timeouts/cancellation as non-retryable, per-source
deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
Client.Allowed() for the future HTML article-scrape path, not on subscribed
feed fetches (user-directed retrieval; several publishers blanket-disallow
generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
-> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
suite is offline and passes under -race. Proven live: cold run 215 new across
7 sources, warm run 0 new (304s + dedup).
Ctrl-C aborts scrape cleanly via signal-aware context.
CarterPerez-dev