Center assets/tui.png below the tagline and above the overview, using the house p-align-center image convention. Carries the badge bump to Project #39 and the What it is section heading.
Match the ja3-ja4 / zingela README convention: ANSI Shadow banner, badge row (Project 37, Go, keyless, single static binary, AGPL), blockquote pitch, then Why / What Works Today / Quick Start with real sample output / just TIP / Learn table / Architecture diagram / Build and Test / Project Structure. Drops the phantom just-watch reference (no such recipe). Content tracks the verified code: news-first 70/30 weights, Log4Shell KAT as CWE-502.
The 70% news bloc is recency+velocity+source+keyword, not recency+velocity alone (which sum to 0.50). Log4Shell surfaces as CWE-502 (first CNA problemType, per TestCVEListParsesLog4Shell), not CWE-20. Also reword the 04-CHALLENGES weight aside to match.
Five teaching docs (overview, concepts, architecture, implementation, challenges) grounded in real incidents (Log4Shell CVE-2021-44228, Citrix Bleed 2 CVE-2025-5777, Equifax CVE-2017-5638, MOVEit CVE-2023-34362). ASCII pipeline + data-model diagrams, a code walkthrough by function name (no line numbers), and challenges from adding a source to replacing clustering with LSH. No em dashes, per the repo's docs style.
One-shot install.sh (curl|bash -> prebuilt release binary, or go install fallback with Go-toolchain bootstrap; installs to ~/.local/bin and wires PATH). .goreleaser.yaml + .github/workflows/release.yml build cross-platform binaries on a v* tag (inert in the monorepo, active once the project dir is the standalone repo root).
Add the AGPL-3.0 LICENSE and rewrite README.md with install + quickstart. Make internal/version.Version a var so goreleaser injects the real version via ldflags. justfile gains watch (run daemon from source) and publish (drift-free rsync mirror to the standalone github.com/CarterPerez-dev/nadezhda checkout).
Add a watch command (--interval, --once, --no-enrich): a long-lived daemon that re-ingests every enabled source on a ticker and, when a webhook is configured, POSTs genuinely new high-signal stories to it (Slack, Discord, or any JSON endpoint via a text/content/items payload).
internal/watch is a pure, stdlib-only scheduler: the pipeline is injected as a Cycle closure, so the daemon is fully unit-testable with a fake ticker and notifier. Graceful shutdown on SIGINT/SIGTERM returns cleanly; a cycle error is logged and the loop continues (fail-soft).
Notify uses a dedicated fetch-time watermark (store.NewlyFetchedClusters) so a freshly ingested but older-dated advisory is surfaced, which a publish-time filter would drop. Notable stories are capped by watch.notify_max_items and filtered by score threshold or KEV status.
Extract the shared ingest and cluster sequence (pipeline.go) so scrape and watch cannot drift, and route scrape and enrich through cmd.Context() so they honor SIGTERM too.
Add an opt-in ideation layer that turns ranked story clusters into content angles. A Provider interface backs four models: qwen (default, local Ollama), OpenAI, and Gemini share one OpenAI-compatible client; an Anthropic Messages client is written against raw net/http (no vendor SDK, uniform + dep-light). Results persist in ai_notes with a UNIQUE(cluster_id, provider) index (migration 0003), so re-ideation upserts in place.
The ideate command skips any (cluster, provider) that already has a note unless --force is passed; benign classifier refusals get their own counter and never fail the run. In the TUI, i ideates the selected cluster inline and renders an AI IDEAS section in the dossier; it is a no-op unless AI is enabled and only acts in the detail view.
Add a re-runnable interactive wizard that configures an AI provider for the ideation layer: Claude, OpenAI, Gemini, or a local Ollama. Keys persist to ~/.config/nadezhda/credentials (0600, atomic temp+rename) and auto-load on every run; a shell-exported key always wins over the file. A key-name allowlist (NADEZHDA_* / *_API_KEY) blocks env-name injection into spawned subprocesses.
The Ollama path detects a running instance and auto-configures it, else writes an embedded compose file plus one docker command, else prints an install hint. Ollama's host port is remapped to a non-default 39847 (OLLAMA_HOST_PORT) to avoid colliding with an existing Ollama on 11434; the container-internal port stays 11434.
Thread a signal-cancellable root context through cobra (ExecuteContext). Default the Anthropic model to claude-sonnet-4-6.
This is a security-news tool; CVE enrichment is optional garnish, never a
setup tax. Drop the NVD API key from the default path entirely.
- internal/cve/cvelist: keyless CVE Program cvelistV5 client (raw GitHub,
carries CISA-ADP / vulnrichment CVSS). Parses cna and adp metrics with
version precedence v4.0 > v3.1 > v3.0 > v2.0, plus CWE and description;
404 maps to not-found; v2 severity is capped at HIGH. Introduces a
neutral CoreResult and a CVESource interface (NVDResult renamed to
CoreResult; NVDClient still satisfies it).
- enrich: Clients.Core is the CVESource. buildCoreSource picks the keyless
cvelist client by default and NVD only when NVD_API_KEY is set (booster).
- scrape auto-enriches referenced CVEs best-effort after clustering
(time-bounded, non-fatal, --no-enrich to skip); enrich kept as an
optional manual refresh. Default flow is scrape -> tui.
- ranking default weights rebalanced news-first (recency / velocity /
source / keyword 70 percent, cvss / kev / epss 30 percent; was 55 percent CVE).
- KAT fixture testdata/cvelist/CVE-2021-44228.json.
Interactive terminal UI over the ranked story clusters. Rave Maximal
cybercore theming: cyan-to-magenta spectrum score bars, severity-reactive
rows, hot KEV chips, cool-ramp outlet dots, and a scrollable detail
dossier with per-outlet links and full CVE cards (CVSS meter, KEV /
ransomware, EPSS, CWE, vector, description). Reuses store.DigestClusters
and rank.Rank verbatim and preloads full store.CVE records for detail.
- internal/tui: model/update/view state machine with an injected Loader
and opener, so Update is fully unit-testable without a terminal or DB
(nav / clamp / open / back / quit / resize / empty-store / too-small).
- o opens the selected story in the default browser, keyboard-only,
cross-platform, and refuses non-http urls.
- tui command replaces the M5 stub; supports --since.
- deps: charmbracelet bubbletea v1.3.10 + bubbles v1.0.0 + lipgloss v1.1.0.
Turn the aggregated store into a ranked, exportable digest of story clusters.
- internal/rank: pure deterministic Score(Signals, cfg.Rank) — recency
(exp half-life decay), cvss/10, kev, epss, velocity (cluster size/age), source
weight, watchlist keyword. Every signal clamped to [0,1]; all weights +
half_life + velocity_norm from config, no literals. recency/velocity are total
(guard half_life<1 and norm<=0). Rank() scores each cluster and stable-sorts
desc, tiebreak freshest. Research 06's two worked examples are the golden-order
test (A ~0.99 strictly above B ~0.13).
- internal/export: per-cluster (story) digest. Headline = freshest member, distinct
sorted outlets, CVEs sorted KEV-first then CVSS. Markdown + deterministic JSON.
- store.DigestClusters: 3-query per-cluster aggregation (clusters by since, member
articles, union of CVE signals) with explicit ORDER BY; per-cluster CVE dedup
via SELECT DISTINCT.
- digest command: --top/--since/--format md|json/--out. Off the stub list.
- Proven live: scrape -> digest --format md and json render correctly, per-cluster
(FortiBleed shows 3 outlets). Suite offline + -race.
One read-only audit agent; 0 Crit/High/Med, golden-order math verified; Low/Nit
fixed in-phase (top<=0 count, pure-fn guards, ORDER BY, headline sanitize, 2-cluster
attach test).
Digest ranks per-cluster (Carter's choice), not per-article.
The intelligence layer. scrape now extracts CVE IDs (cheap, no network) into
article_cves, which activates M2's shared-CVE clustering. A separate 'enrich'
command does the slow, cached API work (Carter's split: fast scrape, deliberate
enrich).
- internal/cve: regex CVE-ID extraction + three clients (stdlib net/http):
- nvd: API 2.0, CVSS precedence v4.0>v3.1>v3.0>v2 with nullable metrics and v2
metric-level severity fallback, totalResults==0 => not-found, apiKey header
(exact case), token-bucket rate limit (5/30s anon, 50/30s keyed) + retry on
429/5xx, timeouts non-retryable.
- kev: one catalog download -> membership map; knownRansomwareCampaignUse is the
STRING 'Known'/'Unknown' mapped explicitly to bool.
- epss: batched; epss/percentile are QUOTED STRINGS parsed with ParseFloat (a
parse failure is skipped, a legitimate 0.0 is kept); partial batches survive
a transient error.
- internal/enrich: enriches only unenriched/stale CVEs (positive + negative TTL),
KEV once per run, EPSS batched, NVD per-CVE. NVD/EPSS soft per-item (resumable),
KEV fatal by design (enriching without it would persist wrong is_kev for a TTL).
- store: UpsertCVEStub/LinkArticleCVE, CVEsNeedingEnrichment (TTL), UpdateCVEEnrichment
(nullable *float64 -> SQL NULL), GetCVE, ArticlesForCVE, ListArticles (parameterized
--source/--since/--min-cvss/--kev/--keyword via EXISTS subqueries).
- commands: enrich, cve, list (cve/list off the stub list). Trimmed kev-sample.json
(7KB) fixture for KAT tests; nvd/epss fixtures committed.
- KAT proven offline AND live: CVE-2021-44228 -> CVSS 10.0 CRITICAL (3.1), CWE-20,
KEV added 2021-12-10 ransomware yes, EPSS 0.99999. 215 articles -> 82 CVEs, and
shared-CVE clustering lit up (multi-source clusters 2 -> 5). Suite offline + -race.
One read-only audit agent run; 0 Crit/High/Med, Low/Nit fixes applied in-phase.
NVD apiKey header set with exact case to bypass Go header canonicalization.
Group the same story across outlets into clusters, the velocity signal for
ranking. Runs automatically after 'scrape' (recompute-from-scratch, idempotent).
- internal/cluster: connected-components engine (union-find). Two articles join
when, within a time window (Cluster.WindowHours, default 72h), EITHER their
normalized-title token-set Jaccard >= threshold (default 0.6) AND they are from
different outlets, OR they share >= 1 CVE (source-agnostic). The cross-outlet
guard on the title edge kills a real false positive where two distinct CISA
advisories with identical boilerplate titles were merging. Cluster key = the
earliest article by time; SourceCount tracks distinct outlets so the reported
'multi-source' metric is truthful.
- Shared-CVE join reads article_cves, which is empty until M3 CVE extraction, so
that path is a natural no-op today and lights up in M3 with no rework.
- store: ClusterCandidates (effective time = published_at else fetched_at, bounded
by lookback), ArticleCVEMap, ReplaceClusters (single atomic tx: wipe + rebuild).
- config: Cluster.LookbackHours (default 168h) bounds the corpus for O(n^2);
validated as >= window_hours so window edges are never silently dropped.
- Proven live: 215 real articles -> 137 clusters, 2 genuinely cross-outlet
(FortiBleed x3, Scattered Spider x2). 17 cluster tests; suite offline + -race.
One read-only audit agent run; findings (multi-source stat, lookback guard) fixed in-phase.
Clustering is recompute-from-scratch each scrape; safe to re-run.
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.
- internal/fetch: per-host token-bucket rate limiting, conditional GET
(ETag/Last-Modified with 304 handling), retry+backoff that honors
Retry-After and treats timeouts/cancellation as non-retryable, per-source
deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
Client.Allowed() for the future HTML article-scrape path, not on subscribed
feed fetches (user-directed retrieval; several publishers blanket-disallow
generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
-> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
suite is offline and passes under -race. Proven live: cold run 215 new across
7 sources, warm run 0 new (304s + dedup).
Ctrl-C aborts scrape cleanly via signal-aware context.
Go engine that aggregates cybersecurity news and enriches CVEs. This
first commit lays the M0 foundation:
- config: fetch/enrich/cluster/rank/AI settings, YAML + validation,
defaults as named constants, AI opt-in and off by default
- source: Source registry with embedded default feeds (7 verified RSS
sources), external-file override, dedup/URL validation
- store: pure-Go SQLite (modernc), WAL + foreign_keys, forward-only
embedded migrations with schema_migrations tracking and a loud
newer-than-binary guard; ErrDuplicate sentinel for dedup
- cmd: cobra skeleton, version + sources wired, remaining commands
stubbed to their milestones
go vet / gofmt / go test all clean.
dependabot[bot]
CarterPerez-dev