docs(nadezhda): rewrite README in the house style
Match the ja3-ja4 / zingela README convention: ANSI Shadow banner, badge row (Project 37, Go, keyless, single static binary, AGPL), blockquote pitch, then Why / What Works Today / Quick Start with real sample output / just TIP / Learn table / Architecture diagram / Build and Test / Project Structure. Drops the phantom just-watch reference (no such recipe). Content tracks the verified code: news-first 70/30 weights, Log4Shell KAT as CWE-502.
This commit is contained in:
CarterPerez-dev
parent
679a86d593
commit
d511634ccd
|
|
@ -1,112 +1,197 @@
|
|||
# nadezhda
|
||||
|
||||
Security news and CVE intelligence, aggregated in your terminal. Keyless by default.
|
||||
|
||||
Nadezhda pulls cybersecurity news from reliable RSS/Atom feeds, clusters the same
|
||||
story across outlets, enriches every referenced CVE with authoritative exploit
|
||||
intelligence, ranks what actually matters, and hands you a browsable dossier. It
|
||||
ships as a single static binary with a local SQLite store, a colorful terminal UI,
|
||||
Markdown/JSON export, an optional AI ideation layer, and an optional watch daemon.
|
||||
|
||||
No API key is required to run it. CVE enrichment comes from keyless authoritative
|
||||
sources (the CVE Program list, CISA KEV, and FIRST EPSS); an NVD key is an optional
|
||||
booster, never a requirement.
|
||||
|
||||
## Install
|
||||
<!-- ©AngelaMos | 2026 -->
|
||||
<!-- README.md -->
|
||||
|
||||
```json
|
||||
███╗ ██╗ █████╗ ██████╗ ███████╗███████╗██╗ ██╗██████╗ █████╗
|
||||
████╗ ██║██╔══██╗██╔══██╗██╔════╝╚══███╔╝██║ ██║██╔══██╗██╔══██╗
|
||||
██╔██╗ ██║███████║██║ ██║█████╗ ███╔╝ ███████║██║ ██║███████║
|
||||
██║╚██╗██║██╔══██║██║ ██║██╔══╝ ███╔╝ ██╔══██║██║ ██║██╔══██║
|
||||
██║ ╚████║██║ ██║██████╔╝███████╗███████╗██║ ██║██████╔╝██║ ██║
|
||||
╚═╝ ╚═══╝╚═╝ ╚═╝╚═════╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
[](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/intermediate/security-news-scraper)
|
||||
[](https://go.dev)
|
||||
[](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)
|
||||
[](https://pkg.go.dev/modernc.org/sqlite)
|
||||
[](https://www.gnu.org/licenses/agpl-3.0)
|
||||
|
||||
> A keyless security-news and CVE intelligence engine in Go. Point it at a set of RSS and Atom feeds and it fetches them politely, clusters the same story as it breaks across outlets, extracts every CVE it mentions, enriches each one with authoritative exploit intelligence, ranks the whole set by what actually matters, and hands you a browsable dossier in your terminal. It ships as a single static binary over a local SQLite store, needs no API key to do any of it, and treats the news as the product with the CVE data as the intelligence that sharpens it.
|
||||
|
||||
## Why aggregate security news
|
||||
|
||||
There is more security news published every day than any person can read, and almost all of it is noise relative to the one story that matters to you right now. The hard problem is not gathering headlines. It is deciding which three of the four hundred you should act on before lunch.
|
||||
|
||||
The signal is usually some combination of the same few factors. A vulnerability that many outlets picked up in the same few hours is trending for a reason. A CVE that CISA just added to its Known Exploited Vulnerabilities catalog is being used against real targets today. A flaw with a high EPSS score is one the rest of the world is about to start exploiting. Nadezhda computes that combination and sorts by it, so a CVSS 6.5 that is on the KEV catalog can outrank a CVSS 9.8 that nobody has touched.
|
||||
|
||||
The cost of missing that signal is not hypothetical. Log4Shell went from a GitHub issue to every security outlet on earth within hours and landed on the KEV catalog almost immediately; a tool that ranks by cross-outlet velocity plus KEV plus EPSS floats it to the top of the first scrape. Equifax fell to a known Apache Struts flaw that had a patch available for months, and the intelligence to prioritize it sat in public feeds the whole time, so the failure was one of triage, not of information. MOVEit was mass-exploited in a tight window where the advisories, the KEV listing, and the coverage all arrived together. Velocity was the tell.
|
||||
|
||||
## What Works Today
|
||||
|
||||
This is not a stub. It ingests real feeds, enriches real CVEs from keyless authoritative sources, clusters and ranks real coverage, and surfaces it four ways. Every capability below is exercised by unit tests, offline fixture-driven tests, and a live run against the real feeds.
|
||||
|
||||
**Ingestion**
|
||||
- Seven seeded RSS/Atom feeds (Krebs on Security, The Hacker News, BleepingComputer, SecurityWeek, Dark Reading, The Register, CISA), fetched concurrently through a per-host rate limiter with an honest User-Agent
|
||||
- Conditional GET: an unchanged feed answers `304 Not Modified` and costs almost nothing, and a retry honors `Retry-After` and never retries a timeout or a cancellation
|
||||
- Fail-soft by construction: one broken feed is reported at the end and never aborts the run, and every article deduplicates at the database on both its canonical URL and a content hash
|
||||
|
||||
**Enrichment, keyless by default**
|
||||
- CVE core from the CVE Program's cvelistV5 records: a CVSS score resolved by a fixed version precedence (v4.0, then v3.1, then v3.0, then v2.0) read across both the vendor and the CISA-ADP containers, plus CWE and description
|
||||
- CISA KEV membership, with the `knownRansomwareCampaignUse` string mapped explicitly to a real ransomware signal rather than treated as a boolean
|
||||
- FIRST EPSS probability and percentile, parsed from the quoted strings the API actually returns so the score never silently reads as zero
|
||||
- A TTL cache with a separate, shorter negative TTL, so a not-yet-published CVE is not re-fetched every run; NVD 2.0 is supported as an optional key-gated booster, never a requirement
|
||||
|
||||
**Clustering and ranking**
|
||||
- Connected-components (union-find) clustering: two items join when they share a CVE, or when they come from different outlets and their title token sets clear a Jaccard threshold inside a time window
|
||||
- A pure, deterministic, news-first weighted score: recency, cross-outlet velocity, source trust, and a keyword watchlist carry 70 percent of the weight, with KEV, CVSS, and EPSS as the supporting 30, so the same corpus always sorts the same way and is pinned by golden-order tests
|
||||
|
||||
**Surfaces**
|
||||
- A colorful bubbletea terminal UI: a ranked list of dossiers with severity-reactive rows and KEV chips, a scrollable detail view with every outlet link and the full CVE card, and `o` to open a story in your browser
|
||||
- A Markdown or JSON digest of the top-ranked clusters, for feeding a newsletter, a report, or another tool
|
||||
- A single-CVE lookup that shows the enriched record and every stored story that mentions it
|
||||
|
||||
**Content ideation, optional AI, off by default**
|
||||
- Turn ranked clusters into summaries and content angles through one `Provider` interface: a local Qwen model via Ollama (keyless default), or Claude, OpenAI, or Gemini behind a single pasted key
|
||||
- A re-runnable in-binary setup wizard writes keys to a `0600` credentials file behind a name allowlist, read from the environment and never logged
|
||||
|
||||
**Watch daemon, optional**
|
||||
- A long-lived scheduler that re-ingests on a timer and posts genuinely new, high-signal stories to Slack, Discord, or any JSON webhook
|
||||
- New is measured on fetch time, not publish time, so a backfilled advisory fetched today still alerts exactly once; the daemon shuts down cleanly on Ctrl-C or SIGTERM and never crashes on a transient feed or network error
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
curl -fsSL https://angelamos.com/nadezhda/install.sh | bash
|
||||
```
|
||||
|
||||
One command, zero further steps: it grabs a prebuilt binary for your platform (no Go
|
||||
toolchain needed), drops it on your `PATH`, and leaves `nadezhda` runnable by name.
|
||||
One command, zero further steps: it grabs a prebuilt binary for your platform (no Go toolchain needed), drops it on your `PATH`, and leaves `nadezhda` runnable by name. Then pull the news and browse it:
|
||||
|
||||
Prefer the Go toolchain? A real `go install` works too:
|
||||
|
||||
```
|
||||
go install github.com/CarterPerez-dev/nadezhda/cmd/nadezhda@latest
|
||||
```bash
|
||||
nadezhda scrape # pull every enabled feed, cluster, and enrich CVEs
|
||||
nadezhda tui # browse the ranked dossier (? for keys, o opens a story)
|
||||
nadezhda digest --top 20 # render a ranked Markdown digest (or --format json)
|
||||
nadezhda cve CVE-2021-44228 # one enriched CVE and the stories that mention it
|
||||
```
|
||||
|
||||
Or build from a clone:
|
||||
The default flow is `scrape` then `tui`; scrape already runs enrichment best-effort, so there is nothing else to wire up. A scrape prints a per-source table and a one-line summary:
|
||||
|
||||
```
|
||||
git clone https://github.com/CarterPerez-dev/nadezhda
|
||||
cd nadezhda
|
||||
just build # -> ./nadezhda (or: go build -o nadezhda ./cmd/nadezhda)
|
||||
SOURCE STATUS PARSED NEW DUP CVE ERR
|
||||
krebs ok 10 3 7 1 0
|
||||
bleepingcomputer ok 18 6 12 4 0
|
||||
cisa 304 - - - - -
|
||||
...
|
||||
12 new, 45 duplicate, 8 CVE refs across 7 sources (0 failed)
|
||||
132 clusters (14 multi-source, largest 5)
|
||||
enriched 80/80 CVEs (14 KEV, 0 not found)
|
||||
```
|
||||
|
||||
Requires Go 1.25+ only when building from source; the toolchain is fetched
|
||||
automatically if you are on an older Go.
|
||||
The optional layers are one command each:
|
||||
|
||||
## Quick start
|
||||
|
||||
```
|
||||
nadezhda scrape # pull every enabled feed, cluster, enrich CVEs
|
||||
nadezhda tui # browse the ranked dossier (press ? for keys, o to open a story)
|
||||
nadezhda digest --top 20 # render a ranked digest to Markdown (or --format json)
|
||||
nadezhda list --kev # list stored articles, filtered
|
||||
nadezhda cve CVE-2021-44228 # show one enriched CVE and the stories that mention it
|
||||
```bash
|
||||
nadezhda ai # set up AI ideation: paste one key, or point at a local Ollama
|
||||
nadezhda watch --interval 1h # run as a daemon, alerting new high-signal stories to a webhook
|
||||
```
|
||||
|
||||
The default flow is `scrape` then `tui`. Scrape already runs enrichment (best effort,
|
||||
never blocking the news), so there is nothing else to wire up.
|
||||
Prefer the Go toolchain? A real `go install github.com/CarterPerez-dev/nadezhda/cmd/nadezhda@latest` works too, and `just build` builds from a checkout. Building from source needs Go 1.25+, fetched automatically if you are on an older Go.
|
||||
|
||||
## Content ideation (optional AI)
|
||||
> [!TIP]
|
||||
> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` to see every recipe.
|
||||
>
|
||||
> Install: `curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin`
|
||||
|
||||
Turn ranked clusters into summaries and content angles. Run the built-in setup wizard
|
||||
once and paste a single key, or point it at a local Ollama:
|
||||
## Learn
|
||||
|
||||
This project ships a full teaching track. Read it in order, or jump to what you need.
|
||||
|
||||
| Doc | What it covers |
|
||||
|-----|----------------|
|
||||
| [`learn/00-OVERVIEW.md`](learn/00-OVERVIEW.md) | What the tool is, prerequisites, the project layout, and a quick tour |
|
||||
| [`learn/01-CONCEPTS.md`](learn/01-CONCEPTS.md) | The CVE intelligence stack, clustering as a velocity signal, and the ranking model, grounded in real incidents |
|
||||
| [`learn/02-ARCHITECTURE.md`](learn/02-ARCHITECTURE.md) | The pipeline, the package map, the data model, and the design decisions |
|
||||
| [`learn/03-IMPLEMENTATION.md`](learn/03-IMPLEMENTATION.md) | A code walkthrough from a feed item to a ranked, enriched cluster |
|
||||
| [`learn/04-CHALLENGES.md`](learn/04-CHALLENGES.md) | Extension ideas, from adding a source to replacing the clustering with LSH |
|
||||
|
||||
## Architecture
|
||||
|
||||
One Go module, no service boundary, no message queue, and no external database. The `scrape` command runs the whole left-to-right flow once; the `watch` daemon runs it on a timer. Everything in `internal` exists to feed one function, `ingestAndCluster`, and everything downstream exists to rank and present what it produced.
|
||||
|
||||
```
|
||||
nadezhda ai # interactive, re-runnable: Claude / OpenAI / Gemini / Ollama
|
||||
nadezhda ideate --top 5 # generate angles for the top clusters
|
||||
sources.yaml (embedded default, or a user --config file)
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ fetch N workers, per-host rate limit, │ internal/fetch
|
||||
│ conditional GET (ETag / 304) │
|
||||
└───────────────────────┬──────────────────────┘
|
||||
│ raw feed bytes
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ parse → normalize → ingest │ parse / normalize / ingest
|
||||
│ RSS/Atom, canonical URL, hash, fail-soft │
|
||||
└───────────────────────┬──────────────────────┘
|
||||
CVE regex │ union-find
|
||||
┌──────────┴──────────┐
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ enrich CVE list / KEV / EPSS, cached, │ enrich / cve
|
||||
│ keyless │
|
||||
└───────────────────────┬──────────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ rank deterministic, news-first score │ internal/rank
|
||||
└───────────────────────┬──────────────────────┘
|
||||
┌────────────┼────────────┬───────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
digest tui ideate watch
|
||||
(export) (browse) (AI) (daemon)
|
||||
```
|
||||
|
||||
The default provider is a local Qwen model via Ollama (keyless). Keys, when used, are
|
||||
read from your environment and a `0600` credentials file; they are never logged. With
|
||||
AI disabled, nadezhda is a complete aggregator.
|
||||
**Design decisions:** the store is `modernc.org/sqlite`, a pure-Go SQLite with no CGO, so the four-platform release binaries fall out of one build and the core needs no database server. Enrichment folds into `scrape` because the keyless CVE source is unthrottled, and it runs best-effort under a timeout so the news never blocks or fails on a slow CVE lookup. The `watch` daemon is a pure scheduler that imports nothing from the store or the pipeline; the concrete work is injected as a closure, so the loop, the graceful shutdown, and the fail-soft behavior are all unit-tested with a fake clock and no network. Alerts use a fetch-time watermark, not publish time, so an advisory published last week but fetched today is correctly new to you.
|
||||
|
||||
## Watch daemon (optional)
|
||||
## Build and Test
|
||||
|
||||
Run nadezhda as a long-lived daemon that re-ingests on an interval and, when a webhook
|
||||
is configured, posts genuinely new high-signal stories to Slack, Discord, or any JSON
|
||||
endpoint:
|
||||
|
||||
```
|
||||
nadezhda watch --interval 1h
|
||||
nadezhda watch --once # a single cycle, for cron
|
||||
```bash
|
||||
just build # -> ./nadezhda (or: go build -o nadezhda ./cmd/nadezhda)
|
||||
just test # go test ./... (18 packages)
|
||||
just ollama-up # stand up the local Qwen runtime in Docker (optional AI)
|
||||
```
|
||||
|
||||
Configure `watch.webhook_url` (and thresholds) in `config.yaml`. The daemon shuts down
|
||||
cleanly on Ctrl-C / SIGTERM and never crashes on a transient feed or network error.
|
||||
Ranking is a pure function pinned to golden-order tests: fixed inputs produce one exact ordering. The keyless CVE client is pinned to a known-answer fixture, Log4Shell (CVE-2021-44228): the CVSS 10.0 score is read out of the CISA-ADP container because the vendor container holds only a placeholder, the weakness resolves to CWE-502, it is KEV-listed, and its EPSS score sits near the top of the scale. The `knownRansomwareCampaignUse` string mapping and the quoted-string EPSS parsing each have a regression test, because both are the quiet kind of bug that decodes wrong with no error. The watch scheduler is tested with a fake ticker a test drives by hand, so its behavior is verified with no clock, no network, and no database.
|
||||
|
||||
## How it works
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
feeds -> fetch (rate-limited, conditional GET) -> parse -> normalize -> dedup
|
||||
-> cluster (cross-outlet + shared-CVE) -> enrich (CVE list / KEV / EPSS)
|
||||
-> rank (news-first weighted score) -> tui | digest | ideate | watch
|
||||
```
|
||||
|
||||
Ranking is pure and deterministic: recency, cross-outlet velocity, source trust, and a
|
||||
keyword watchlist dominate, with CVSS / KEV / EPSS as supporting signals. Everything is
|
||||
configurable in `config.yaml`; nothing is a magic constant in the code.
|
||||
|
||||
## Configuration
|
||||
|
||||
Nadezhda runs with sensible defaults and an embedded source list. To customize, pass
|
||||
`--config config.yaml`. Notable keys: `watchlist`, `fetch.*`, `cluster.*`, `rank.*`,
|
||||
`ai.*`, and `watch.*`. The Ollama host port defaults to a non-standard `39847`
|
||||
(override with `OLLAMA_HOST_PORT`) so it never collides with an existing Ollama.
|
||||
|
||||
## Development
|
||||
|
||||
```
|
||||
just build # build the binary
|
||||
just test # go test ./...
|
||||
just watch # run the daemon from source
|
||||
just ollama-up # stand up the local Qwen runtime (Docker)
|
||||
security-news-scraper/
|
||||
├── cmd/nadezhda/ # the CLI: one file per command + the shared pipeline seam
|
||||
│ ├── scrape.go # the whole tool in miniature: runScrape
|
||||
│ ├── pipeline.go # ingestAndCluster, the seam shared by scrape and watch
|
||||
│ ├── tui.go # digest.go ideate.go watch.go cve.go list.go sources.go ai.go
|
||||
│ └── main.go # root.go, the cobra command tree
|
||||
├── internal/
|
||||
│ ├── fetch/ # concurrent, rate-limited, conditional HTTP (+ an unused robots gate)
|
||||
│ ├── parse/ # RSS/Atom via gofeed, HTML fallback via goquery
|
||||
│ ├── normalize/ # canonical URLs, content hashing, time parsing
|
||||
│ ├── ingest/ # the fan-out orchestrator, fail-soft per source
|
||||
│ ├── cluster/ # union-find clustering by title similarity and shared CVE
|
||||
│ ├── cve/ # keyless clients: cvelistV5, CISA KEV, FIRST EPSS (+ optional NVD)
|
||||
│ ├── enrich/ # enrichment orchestration and the TTL cache
|
||||
│ ├── rank/ # the pure, deterministic weighted score
|
||||
│ ├── store/ # SQLite: connection, migrations, and every typed query
|
||||
│ ├── ai/ # the opt-in ideation layer, four providers behind one interface
|
||||
│ ├── setup/ # the in-binary credential wizard
|
||||
│ ├── watch/ # the daemon scheduler, decoupled from the work it runs
|
||||
│ ├── tui/ # the bubbletea terminal browser
|
||||
│ ├── export/ # the Markdown and JSON digest renderers
|
||||
│ ├── source/ # the source registry and the embedded sources.yaml
|
||||
│ ├── config/ # every tunable, with defaults and validation
|
||||
│ └── version/ # the ldflags-injected version string
|
||||
├── testdata/ # captured feed and API fixtures, driving the offline tests
|
||||
├── install.sh # the one-shot curl-able installer
|
||||
├── learn/ # the teaching track (public)
|
||||
└── justfile # every recipe
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
AGPL-3.0. See [LICENSE](LICENSE).
|
||||
[AGPL 3.0](LICENSE).
|
||||
|
|
|
|||
Loading…
Reference in New Issue