Cybersecurity-Projects/PROJECTS/intermediate/security-news-scraper/README.md

113 lines
4.0 KiB
Markdown

# nadezhda
Security news and CVE intelligence, aggregated in your terminal. Keyless by default.
Nadezhda pulls cybersecurity news from reliable RSS/Atom feeds, clusters the same
story across outlets, enriches every referenced CVE with authoritative exploit
intelligence, ranks what actually matters, and hands you a browsable dossier. It
ships as a single static binary with a local SQLite store, a colorful terminal UI,
Markdown/JSON export, an optional AI ideation layer, and an optional watch daemon.
No API key is required to run it. CVE enrichment comes from keyless authoritative
sources (the CVE Program list, CISA KEV, and FIRST EPSS); an NVD key is an optional
booster, never a requirement.
## Install
```
curl -fsSL https://angelamos.com/nadezhda/install.sh | bash
```
One command, zero further steps: it grabs a prebuilt binary for your platform (no Go
toolchain needed), drops it on your `PATH`, and leaves `nadezhda` runnable by name.
Prefer the Go toolchain? A real `go install` works too:
```
go install github.com/CarterPerez-dev/nadezhda/cmd/nadezhda@latest
```
Or build from a clone:
```
git clone https://github.com/CarterPerez-dev/nadezhda
cd nadezhda
just build # -> ./nadezhda (or: go build -o nadezhda ./cmd/nadezhda)
```
Requires Go 1.25+ only when building from source; the toolchain is fetched
automatically if you are on an older Go.
## Quick start
```
nadezhda scrape # pull every enabled feed, cluster, enrich CVEs
nadezhda tui # browse the ranked dossier (press ? for keys, o to open a story)
nadezhda digest --top 20 # render a ranked digest to Markdown (or --format json)
nadezhda list --kev # list stored articles, filtered
nadezhda cve CVE-2021-44228 # show one enriched CVE and the stories that mention it
```
The default flow is `scrape` then `tui`. Scrape already runs enrichment (best effort,
never blocking the news), so there is nothing else to wire up.
## Content ideation (optional AI)
Turn ranked clusters into summaries and content angles. Run the built-in setup wizard
once and paste a single key, or point it at a local Ollama:
```
nadezhda ai # interactive, re-runnable: Claude / OpenAI / Gemini / Ollama
nadezhda ideate --top 5 # generate angles for the top clusters
```
The default provider is a local Qwen model via Ollama (keyless). Keys, when used, are
read from your environment and a `0600` credentials file; they are never logged. With
AI disabled, nadezhda is a complete aggregator.
## Watch daemon (optional)
Run nadezhda as a long-lived daemon that re-ingests on an interval and, when a webhook
is configured, posts genuinely new high-signal stories to Slack, Discord, or any JSON
endpoint:
```
nadezhda watch --interval 1h
nadezhda watch --once # a single cycle, for cron
```
Configure `watch.webhook_url` (and thresholds) in `config.yaml`. The daemon shuts down
cleanly on Ctrl-C / SIGTERM and never crashes on a transient feed or network error.
## How it works
```
feeds -> fetch (rate-limited, conditional GET) -> parse -> normalize -> dedup
-> cluster (cross-outlet + shared-CVE) -> enrich (CVE list / KEV / EPSS)
-> rank (news-first weighted score) -> tui | digest | ideate | watch
```
Ranking is pure and deterministic: recency, cross-outlet velocity, source trust, and a
keyword watchlist dominate, with CVSS / KEV / EPSS as supporting signals. Everything is
configurable in `config.yaml`; nothing is a magic constant in the code.
## Configuration
Nadezhda runs with sensible defaults and an embedded source list. To customize, pass
`--config config.yaml`. Notable keys: `watchlist`, `fetch.*`, `cluster.*`, `rank.*`,
`ai.*`, and `watch.*`. The Ollama host port defaults to a non-standard `39847`
(override with `OLLAMA_HOST_PORT`) so it never collides with an existing Ollama.
## Development
```
just build # build the binary
just test # go test ./...
just watch # run the daemon from source
just ollama-up # stand up the local Qwen runtime (Docker)
```
## License
AGPL-3.0. See [LICENSE](LICENSE).