RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

docs(nadezhda): M8 learn/ folder 

Five teaching docs (overview, concepts, architecture, implementation, challenges) grounded in real incidents (Log4Shell CVE-2021-44228, Citrix Bleed 2 CVE-2025-5777, Equifax CVE-2017-5638, MOVEit CVE-2023-34362). ASCII pipeline + data-model diagrams, a code walkthrough by function name (no line numbers), and challenges from adding a source to replacing clustering with LSH. No em dashes, per the repo's docs style.
This commit is contained in:
CarterPerez-dev 2026-07-07 02:37:39 -04:00
parent a27cae6ab7
commit 5ece03699b
5 changed files with 604 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

143
PROJECTS/intermediate/security-news-scraper/learn/00-OVERVIEW.md Normal file
View File

@ -0,0 +1,143 @@
<!-- ©AngelaMos | 2026 -->
<!-- 00-OVERVIEW.md -->
# Nadezhda: Overview
## What This Is
A security news and CVE intelligence engine written in Go. You point it at a set of RSS feeds, and it fetches them politely, clusters the same story as it appears across different outlets, extracts every CVE mentioned, enriches each one with authoritative exploit intelligence, ranks the whole set by real world significance, and hands you a browsable dossier in your terminal. It ships as a single static binary with a local SQLite store, and it needs no API key to do any of this.
The point of the project is to understand, by building it, how you turn a firehose of security headlines into a ranked signal. You get a working aggregator you can run against real feeds, a keyless enrichment pipeline seeded from the same authoritative sources a SOC would use, and a codebase small enough to read in an afternoon.
## Why This Matters
There is more security news published every day than any person can read, and almost all of it is noise relative to the one story that matters to you right now. The hard problem is not gathering headlines. It is deciding which three of the four hundred you should act on before lunch.
The signal is usually some combination of the same factors. A vulnerability that many outlets picked up in the same few hours is trending for a reason. A CVE that CISA just added to its Known Exploited Vulnerabilities catalog is being used against real targets today. A flaw with a high EPSS score is one the rest of the world is about to start exploiting. Nadezhda exists to compute that combination for you and sort by it.
The cost of missing the signal is not hypothetical.
- **Log4Shell, December 2021.** CVE-2021-44228, a remote code execution flaw in Apache Log4j 2, went from a GitHub issue to every security outlet on earth within hours. Its CVSS base score is a perfect 10.0, it landed on the CISA KEV catalog almost immediately, and its EPSS score sat near the top of the scale. A tool that ranks by cross outlet velocity plus KEV plus EPSS would have floated it to the top of the list on the first scrape, which is exactly the behavior this project is built to produce.
- **The Equifax breach, 2017.** Attackers walked in through a known Apache Struts vulnerability, CVE-2017-5638, that had a patch available for months. The information needed to prioritize that patch existed in public feeds the whole time. The failure was one of triage, not of intelligence.
- **MOVEit, 2023.** The Cl0p group mass exploited CVE-2023-34362 in the MOVEit Transfer product and hit hundreds of organizations. The advisories, the KEV listing, and the outlet coverage all arrived in a tight window. Velocity was the tell.
**Real world scenarios where this applies:**
- **Threat intelligence triage.** An analyst runs one scrape each morning and reads the top of a ranked dossier instead of forty browser tabs.
- **Content and research.** A writer or educator finds the cluster every outlet is covering, sees the CVEs and the exploit signals behind it, and turns it into a post or a lesson.
- **Personal situational awareness.** An engineer keeps a watchlist of the vendors and products they run, and gets alerted when a KEV listed flaw in one of them starts trending.
## What You'll Learn
This project teaches how a real aggregation and prioritization pipeline is built. By building it yourself, you will understand:
**Security concepts:**
- **The CVE intelligence stack.** How CVSS, CWE, the CVE Program record, the CISA KEV catalog, and FIRST EPSS each answer a different question about a vulnerability, and how to combine them without an API key.
- **Why KEV and EPSS beat CVSS alone.** A CVSS 9.8 that nobody is exploiting is less urgent than a CVSS 6.5 that is on the KEV catalog. The project encodes that judgment in its ranking.
- **Cross source clustering as a signal.** Why the same story appearing across many outlets quickly is itself the most useful thing you can measure, and how to measure it.
- **Polite automated retrieval.** Conditional requests, per host rate limiting, and where robots.txt does and does not apply.
**Technical skills:**
- **Concurrent, fail soft ingestion.** Fetching many feeds in parallel where one broken feed never aborts the run.
- **Deterministic ranking.** Building a scoring model whose output is a fixed function of its inputs, so the same corpus always sorts the same way and can be tested against golden order.
- **A single static binary with an embedded database.** Using pure Go SQLite so there is no CGO, no external database, and no runtime dependency.
- **A decoupled daemon.** Designing a scheduler that knows nothing about the work it runs, so it can be unit tested with a fake clock.
**Tools and techniques:**
- **`gofeed`** for RSS and Atom parsing, and **`goquery`** for the HTML fallback path.
- **`modernc.org/sqlite`**, a cgo free SQLite, in WAL mode so a reader and a writer coexist.
- **`bubbletea`** and **`lipgloss`** for the terminal UI.
- **Ollama** for a local, keyless language model, with OpenAI, Gemini, and Anthropic available as opt in alternatives.
## Prerequisites
You do not need prior threat intelligence experience. You do need some comfort with the following.
**Required knowledge:**
- **Go basics.** Structs, interfaces, goroutines, and errors. If you can read a `for` loop over a channel you can read this code.
- **SQL at a beginner level.** The store is plain SQLite with a handful of tables and hand written queries.
- **What a CVE is.** That CVE-2021-44228 names one vulnerability, that CVSS scores its severity, and that a patch usually exists. The rest is explained in [01-CONCEPTS.md](./01-CONCEPTS.md).
**Tools you'll need:**
- **A Go toolchain**, 1.25 or newer. The `install.sh` script installs one for you if it is missing.
- **Nothing else to run the core.** No API keys, no database server, no Docker. The default enrichment sources are all keyless.
**Helpful but not required:**
- **Docker**, only if you want the optional local language model, which runs as an Ollama container.
- A skim of the [CISA KEV catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog) and the [FIRST EPSS](https://www.first.org/epss/) documentation.
## Quick Start
```bash
# Install (grabs a prebuilt binary, no Go needed):
curl -fsSL https://angelamos.com/nadezhda/install.sh | bash
# or, with a Go toolchain:
go install github.com/CarterPerez-dev/nadezhda/cmd/nadezhda@latest
# Pull the latest news, cluster it, and enrich every CVE:
nadezhda scrape
# Browse the ranked dossier:
nadezhda tui
# Render a Markdown digest of the top stories:
nadezhda digest --top 20
```
Expected output: `scrape` prints a per source table (parsed, new, duplicate, CVE counts), then a line like `132 clusters (14 multi-source, largest 5)`, then an enrichment summary like `enriched 80/80 CVEs (14 KEV, 0 not found)`. `tui` opens a colored, scrollable list where each story shows its rank, its outlets, and its worst CVE, and pressing a key opens the full dossier for the selected cluster.
## Project Structure
```
security-news-scraper/
├── cmd/nadezhda/ # the CLI: one file per command, plus the shared pipeline seam
├── internal/
│ ├── fetch/ # concurrent, rate-limited, conditional HTTP
│ ├── parse/ # RSS/Atom via gofeed, HTML fallback via goquery
│ ├── normalize/ # canonical URLs, content hashing, time parsing
│ ├── ingest/ # the fan-out orchestrator (fail soft per source)
│ ├── cluster/ # union-find clustering by title and shared CVE
│ ├── cve/ # keyless CVE clients: CVE list, KEV, EPSS (+ optional NVD)
│ ├── enrich/ # enrichment orchestration and the TTL cache
│ ├── rank/ # the deterministic weighted scoring model
│ ├── store/ # SQLite: connection, migrations, typed queries
│ ├── ai/ # the opt-in ideation layer (four providers)
│ ├── setup/ # the in-binary credential wizard
│ ├── watch/ # the daemon scheduler (pure, injected pipeline)
│ ├── tui/ # the bubbletea browser
│ └── export/ # Markdown and JSON digest renderers
├── testdata/ # captured feed and API fixtures (drive offline tests)
├── install.sh # the one-shot installer
└── justfile
```
The single most important thing to understand first is the pipeline in `cmd/nadezhda/pipeline.go` and `scrape.go`. Everything in `internal` exists to feed one function, `ingestAndCluster`, and everything downstream exists to rank and present what it produced.
## Next Steps
1. **Understand the ideas.** Read [01-CONCEPTS.md](./01-CONCEPTS.md) for the CVE intelligence stack, clustering as a signal, and the ranking model, each grounded in a real incident.
2. **See the design.** Read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) for the package layout, the keyless enrichment decision, and the data flow from feed to dossier.
3. **Walk the code.** Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) to trace one CVE from a feed item to a ranked, enriched cluster.
4. **Extend it.** Read [04-CHALLENGES.md](./04-CHALLENGES.md) for projects from "add a source" to "replace the clustering with SimHash".
## Common Issues
**`scrape` shows a source with an error**
```
theregister: fetch theregister: context deadline exceeded
```
Solution: one slow or broken feed never aborts a run. The other sources still ingest, the failed one is reported at the bottom, and the next scrape retries it. A persistent failure usually means the feed URL moved. Check `nadezhda sources`.
**`tui` says there is nothing to show**
```
no clusters yet. run: nadezhda scrape
```
Solution: the TUI reads what is already in the store. Run `nadezhda scrape` once first. Scrape enriches automatically, so there is no separate step.
**`ideate` says AI is not configured**
Solution: the AI layer is opt in and off by default. Run `nadezhda ai` to set up a provider (paste one key, or point it at a local Ollama). The aggregator is fully useful with AI disabled.
## Related Projects
If you found this interesting, look at:
- **sbom-generator-vulnerability-matcher**: the other half of the CVE story, matching the vulnerabilities in this feed against the dependencies you actually ship.
- **ja3-ja4-tls-fingerprinting**: the same shape of tool, a keyless intelligence engine over an embedded SQLite store, applied to network fingerprints instead of news.

94
PROJECTS/intermediate/security-news-scraper/learn/01-CONCEPTS.md Normal file
View File

@ -0,0 +1,94 @@
<!-- ©AngelaMos | 2026 -->
<!-- 01-CONCEPTS.md -->
# Nadezhda: Concepts
This document explains the ideas the project is built on. If you read only one file to understand *why* nadezhda works the way it does, read this one. Every concept here maps to a package you will meet in [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) and a function you will read in [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md).
## Feeds are a contract, scraping is a fallback
Publishers offer RSS and Atom feeds as a machine readable promise: here are my latest items, structured, with a title, a link, a publish time, and a summary. Nadezhda takes that promise at face value and treats feeds as the primary path. It only falls back to parsing raw HTML when a source offers no feed, and even then it does so per source and behind a switch.
This choice has an ethical edge. Fetching a feed a user configured is user directed retrieval, the same category as a browser loading a page or a feed reader polling for updates. Scraping arbitrary HTML across a site is crawler behavior. The two deserve different rules, and nadezhda enforces that difference: robots.txt is honored on the HTML article scrape path and is deliberately not applied to subscribed feed fetches. Several publishers, The Register among them, blanket disallow generic bots in robots.txt while serving a public feed for exactly this kind of reader. Respecting robots on the feed would mean refusing a document the publisher explicitly built for you.
The seven seed feeds (Krebs on Security, The Hacker News, BleepingComputer, SecurityWeek, Dark Reading, The Register, and CISA) were each verified live. A useful thing you learn immediately: only two of the seven ship the full article body in the feed. The other five are summary only. That bounds how many CVEs you can extract, because you can only find a CVE ID in text the feed actually gives you.
## The CVE intelligence stack
A CVE identifier like CVE-2021-44228 is just a name. It says "this specific vulnerability exists" and nothing about how bad it is, whether it is being exploited, or how likely exploitation is. Four different data sources answer those questions, and each answers a different one. Understanding what each is for is the core of this project.
### CVSS: how severe is it, in theory
The Common Vulnerability Scoring System turns a vulnerability's characteristics into a number from 0.0 to 10.0 and a band (LOW, MEDIUM, HIGH, CRITICAL). There are several versions in circulation. Version 2 is old. Versions 3.0 and 3.1 are the common ones. Version 4.0 arrived in 2023 and is slowly spreading. A single CVE record may carry a score in several versions, one version, or none at all.
Nadezhda resolves this with a fixed precedence: prefer v4.0, then v3.1, then v3.0, then v2.0. Newer is better, and a missing score is a real possibility the code handles rather than assumes away. This is why the stored score is a nullable value and not a plain number.
The subtle part is *where* the score lives. In the CVE Program record for **Log4Shell (CVE-2021-44228)**, the CVSS metric does not sit in the vendor's own container. Apache's container carries only a placeholder `other` metric. The real 10.0 score lives in a second container added by CISA's enrichment program. A parser that only looks at the vendor container reports "no score" for the single most famous vulnerability of the decade. Contrast **Citrix Bleed 2 (CVE-2025-5777)**, whose CVSS 4.0 score of 9.3 sits in the vendor container where you would expect it. The lesson is to look in both places and apply the precedence across everything you find.
### CWE: what kind of weakness is it
Where CVSS scores severity, the Common Weakness Enumeration classifies the *kind* of flaw. CVE-2025-5777 is CWE-125, an out of bounds read. Log4Shell enriches to CWE-20, improper input validation. CWE is what lets you say "we keep shipping the same class of bug" across many CVEs, and it is a natural axis to filter or group on.
### CISA KEV: is it being exploited right now
The single most actionable signal in the stack is the CISA Known Exploited Vulnerabilities catalog. A CVE on the KEV catalog is not a theoretical risk. CISA lists it because it has evidence of active exploitation against real targets. A CVSS 6.5 on the KEV catalog frequently deserves your attention before a CVSS 9.8 that nobody has touched.
The catalog is a single JSON download, which makes it cheap to fetch once per run and diff. It carries one field that trips up almost everyone: `knownRansomwareCampaignUse` is not a boolean. It is the string `"Known"` or `"Unknown"`. Treat it as a boolean and every entry looks the same. Map the string explicitly and you recover a genuinely useful signal, because a KEV entry with known ransomware use is about as urgent as security news gets.
### FIRST EPSS: how likely is exploitation
The Exploit Prediction Scoring System, published by FIRST, is a daily updated probability between 0 and 1 that a given CVE will be exploited in the next 30 days, plus a percentile that places it against every other scored CVE. Where KEV tells you exploitation is already happening, EPSS tells you it is coming. Log4Shell's EPSS score sat near the very top of the scale, which is what you would expect for a flaw the whole internet was scanning for.
EPSS has its own trap. In the API response, the `epss` and `percentile` values are JSON *strings*, not numbers. Parse them as floats or the score silently reads as zero, which quietly disables one of your ranking signals without any error to tell you it happened.
## Keyless by default
You can get all of this without an API key. The CVE core comes from the CVE Program's cvelistV5 repository, which publishes every record as raw JSON at a predictable, bucketed URL. CISA KEV is a public download. FIRST EPSS is a public API. None of them gate the data behind a key.
The NVD API 2.0 is the source most tutorials reach for, and nadezhda supports it, but as an optional booster rather than a requirement. Without a key it is rate limited to five requests per thirty seconds, which is painful for a batch. With a key it is faster. The design decision was to make the keyless trio the default so the tool is fully useful the moment it is installed, and to let a user who has an NVD key opt into it. A tool whose core value is locked behind a key setup step is a tool most people never finish setting up.
## Clustering turns coverage into velocity
If BleepingComputer, The Hacker News, and Krebs all publish about the same flaw in the same afternoon, that is a stronger signal than any one of them alone. Measuring it requires recognizing that three differently worded headlines describe one story. That is clustering.
Nadezhda clusters with a connected components approach (union-find). Two items join the same cluster when they fall within a time window (72 hours by default) and either share a CVE ID or, if they come from different outlets, have titles similar enough by token set Jaccard overlap. The cross outlet condition on the title match is deliberate. Grouping two same titled items from the *same* source merges a publisher's own follow up posts, which is noise. Requiring the title match to cross outlets is what killed a real false positive during development, where two distinct CISA advisories happened to share a generic title.
The size and growth rate of a cluster become the *velocity* signal in ranking. A story eight outlets picked up in six hours ranks above a story one outlet ran a day ago, even if the second has a scarier CVE.
## Ranking is deterministic and news first
The final score for a cluster is a weighted sum of normalized signals:
```
score = w_recency * recency_decay(age)
+ w_velocity * normalized(cluster_size / age)
+ w_kev * is_kev
+ w_cvss * normalized(max_cvss)
+ w_epss * max_epss
+ w_source * source_weight
+ w_keyword * keyword_match(watchlist)
```
`recency_decay` is an exponential half life function, so a story loses half its recency weight every configured number of hours. Every weight lives in configuration, not in the code, so there are no magic numbers to hunt for and the model is fully tunable.
The default weights are news first: recency and velocity together carry 70 percent, and the CVE signals (KEV, CVSS, EPSS) carry the remaining 30. This reflects a specific product decision. Nadezhda is a security *news* tool. A breach or a campaign with no CVE at all should not be buried beneath a routine patch note just because the patch note has a number attached. The CVE data is intelligence that enriches a story, not the reason the story matters.
Because ranking is a pure function of stored inputs, the same corpus always produces the same order. That property is what lets the project assert its ranking with golden order tests: feed fixed inputs, expect one exact ordering.
## References
- **MITRE CVE**: the identifier system. https://www.cve.org
- **CVE Program cvelistV5**: the keyless record source. https://github.com/CVEProject/cvelistV5
- **CVSS**, published by FIRST: the scoring system and its versions. https://www.first.org/cvss/
- **CWE**, MITRE: the weakness taxonomy. https://cwe.mitre.org
- **CISA KEV**: the exploited-in-the-wild catalog. https://www.cisa.gov/known-exploited-vulnerabilities-catalog
- **FIRST EPSS**: exploitation probability. https://www.first.org/epss/
- **NVD API 2.0**: the optional booster. https://nvd.nist.gov/developers/vulnerabilities
## Testing your understanding
- Why does a CVSS 6.5 on the KEV catalog often outrank a CVSS 9.8 that is not? What would you change in the weights to make that happen or stop it?
- The project extracts fewer CVEs from five of the seven feeds than from the other two. Why, and where in the pipeline is that limit imposed?
- If EPSS scores are parsed as zero because of the string typing bug, which ranking signal disappears, and would any test catch it?
- Why is the title similarity condition restricted to cross outlet pairs? Construct an example where dropping that restriction produces a wrong cluster.
- Log4Shell's CVSS score is not in the vendor's container of its CVE record. Where is it, and what does a naive parser report instead?

163
PROJECTS/intermediate/security-news-scraper/learn/02-ARCHITECTURE.md Normal file
View File

@ -0,0 +1,163 @@
<!-- ©AngelaMos | 2026 -->
<!-- 02-ARCHITECTURE.md -->
# Nadezhda: Architecture
This document is the map. It shows the pipeline end to end, the package that owns each stage, the data model underneath, and the design decisions that shaped all of it. Read [01-CONCEPTS.md](./01-CONCEPTS.md) first for the *why* of each stage; this file is the *how it fits together*.
## The pipeline
Everything flows one direction, from a list of feeds to a ranked set of surfaces.
```
sources.yaml (embedded default, or a user file)
│
▼
┌───────────────────────────────────┐
│ fetch N workers, per-host rate │ internal/fetch
│ limit, conditional GET │
│ (ETag / Last-Modified) │
└───────────────┬───────────────────┘
│ raw bytes + fetch_state
▼
┌───────────────────────────────────┐
│ parse RSS / Atom (gofeed) │ internal/parse
│ HTML fallback (goquery) │
└───────────────┬───────────────────┘
│ raw items
▼
┌───────────────────────────────────┐
│ normalize canonical URL, strip │ internal/normalize
│ HTML, content hash │
└───────────────┬───────────────────┘
│
▼
┌───────────────────────────────────┐
│ ingest fan-out, dedup, store │ internal/ingest
└───────┬───────────────────┬───────┘
│ │
CVE extract (regex) cluster (union-find) internal/cluster
│ │
▼ ▼
┌───────────────────────────────────┐
│ enrich CVE list / KEV / EPSS │ internal/enrich + internal/cve
│ cached, keyless │
└───────────────┬───────────────────┘
│
▼
┌───────────────────────────────────┐
│ rank deterministic weighted │ internal/rank
│ score, news first │
└───────────────┬───────────────────┘
│ ordered clusters
┌────────────┼────────────┬────────────┐
▼ ▼ ▼ ▼
digest tui ai watch
(export) (browse) (ideation) (daemon)
```
A single Go module owns all of it. There is no service boundary, no message queue, and no external database. The `scrape` command runs the whole left to right flow once; the `watch` daemon runs it on a timer.
## Packages and responsibilities
Each `internal` package has one job and a small exported surface.
| Package | Responsibility |
|---|---|
| `config` | Load and validate configuration and the source list. Every tunable lives here. |
| `source` | The source registry and the per source extractor selection. |
| `fetch` | Concurrent HTTP: worker pool, per host rate limit, conditional GET, retry with backoff. |
| `parse` | RSS and Atom via gofeed, an HTML fallback via goquery. |
| `normalize` | Canonical URL, HTML stripping, content and title hashing, time parsing. |
| `ingest` | The fan out orchestrator. Fail soft per source, dedup on insert. |
| `cluster` | Union find clustering by title similarity and shared CVE. |
| `cve` | The keyless CVE clients: CVE list, KEV, EPSS, and the optional NVD. |
| `enrich` | Enrichment orchestration and the TTL cache. |
| `rank` | The pure, deterministic scoring model. |
| `store` | SQLite: connection, migrations, and every typed query. |
| `ai` | The opt in ideation layer: a shared OpenAI compatible client plus a bespoke Anthropic client. |
| `setup` | The in binary credential wizard. |
| `watch` | The daemon scheduler, decoupled from the work it runs. |
| `tui` | The bubbletea terminal browser. |
| `export` | Markdown and JSON digest renderers. |
Orchestration does not live inside any leaf package. It lives in `cmd/nadezhda`, and the shared core of it is one function, `ingestAndCluster` in `pipeline.go`, which both `scrape` and `watch` call. That is the seam that keeps the two commands from drifting: when a step is added to the pipeline, both paths get it because there is only one place to add it.
## Design decisions
### A single static binary
The store is `modernc.org/sqlite`, a SQLite implementation written in pure Go. There is no CGO, so cross compilation is trivial and the release binaries for four platforms fall out of one build. There is no database server to run, no connection string to configure, and no container required for the core tool. The database is a file. This is the same tradeoff the ja3-ja4 project makes for its intelligence store, and it is the right one for a tool a person runs on their own machine.
### Keyless enrichment, folded into scrape
Early versions used the NVD API as the CVE source and split enrichment into its own command, because NVD without a key is rate limited to five requests per thirty seconds and you do not want that throttle on the news path. The keyless pivot removed the reason for both choices. The CVE Program's cvelistV5 records are keyless and unthrottled, so enrichment is fast, and because it is fast it folds directly into `scrape`. The default flow is now one command, `scrape`, then `tui`.
Enrichment inside `scrape` runs best effort under a five minute timeout, and its errors are non fatal. If the CVE list endpoint is down, the news still ingests, clusters, and ranks; you simply get less CVE detail that run. The news path never blocks or fails on enrichment, because the news is the product.
### The daemon knows nothing about the work
The `watch` package is a pure scheduler. It defines two interfaces, a `Ticker` and a `Notifier`, and it takes the pipeline as an injected function:
```go
type Options struct {
Interval time.Duration
RunAtStart bool
Cycle func(context.Context) (Report, error)
Notifier Notifier
NewTicker func(time.Duration) Ticker
Out io.Writer
}
```
`internal/watch` imports nothing from `store`, `ingest`, or `enrich`. The command in `cmd/nadezhda/watch.go` builds the concrete `Cycle` closure that runs the real pipeline, and hands it to the scheduler. The payoff is testability: the scheduler is unit tested with a fake ticker whose channel a test drives by hand and a fake cycle that just counts, so the loop, the graceful shutdown, and the fail soft behavior are all verified with no clock, no network, and no database. This is the same discipline as splitting an engine from its I/O so the engine can be tested in isolation.
### Fetch time, not publish time, for alerts
The watch daemon posts a webhook when genuinely new, high signal stories appear. Defining "new" is where a subtle bug lives. A cluster's timestamps are derived from article *publish* time. But an advisory published last week and fetched for the first time today is new *to you*, even though its publish time is old. Filtering alerts by publish time would silently drop it.
So the daemon has a dedicated store query, `NewlyFetchedClusters`, that filters on `articles.fetched_at`, which is monotonic with each scrape cycle. A duplicate article never updates `fetched_at` (inserts are insert only, and an unchanged feed returns a 304 that touches nothing), so a story alerts exactly once, when it is first ingested. This is the kind of correctness detail that only shows up when you think carefully about what your watermark actually measures.
## The data model
```
sources(id, name, title, url, type, weight, tags, enabled)
fetch_state(source_id, etag, last_modified, last_fetched, last_status)
articles(id, source_id,
canonical_url UNIQUE, -- exact dedup
content_hash UNIQUE, -- exact dedup
title, summary, body, author, published_at, fetched_at)
cves(id TEXT PRIMARY KEY,
description,
cvss_score, cvss_version, cvss_severity, cvss_vector, -- score nullable
cwe, is_kev, kev_date_added, kev_ransomware,
epss, epss_percentile, nvd_published, nvd_modified,
enriched_at, enrich_status)
article_cves(article_id, cve_id) -- many to many
clusters(id, cluster_key, first_seen, last_seen, size)
cluster_members(cluster_id, article_id)
ai_notes(id, cluster_id, provider, ... UNIQUE(cluster_id, provider))
schema_migrations(version, applied_at)
```
Two design points matter here. First, deduplication is enforced by the database, not by application logic. An article is unique on both its canonical URL and its content hash, so re-ingesting the same item is a constraint violation the code catches and treats as a normal "already have it" outcome, not an error. Second, migrations are versioned and forward only, applied automatically when the store opens. A schema version mismatch on an existing database is a loud error, never a silent wipe. The store opens in WAL mode with a busy timeout, so the TUI or a dashboard can read while a scrape writes.
## A story from feed to dossier
Trace one item end to end.
1. `fetch` requests BleepingComputer's feed with the ETag it saved last time. The server returns 200 with new content (a 304 would end the story here, cheaply).
2. `parse` turns the RSS into items. `normalize` canonicalizes each URL, strips tracking parameters, and computes a content hash.
3. `ingest` inserts the new article. A CVE regex finds `CVE-2025-5777` in the title and links it. The insert of a second outlet's article about the same flaw succeeds as its own row.
4. `cluster` rebuilds. The two articles share `CVE-2025-5777`, so they join one cluster. Its size is 2, and its `last_seen` advances.
5. `enrich` sees `CVE-2025-5777` needs data, fetches its cvelistV5 record, reads the CVSS 4.0 score of 9.3 from the vendor container, checks the KEV catalog (listed, ransomware "Known"), and fetches its EPSS score. All of it caches with a TTL.
6. `rank` scores the cluster. Two outlets plus a fresh publish time plus a KEV listing plus a high CVSS put it near the top.
7. `tui` renders it: a colored row with a rank, two outlet dots, a CRITICAL severity marker, and a KEV chip. Pressing enter opens the dossier with both source links and the full CVE card.
## Performance and security considerations
- **Politeness is enforced, not optional.** Every fetch is rate limited per host, uses conditional GET so an unchanged feed costs one cheap 304, and carries an honest User-Agent. A retry honors `Retry-After` and never retries a timeout or a cancellation.
- **Credentials are handled carefully.** The AI wizard writes keys to `~/.config/nadezhda/credentials` with `0600` permissions, using a temp file and atomic rename so an existing file's permissions are tightened rather than left as they were. A shell exported key always wins over the file, and the file loader only accepts key names on an allowlist, which blocks a tampered credentials file from injecting something like `LD_PRELOAD` into the process environment.
- **The AI clients do not follow redirects.** A redirect could leak an `x-api-key` header to an unintended host, so the clients refuse to follow them.
- **Subprocess environments are sanitized.** The "open in browser" action in the TUI launches a child process with a scrubbed environment, not the full one, so a secret in the parent process is not handed to the browser launcher.
- **Failure is soft where it should be and loud where it must be.** One broken feed does not abort a scrape. A failing enrichment does not block the news. But a corrupt database or a failed migration stops the program, because continuing past those would silently corrupt your results.

147
PROJECTS/intermediate/security-news-scraper/learn/03-IMPLEMENTATION.md Normal file
View File

@ -0,0 +1,147 @@
<!-- ©AngelaMos | 2026 -->
<!-- 03-IMPLEMENTATION.md -->
# Nadezhda: Implementation
This is the code walkthrough. It follows the same left to right order as the pipeline in [02-ARCHITECTURE.md](./02-ARCHITECTURE.md), naming the real functions so you can open each one as you read. Code is referenced by package and function, never by line number, because line numbers rot the moment anyone edits a file.
## Where to start reading
Open `cmd/nadezhda/scrape.go` and find `runScrape`. It is the whole tool in miniature: load config, load sources, open the store, build a fetch client, run the pipeline, print the summary, enrich. The one line that does the real work is:
```go
summary, cstats, err := ingestAndCluster(ctx, fc, st, cfg, targets, now)
```
`ingestAndCluster` lives in `pipeline.go`, and it is the shared seam that `watch` also calls. Everything below is what happens inside it and after it.
## Fetching politely
`internal/fetch` is a worker pool over `net/http`. The interesting behavior is conditional GET. On the first fetch of a source it stores the `ETag` and `Last-Modified` response headers in the `fetch_state` table. On the next fetch it sends them back as `If-None-Match` and `If-Modified-Since`. An unchanged feed answers `304 Not Modified` with no body, which costs almost nothing, and the source's row in the scrape summary shows `304` instead of a parse count.
The retry logic is deliberately conservative. It honors a `Retry-After` header when the server sends one, and it does *not* retry a timeout or a cancelled context, because retrying a timeout usually just means waiting for the same timeout again. Per host rate limiting uses `golang.org/x/time/rate`, so two feeds on the same domain do not hammer it in parallel.
## Parsing and normalizing
`internal/parse` uses `gofeed`, which handles RSS and Atom and most of their real world deviations. One lesson worth internalizing lives in the time parsing: feeds do not agree on date formats. CISA's feed uses RFC822 with a numeric zone, which an earlier version of the research wrongly assumed was RFC1123Z. The parser carries a list of fallback layouts and tries them in order, because a feed that fails to parse its dates silently loses its recency signal.
`internal/normalize` does three small but load bearing things. It canonicalizes URLs, stripping tracking parameters like `utm_*` and `fbclid`, so the same article shared with different tracking tails deduplicates correctly. It strips HTML from summaries with `goquery`. And it computes a SHA-256 content hash and a title hash, which are the exact dedup keys.
## Ingesting, fail soft
`internal/ingest`'s `Run` fans out across sources with an `errgroup`, but with a twist: the per source goroutines always return `nil`. A source's error is recorded in its result struct, not propagated up, so one broken feed never cancels the group and aborts every other source. The command sums the results and reports failures at the end. This is why you can lose The Register to a timeout and still get a full scrape from the other six.
Dedup happens at the database. `store.InsertArticle` is a plain `INSERT`, and when the unique constraint on `canonical_url` or `content_hash` fires, it returns a sentinel:
```go
if errors.As(err, &se) && se.Code() == sqlite3.SQLITE_CONSTRAINT_UNIQUE {
return 0, ErrDuplicate
}
```
The caller treats `ErrDuplicate` as a normal "already have this" and increments the duplicate counter. There is no read before write race, because the database is the single source of truth for what exists.
## Clustering with union find
`internal/cluster` implements connected components. `Compute` takes the candidate items, tokenizes each title, and runs a `unionFind` over every pair inside the time window. Two items union when they share a CVE, or when they come from different sources and their title token sets have Jaccard overlap at or above the threshold:
```go
sharedCVE := shareAny(prepared[i].cves, prepared[j].cves)
crossSource := prepared[i].item.SourceID != prepared[j].item.SourceID
titleMatch := crossSource &&
jaccard(prepared[i].tokens, prepared[j].tokens) >= jaccardThreshold
if sharedCVE || titleMatch {
uf.union(i, j)
}
```
`buildCluster` then derives the cluster's `FirstSeen` and `LastSeen` from the earliest and latest member times and its `SourceCount` from the distinct sources. `Rebuild` is the entry point the pipeline calls: it pulls candidate articles with `ClusterCandidates`, runs `Compute`, and writes the result with `ReplaceClusters`, which clears and rewrites the cluster tables in a single transaction so the operation is idempotent. Re-running a scrape never produces drifting or duplicated clusters.
Note the timestamp expression in `ClusterCandidates`:
```sql
COALESCE(NULLIF(published_at, 0), fetched_at)
```
An item with no publish time falls back to its fetch time, so a feed that omits dates still clusters within the window instead of collapsing to the epoch.
## The keyless CVE clients
`internal/cve` holds three clients behind a common interface. The keyless CVE core, in the CVE list client, is where the research paid off. It parses the cvelistV5 record, which has a `containers.cna` block from the vendor and zero or more `containers.adp` blocks from authorized data publishers like CISA. It collects every CVSS metric from both, then applies the version precedence: prefer v4.0, then v3.1, then v3.0, then v2.0. This two place search is what correctly reads Log4Shell's 10.0 score out of the CISA-ADP container when the vendor container holds only a placeholder.
The KEV client downloads the catalog once and builds a membership map. It maps the ransomware field explicitly, because the source ships a string:
```go
ransomware := entry.KnownRansomwareCampaignUse == "Known"
```
The EPSS client parses the score and percentile with `ParseFloat`, because the API returns them as quoted strings. Read them as JSON numbers and they decode to zero with no error, which is the quiet kind of bug that survives to production.
## Enriching with a cache
`internal/enrich`'s `Run` asks the store which CVEs need work with `CVEsNeedingEnrichment`, which returns any CVE never enriched, or one whose positive result is older than the cache TTL, or one whose negative result (not found) is older than the shorter negative TTL. The negative cache is what stops the tool from hammering the CVE source every run for a CVE that does not exist yet.
KEV failure is fatal to the enrichment run by design. If the KEV catalog fetch fails, the code refuses to write `is_kev = false` for everything, because a wrong `false` would persist for a full TTL and silently hide exploited vulnerabilities. NVD and EPSS failures are soft and resumable.
## Ranking, purely
`internal/rank`'s `Score` is a pure function of a `Signals` struct and the config weights. `recency` is the exponential half life decay, and `velocity` divides cluster size by age with a floor so a one item cluster scores zero velocity:
```go
func recency(ageHours float64, halfLifeHours int) float64 {
return math.Exp(-math.Ln2 * ageHours / float64(halfLifeHours))
}
```
`signalsFor` builds the signal struct for a cluster, walking its CVEs to take the maximum CVSS, the maximum EPSS, and whether any is KEV listed. `Rank` scores every cluster and stable sorts descending, breaking ties by the freshest `LastSeen`. Because it is pure, the golden order tests can assert an exact ordering from fixed inputs.
## The store queries that matter
`internal/store` hides all SQL. Two queries are worth knowing by name. `DigestClusters(since)` filters `WHERE last_seen >= ?`, which is publish time based, and it feeds the TUI and the digest. `NewlyFetchedClusters(sinceFetched)` filters `WHERE a.fetched_at >= ?`, which is fetch time based, and it feeds only the watch daemon's alerts. They exist as two separate queries on purpose, for the fetch time versus publish time reason explained in the architecture doc, and a store test asserts the distinction directly so nobody collapses them later.
## The watch daemon
`internal/watch`'s `Run` is the loop:
```go
for {
select {
case <-ctx.Done():
return shutdown(out)
case <-ticker.C():
if stop, err := cycleAndNotify(ctx, opts, out); stop {
return shutdown(out)
} else if err != nil {
fmt.Fprintf(out, "watch: cycle error: %v\n", err)
}
}
}
```
A cycle error is logged and the loop continues, which is the fail soft contract for a long running daemon. `ctx.Done()` returns `nil`, so Ctrl-C or SIGTERM is a clean exit, not an error. `Once` runs the same `cycleAndNotify` a single time for cron use.
The concrete cycle lives in `cmd/nadezhda/watch.go`. After running `ingestAndCluster` and enrichment, it calls `buildNotable`, which pulls `NewlyFetchedClusters`, ranks them, and keeps the notable ones up to a configured cap. `isNotable` is the policy: a cluster is notable if its score clears the threshold or, when `notify_on_kev` is set, if it contains a KEV listed CVE. `representativeArticle` picks the highest trust outlet's headline to represent the cluster in the alert.
## The AI layer
`internal/ai` defines one `Provider` interface with a `Generate` method. Three providers (qwen, openai, gemini) share one OpenAI compatible client, and Anthropic gets its own client, written against raw `net/http`, because it differs on auth header, system prompt placement, and response shape. Keys are read from the environment only. A model like a local Qwen tends to wrap its JSON in prose, so the result parser extracts JSON by scanning balanced braces and trying each candidate object rather than assuming the whole response is JSON.
## The setup wizard
`internal/setup` is the seamless install story. `nadezhda ai` runs an interactive, re-runnable wizard. It writes keys to a `0600` file with a temp file and atomic rename so it tightens an existing file's permissions instead of trusting them. `Load` runs on every command and sets each key into the environment only if unset, so a shell variable always wins, and it accepts only allowlisted key names so a tampered file cannot inject arbitrary environment variables into the process.
## Common pitfalls
These are the real bugs that were caught and fixed, and the ones you are most likely to reintroduce.
- **Parsing EPSS or KEV fields with the wrong type.** EPSS scores are strings; the ransomware flag is a string. Both decode wrong silently.
- **Reading CVSS from only the vendor container.** You will report "no score" for exactly the most important CVEs.
- **Using publish time for alerts.** A backfilled advisory fetched today has an old publish time. Alert on fetch time.
- **Writing `is_kev = false` when the KEV fetch failed.** A wrong negative persists for a full TTL. Fail the run instead.
- **Letting one broken feed abort the scrape.** Fan out fail soft, record the error per source.
## Debugging tips
- Run `nadezhda scrape` twice. The second run should show every source as `304` or all duplicates, proving idempotency. If it shows new articles both times, dedup is broken.
- Point `nadezhda cve CVE-2021-44228` at a known CVE after a scrape that mentions it. You should see CVSS 10.0, a KEV listing, and a high EPSS. If the score is missing, your container search is the suspect.
- For the daemon, run `nadezhda watch --once` against a scratch database and read the one line cycle report. It prints new, duplicate, cluster, enriched, KEV, and notable counts, which tells you which stage is misbehaving.

57
PROJECTS/intermediate/security-news-scraper/learn/04-CHALLENGES.md Normal file
View File

@ -0,0 +1,57 @@
<!-- ©AngelaMos | 2026 -->
<!-- 04-CHALLENGES.md -->
# Nadezhda: Challenges
Ways to extend the project, ordered roughly from an afternoon to a serious undertaking. Each one names the code you would touch and gives a hint, not a full solution. The best way to understand a pipeline is to add a stage to it.
## Warm up
**Add a source.** The source list is `sources.yaml`, embedded by default. Add an entry with a name, feed URL, type, and weight, then run `nadezhda sources` to persist it and `nadezhda scrape` to pull it. Pick a feed that overlaps with the existing seven (say, a second general security outlet) and watch it start joining clusters. The source `weight` feeds ranking, so a source you trust more nudges its stories up.
**Retune the ranking.** Every weight lives in `config`, under `rank.weights`. The defaults are news first (recency and velocity dominate). Flip that: crank `cvss`, `kev`, and `epss` up and `recency` down, scrape, and compare the top of `nadezhda digest`. You will see the list reorder toward raw severity. There is no code change here, which is the point: the model is data, not logic.
**Add a keyword watchlist.** Set `watchlist` in config to the vendors and products you actually run. `matchesWatchlist` in `internal/rank` scores a cluster higher when a term appears in its titles or CVEs. Confirm it works by adding a vendor you know is in today's news and watching its stories climb.
**Add an export format.** `internal/export` renders Markdown and JSON. Add a third, for instance a plain text digest or an RSS feed of the ranked clusters, so nadezhda can feed another tool. Keep the renderer a pure function of the scored clusters, the way `export.Markdown` is, so it stays testable.
## Medium
**Add an AI provider.** `internal/ai` has one `Provider` interface. Three providers share the OpenAI compatible client. Add a new OpenAI compatible endpoint (a different local runtime, or a hosted model) by wiring its base URL and model into the config and the factory. The harder version is a provider that is not OpenAI compatible, which is why Anthropic has its own client: you will learn where the response shapes actually differ.
**Turn on the HTML article scrape path.** The fetch client already has `Allowed`, a robots.txt gate, built and tested but intentionally never called, because the feed path does not use it. Wire it into a per source HTML extractor for a source whose feed is summary only, so you can pull the full article body and extract more CVEs. This is the one place robots.txt must be honored, so route the fetch through `Allowed` first. You are building the crawler path the architecture deliberately kept separate.
**Make the webhook smarter.** The watch daemon posts a `text`/`content`/`items` JSON payload that works for Slack, Discord, or a generic endpoint. Add a real Slack Block Kit or Discord embed formatter behind a config switch, so an alert renders as a rich card instead of a text line. The notable set is already capped and ranked, so you are formatting, not filtering.
**Report per source health.** `fetch_state` already stores each source's last status and last fetch time. Add a `nadezhda sources --health` view that flags a source that has returned an error or a 304 for many runs in a row, so a silently dead feed becomes visible.
## Hard
**Replace pairwise clustering with LSH.** `internal/cluster` compares every candidate pair inside the window, which is fine for hundreds of articles and quadratic for tens of thousands. The research deferred SimHash as a scale optimization for exactly this reason. Implement locality sensitive hashing over the title token sets so near duplicates land in the same bucket and you only compare within a bucket. Keep the existing `Compute` as the reference implementation and assert the LSH version produces the same clusters on the test fixtures before you trust it.
**Persist the alert watermark.** The daemon derives "new since last cycle" from the current cycle's wall clock, which is correct except across a backward clock step, where an article could alert twice. Add a small `watch_state` table that records the last notified point, and have `buildNotable` read it instead of the in memory cycle start. This also makes alerts survive a restart cleanly, so a daemon that crashes and comes back does not re-alert the backlog.
**Embed the real version in `go install` builds.** The release binaries carry their version because goreleaser injects it with `ldflags`. A plain `go install` build shows the development default, because `go install` does not inject anything. Read the module version from `runtime/debug.ReadBuildInfo` as a fallback in `internal/version` so a tagged `go install` build reports its tag. This is a small change that teaches you how Go embeds build metadata.
**Add full text search.** SQLite ships FTS5. Add a virtual table over article titles and bodies and a `nadezhda search "term"` command, so a user can search the whole archive by product, whether or not it is currently trending. Mind the migration: it is a new table, applied forward only, the way `internal/store` applies every other schema change.
## Expert
**Build a read only dashboard.** The store opens in WAL mode specifically so a reader can run while a scrape writes. Build a small web UI that reads `DigestClusters` and serves the same ranked dossier the TUI shows, with no write path of its own. The interesting constraint is that it must never migrate or write, so it opens the database read only and fails loudly if the schema is newer than it understands.
**Make clustering incremental.** `Rebuild` recomputes every cluster from scratch each scrape, which is correct and simple and eventually slow. Design an incremental version that only reclusters articles inside the active window and merges them into existing clusters, and prove it converges to the same result as a full rebuild. This is genuinely hard, because cluster identity has to stay stable across runs for the velocity signal to mean anything.
**Learn a relevance model.** The ranking is a fixed weighted sum. Collect which stories a user actually opens in the TUI and train a model that predicts relevance from the same signals plus the user's history. Keep the deterministic model as the cold start and the fallback, and treat the learned score as one more weighted input, so the system degrades gracefully when the model is absent or wrong.
**Ship a diff.** Add a `nadezhda digest --since-last` that shows only what changed since the previous digest: new clusters, clusters that grew, and CVEs that newly landed on KEV. This is the report a person actually wants each morning, and it forces you to think about what "changed" means for a cluster, which is the same watermark problem the watch daemon solves.
## Real world applications
- A morning triage feed for a small security team, replacing a folder of browser bookmarks with one ranked list and a Slack alert for KEV listed spikes.
- A research input for a writer or educator who needs to know what the security world is talking about today, with the CVE evidence attached.
- A personal monitor keyed to a watchlist of the exact products you run, so you hear about a trending flaw in one of them before your vendor's email arrives.
## Connections to other projects
- **sbom-generator-vulnerability-matcher** answers the question this project raises. Nadezhda tells you a CVE is trending and exploited. The SBOM matcher tells you whether that CVE is in something you actually ship. Together they are a prioritization loop.
- **ja3-ja4-tls-fingerprinting** is the same architecture applied to a different signal: a keyless intelligence engine over an embedded SQLite store, with the same discipline of pinning outputs to known answer vectors and keeping the engine testable in isolation. Read its clustering ideas alongside this one.