11 KiB

Raw Permalink Blame History

Nadezhda: Implementation

This is the code walkthrough. It follows the same left to right order as the pipeline in 02-ARCHITECTURE.md, naming the real functions so you can open each one as you read. Code is referenced by package and function, never by line number, because line numbers rot the moment anyone edits a file.

Where to start reading

Open cmd/nadezhda/scrape.go and find runScrape. It is the whole tool in miniature: load config, load sources, open the store, build a fetch client, run the pipeline, print the summary, enrich. The one line that does the real work is:

summary, cstats, err := ingestAndCluster(ctx, fc, st, cfg, targets, now)

ingestAndCluster lives in pipeline.go, and it is the shared seam that watch also calls. Everything below is what happens inside it and after it.

Fetching politely

internal/fetch is a worker pool over net/http. The interesting behavior is conditional GET. On the first fetch of a source it stores the ETag and Last-Modified response headers in the fetch_state table. On the next fetch it sends them back as If-None-Match and If-Modified-Since. An unchanged feed answers 304 Not Modified with no body, which costs almost nothing, and the source's row in the scrape summary shows 304 instead of a parse count.

The retry logic is deliberately conservative. It honors a Retry-After header when the server sends one, and it does not retry a timeout or a cancelled context, because retrying a timeout usually just means waiting for the same timeout again. Per host rate limiting uses golang.org/x/time/rate, so two feeds on the same domain do not hammer it in parallel.

Parsing and normalizing

internal/parse uses gofeed, which handles RSS and Atom and most of their real world deviations. One lesson worth internalizing lives in the time parsing: feeds do not agree on date formats. CISA's feed uses RFC822 with a numeric zone, which an earlier version of the research wrongly assumed was RFC1123Z. The parser carries a list of fallback layouts and tries them in order, because a feed that fails to parse its dates silently loses its recency signal.

internal/normalize does three small but load bearing things. It canonicalizes URLs, stripping tracking parameters like utm_* and fbclid, so the same article shared with different tracking tails deduplicates correctly. It strips HTML from summaries with goquery. And it computes a SHA-256 content hash and a title hash, which are the exact dedup keys.

Ingesting, fail soft

internal/ingest's Run fans out across sources with an errgroup, but with a twist: the per source goroutines always return nil. A source's error is recorded in its result struct, not propagated up, so one broken feed never cancels the group and aborts every other source. The command sums the results and reports failures at the end. This is why you can lose The Register to a timeout and still get a full scrape from the other six.

Dedup happens at the database. store.InsertArticle is a plain INSERT, and when the unique constraint on canonical_url or content_hash fires, it returns a sentinel:

if errors.As(err, &se) && se.Code() == sqlite3.SQLITE_CONSTRAINT_UNIQUE {
    return 0, ErrDuplicate
}

The caller treats ErrDuplicate as a normal "already have this" and increments the duplicate counter. There is no read before write race, because the database is the single source of truth for what exists.

Clustering with union find

internal/cluster implements connected components. Compute takes the candidate items, tokenizes each title, and runs a unionFind over every pair inside the time window. Two items union when they share a CVE, or when they come from different sources and their title token sets have Jaccard overlap at or above the threshold:

sharedCVE := shareAny(prepared[i].cves, prepared[j].cves)
crossSource := prepared[i].item.SourceID != prepared[j].item.SourceID
titleMatch := crossSource &&
    jaccard(prepared[i].tokens, prepared[j].tokens) >= jaccardThreshold
if sharedCVE || titleMatch {
    uf.union(i, j)
}

buildCluster then derives the cluster's FirstSeen and LastSeen from the earliest and latest member times and its SourceCount from the distinct sources. Rebuild is the entry point the pipeline calls: it pulls candidate articles with ClusterCandidates, runs Compute, and writes the result with ReplaceClusters, which clears and rewrites the cluster tables in a single transaction so the operation is idempotent. Re-running a scrape never produces drifting or duplicated clusters.

Note the timestamp expression in ClusterCandidates:

COALESCE(NULLIF(published_at, 0), fetched_at)

An item with no publish time falls back to its fetch time, so a feed that omits dates still clusters within the window instead of collapsing to the epoch.

The keyless CVE clients

internal/cve holds three clients behind a common interface. The keyless CVE core, in the CVE list client, is where the research paid off. It parses the cvelistV5 record, which has a containers.cna block from the vendor and zero or more containers.adp blocks from authorized data publishers like CISA. It collects every CVSS metric from both, then applies the version precedence: prefer v4.0, then v3.1, then v3.0, then v2.0. This two place search is what correctly reads Log4Shell's 10.0 score out of the CISA-ADP container when the vendor container holds only a placeholder.

The KEV client downloads the catalog once and builds a membership map. It maps the ransomware field explicitly, because the source ships a string:

ransomware := entry.KnownRansomwareCampaignUse == "Known"

The EPSS client parses the score and percentile with ParseFloat, because the API returns them as quoted strings. Read them as JSON numbers and they decode to zero with no error, which is the quiet kind of bug that survives to production.

Enriching with a cache

internal/enrich's Run asks the store which CVEs need work with CVEsNeedingEnrichment, which returns any CVE never enriched, or one whose positive result is older than the cache TTL, or one whose negative result (not found) is older than the shorter negative TTL. The negative cache is what stops the tool from hammering the CVE source every run for a CVE that does not exist yet.

KEV failure is fatal to the enrichment run by design. If the KEV catalog fetch fails, the code refuses to write is_kev = false for everything, because a wrong false would persist for a full TTL and silently hide exploited vulnerabilities. NVD and EPSS failures are soft and resumable.

Ranking, purely

internal/rank's Score is a pure function of a Signals struct and the config weights. recency is the exponential half life decay, and velocity divides cluster size by age with a floor so a one item cluster scores zero velocity:

func recency(ageHours float64, halfLifeHours int) float64 {
    return math.Exp(-math.Ln2 * ageHours / float64(halfLifeHours))
}

signalsFor builds the signal struct for a cluster, walking its CVEs to take the maximum CVSS, the maximum EPSS, and whether any is KEV listed. Rank scores every cluster and stable sorts descending, breaking ties by the freshest LastSeen. Because it is pure, the golden order tests can assert an exact ordering from fixed inputs.

The store queries that matter

internal/store hides all SQL. Two queries are worth knowing by name. DigestClusters(since) filters WHERE last_seen >= ?, which is publish time based, and it feeds the TUI and the digest. NewlyFetchedClusters(sinceFetched) filters WHERE a.fetched_at >= ?, which is fetch time based, and it feeds only the watch daemon's alerts. They exist as two separate queries on purpose, for the fetch time versus publish time reason explained in the architecture doc, and a store test asserts the distinction directly so nobody collapses them later.

The watch daemon

internal/watch's Run is the loop:

for {
    select {
    case <-ctx.Done():
        return shutdown(out)
    case <-ticker.C():
        if stop, err := cycleAndNotify(ctx, opts, out); stop {
            return shutdown(out)
        } else if err != nil {
            fmt.Fprintf(out, "watch: cycle error: %v\n", err)
        }
    }
}

A cycle error is logged and the loop continues, which is the fail soft contract for a long running daemon. ctx.Done() returns nil, so Ctrl-C or SIGTERM is a clean exit, not an error. Once runs the same cycleAndNotify a single time for cron use.

The concrete cycle lives in cmd/nadezhda/watch.go. After running ingestAndCluster and enrichment, it calls buildNotable, which pulls NewlyFetchedClusters, ranks them, and keeps the notable ones up to a configured cap. isNotable is the policy: a cluster is notable if its score clears the threshold or, when notify_on_kev is set, if it contains a KEV listed CVE. representativeArticle picks the highest trust outlet's headline to represent the cluster in the alert.

The AI layer

internal/ai defines one Provider interface with a Generate method. Three providers (qwen, openai, gemini) share one OpenAI compatible client, and Anthropic gets its own client, written against raw net/http, because it differs on auth header, system prompt placement, and response shape. Keys are read from the environment only. A model like a local Qwen tends to wrap its JSON in prose, so the result parser extracts JSON by scanning balanced braces and trying each candidate object rather than assuming the whole response is JSON.

The setup wizard

internal/setup is the seamless install story. nadezhda ai runs an interactive, re-runnable wizard. It writes keys to a 0600 file with a temp file and atomic rename so it tightens an existing file's permissions instead of trusting them. Load runs on every command and sets each key into the environment only if unset, so a shell variable always wins, and it accepts only allowlisted key names so a tampered file cannot inject arbitrary environment variables into the process.

Common pitfalls

These are the real bugs that were caught and fixed, and the ones you are most likely to reintroduce.

Parsing EPSS or KEV fields with the wrong type. EPSS scores are strings; the ransomware flag is a string. Both decode wrong silently.
Reading CVSS from only the vendor container. You will report "no score" for exactly the most important CVEs.
Using publish time for alerts. A backfilled advisory fetched today has an old publish time. Alert on fetch time.
Writing is_kev = false when the KEV fetch failed. A wrong negative persists for a full TTL. Fail the run instead.
Letting one broken feed abort the scrape. Fan out fail soft, record the error per source.

Debugging tips

Run nadezhda scrape twice. The second run should show every source as 304 or all duplicates, proving idempotency. If it shows new articles both times, dedup is broken.
Point nadezhda cve CVE-2021-44228 at a known CVE after a scrape that mentions it. You should see CVSS 10.0, a KEV listing, and a high EPSS. If the score is missing, your container search is the suspect.
For the daemon, run nadezhda watch --once against a scratch database and read the one line cycle report. It prints new, duplicate, cluster, enriched, KEV, and notable counts, which tells you which stage is misbehaving.

11 KiB Raw Permalink Blame History