Center assets/tui.png below the tagline and above the overview, using the house p-align-center image convention. Carries the badge bump to Project #39 and the What it is section heading.
Match the ja3-ja4 / zingela README convention: ANSI Shadow banner, badge row (Project 37, Go, keyless, single static binary, AGPL), blockquote pitch, then Why / What Works Today / Quick Start with real sample output / just TIP / Learn table / Architecture diagram / Build and Test / Project Structure. Drops the phantom just-watch reference (no such recipe). Content tracks the verified code: news-first 70/30 weights, Log4Shell KAT as CWE-502.
The 70% news bloc is recency+velocity+source+keyword, not recency+velocity alone (which sum to 0.50). Log4Shell surfaces as CWE-502 (first CNA problemType, per TestCVEListParsesLog4Shell), not CWE-20. Also reword the 04-CHALLENGES weight aside to match.
Five teaching docs (overview, concepts, architecture, implementation, challenges) grounded in real incidents (Log4Shell CVE-2021-44228, Citrix Bleed 2 CVE-2025-5777, Equifax CVE-2017-5638, MOVEit CVE-2023-34362). ASCII pipeline + data-model diagrams, a code walkthrough by function name (no line numbers), and challenges from adding a source to replacing clustering with LSH. No em dashes, per the repo's docs style.
One-shot install.sh (curl|bash -> prebuilt release binary, or go install fallback with Go-toolchain bootstrap; installs to ~/.local/bin and wires PATH). .goreleaser.yaml + .github/workflows/release.yml build cross-platform binaries on a v* tag (inert in the monorepo, active once the project dir is the standalone repo root).
Add the AGPL-3.0 LICENSE and rewrite README.md with install + quickstart. Make internal/version.Version a var so goreleaser injects the real version via ldflags. justfile gains watch (run daemon from source) and publish (drift-free rsync mirror to the standalone github.com/CarterPerez-dev/nadezhda checkout).
Add a watch command (--interval, --once, --no-enrich): a long-lived daemon that re-ingests every enabled source on a ticker and, when a webhook is configured, POSTs genuinely new high-signal stories to it (Slack, Discord, or any JSON endpoint via a text/content/items payload).
internal/watch is a pure, stdlib-only scheduler: the pipeline is injected as a Cycle closure, so the daemon is fully unit-testable with a fake ticker and notifier. Graceful shutdown on SIGINT/SIGTERM returns cleanly; a cycle error is logged and the loop continues (fail-soft).
Notify uses a dedicated fetch-time watermark (store.NewlyFetchedClusters) so a freshly ingested but older-dated advisory is surfaced, which a publish-time filter would drop. Notable stories are capped by watch.notify_max_items and filtered by score threshold or KEV status.
Extract the shared ingest and cluster sequence (pipeline.go) so scrape and watch cannot drift, and route scrape and enrich through cmd.Context() so they honor SIGTERM too.
Add an opt-in ideation layer that turns ranked story clusters into content angles. A Provider interface backs four models: qwen (default, local Ollama), OpenAI, and Gemini share one OpenAI-compatible client; an Anthropic Messages client is written against raw net/http (no vendor SDK, uniform + dep-light). Results persist in ai_notes with a UNIQUE(cluster_id, provider) index (migration 0003), so re-ideation upserts in place.
The ideate command skips any (cluster, provider) that already has a note unless --force is passed; benign classifier refusals get their own counter and never fail the run. In the TUI, i ideates the selected cluster inline and renders an AI IDEAS section in the dossier; it is a no-op unless AI is enabled and only acts in the detail view.
Add a re-runnable interactive wizard that configures an AI provider for the ideation layer: Claude, OpenAI, Gemini, or a local Ollama. Keys persist to ~/.config/nadezhda/credentials (0600, atomic temp+rename) and auto-load on every run; a shell-exported key always wins over the file. A key-name allowlist (NADEZHDA_* / *_API_KEY) blocks env-name injection into spawned subprocesses.
The Ollama path detects a running instance and auto-configures it, else writes an embedded compose file plus one docker command, else prints an install hint. Ollama's host port is remapped to a non-default 39847 (OLLAMA_HOST_PORT) to avoid colliding with an existing Ollama on 11434; the container-internal port stays 11434.
Thread a signal-cancellable root context through cobra (ExecuteContext). Default the Anthropic model to claude-sonnet-4-6.
_PATTERN_RULES list had XXE_INJECTION (0.82) after XSS (0.80) and
CRLF_INJECTION (0.65) after PATH_TRAVERSAL (0.60), placing higher-
scored rules below lower-scored ones. Reordered to strict descending
order and updated the module docstring to match. No change to
scoring behavior — score_request() takes max + boost regardless
of list position.
overrides.vite was pinned to npm:rolldown-vite@7.2.5 while
devDependencies and pnpm-lock.yaml both resolved 7.3.1. The mismatch
can produce ERR_PNPM_OUTDATED_LOCKFILE on --frozen-lockfile installs.
Updated override to 7.3.1 to match the resolved version throughout
the lockfile. No lockfile edits required.
Moved entrypoint.sh copy target from /app/entrypoint.sh to
/entrypoint.sh in the dev Dockerfile. The dev compose bind mount
./backend:/app overlays /app entirely; placing the entrypoint above
that boundary ensures the image-baked script is never shadowed by
host directory contents. No startup failure was reproduced — this
is a preventive hardening change.
urllib.parse.quote(path, safe='/?&=') was re-encoding % to %25 in
already-encoded attack paths (XXE, XSS, traversal payloads). Replaced
with a manual first-? split: path component encoded with safe='/%\'
(preserving existing percent sequences and backslash traversal vectors),
query string passed through untouched. Covers all 53 attack payloads
across 7 simulation modes. Inline comments removed from XXE_PAYLOADS
to match the style of all other payload lists.
The final branch of _XXE_INJECTION was missing % separators between
hex pairs. Pattern %(?:25)?53(?:25)?59... never matched because after
consuming %53 the next sub-pattern expected 5 or 2, not %. Fixed by
inserting % anchors between each pair so both single-encoded
(%53%59%53%54%45%4D) and double-encoded (%2553%2559...) SYSTEM strings
match correctly. All existing XXE branches are unaffected.
This is a security-news tool; CVE enrichment is optional garnish, never a
setup tax. Drop the NVD API key from the default path entirely.
- internal/cve/cvelist: keyless CVE Program cvelistV5 client (raw GitHub,
carries CISA-ADP / vulnrichment CVSS). Parses cna and adp metrics with
version precedence v4.0 > v3.1 > v3.0 > v2.0, plus CWE and description;
404 maps to not-found; v2 severity is capped at HIGH. Introduces a
neutral CoreResult and a CVESource interface (NVDResult renamed to
CoreResult; NVDClient still satisfies it).
- enrich: Clients.Core is the CVESource. buildCoreSource picks the keyless
cvelist client by default and NVD only when NVD_API_KEY is set (booster).
- scrape auto-enriches referenced CVEs best-effort after clustering
(time-bounded, non-fatal, --no-enrich to skip); enrich kept as an
optional manual refresh. Default flow is scrape -> tui.
- ranking default weights rebalanced news-first (recency / velocity /
source / keyword 70 percent, cvss / kev / epss 30 percent; was 55 percent CVE).
- KAT fixture testdata/cvelist/CVE-2021-44228.json.
Interactive terminal UI over the ranked story clusters. Rave Maximal
cybercore theming: cyan-to-magenta spectrum score bars, severity-reactive
rows, hot KEV chips, cool-ramp outlet dots, and a scrollable detail
dossier with per-outlet links and full CVE cards (CVSS meter, KEV /
ransomware, EPSS, CWE, vector, description). Reuses store.DigestClusters
and rank.Rank verbatim and preloads full store.CVE records for detail.
- internal/tui: model/update/view state machine with an injected Loader
and opener, so Update is fully unit-testable without a terminal or DB
(nav / clamp / open / back / quit / resize / empty-store / too-small).
- o opens the selected story in the default browser, keyboard-only,
cross-platform, and refuses non-http urls.
- tui command replaces the M5 stub; supports --since.
- deps: charmbracelet bubbletea v1.3.10 + bubbles v1.0.0 + lipgloss v1.1.0.
M4:
- Add 01-read-the-hex (hex-reading) and 02-find-the-entry (elf-anatomy)
gradeable challenges over the gate binary, completing the five-module
pedagogy
- Swap in-memory progress for a SQLite ProgressStore behind the existing
Protocol; routes untouched, migration-free, thread-safe
M5:
- Self-host Docker: prod (nginx + uvicorn) and dev (nginx + Vite HMR) compose,
justfile, Dockerfiles, and nginx configs; zero-config clone-and-run
- learn/ teaching docs (00-04) and public README
- One-shot install.sh and uninstall.sh
Turn the aggregated store into a ranked, exportable digest of story clusters.
- internal/rank: pure deterministic Score(Signals, cfg.Rank) — recency
(exp half-life decay), cvss/10, kev, epss, velocity (cluster size/age), source
weight, watchlist keyword. Every signal clamped to [0,1]; all weights +
half_life + velocity_norm from config, no literals. recency/velocity are total
(guard half_life<1 and norm<=0). Rank() scores each cluster and stable-sorts
desc, tiebreak freshest. Research 06's two worked examples are the golden-order
test (A ~0.99 strictly above B ~0.13).
- internal/export: per-cluster (story) digest. Headline = freshest member, distinct
sorted outlets, CVEs sorted KEV-first then CVSS. Markdown + deterministic JSON.
- store.DigestClusters: 3-query per-cluster aggregation (clusters by since, member
articles, union of CVE signals) with explicit ORDER BY; per-cluster CVE dedup
via SELECT DISTINCT.
- digest command: --top/--since/--format md|json/--out. Off the stub list.
- Proven live: scrape -> digest --format md and json render correctly, per-cluster
(FortiBleed shows 3 outlets). Suite offline + -race.
One read-only audit agent; 0 Crit/High/Med, golden-order math verified; Low/Nit
fixed in-phase (top<=0 count, pure-fn guards, ORDER BY, headline sanitize, 2-cluster
attach test).
Digest ranks per-cluster (Carter's choice), not per-article.
The intelligence layer. scrape now extracts CVE IDs (cheap, no network) into
article_cves, which activates M2's shared-CVE clustering. A separate 'enrich'
command does the slow, cached API work (Carter's split: fast scrape, deliberate
enrich).
- internal/cve: regex CVE-ID extraction + three clients (stdlib net/http):
- nvd: API 2.0, CVSS precedence v4.0>v3.1>v3.0>v2 with nullable metrics and v2
metric-level severity fallback, totalResults==0 => not-found, apiKey header
(exact case), token-bucket rate limit (5/30s anon, 50/30s keyed) + retry on
429/5xx, timeouts non-retryable.
- kev: one catalog download -> membership map; knownRansomwareCampaignUse is the
STRING 'Known'/'Unknown' mapped explicitly to bool.
- epss: batched; epss/percentile are QUOTED STRINGS parsed with ParseFloat (a
parse failure is skipped, a legitimate 0.0 is kept); partial batches survive
a transient error.
- internal/enrich: enriches only unenriched/stale CVEs (positive + negative TTL),
KEV once per run, EPSS batched, NVD per-CVE. NVD/EPSS soft per-item (resumable),
KEV fatal by design (enriching without it would persist wrong is_kev for a TTL).
- store: UpsertCVEStub/LinkArticleCVE, CVEsNeedingEnrichment (TTL), UpdateCVEEnrichment
(nullable *float64 -> SQL NULL), GetCVE, ArticlesForCVE, ListArticles (parameterized
--source/--since/--min-cvss/--kev/--keyword via EXISTS subqueries).
- commands: enrich, cve, list (cve/list off the stub list). Trimmed kev-sample.json
(7KB) fixture for KAT tests; nvd/epss fixtures committed.
- KAT proven offline AND live: CVE-2021-44228 -> CVSS 10.0 CRITICAL (3.1), CWE-20,
KEV added 2021-12-10 ransomware yes, EPSS 0.99999. 215 articles -> 82 CVEs, and
shared-CVE clustering lit up (multi-source clusters 2 -> 5). Suite offline + -race.
One read-only audit agent run; 0 Crit/High/Med, Low/Nit fixes applied in-phase.
NVD apiKey header set with exact case to bypass Go header canonicalization.
M3 React web app over the existing FastAPI: hex viewer, disassembly pane, section map, strings, challenge runner with reveal-after-solve, and progress. Scaffolded from the open no-auth react-scss template; components kept self-contained for CertGames extraction.
M3.5 static-analysis depth, all still no-execution and KAT-traced to the gate binary: PLT/GOT import resolution (plt.py), RIP-relative cross-references and callers (xref.py), basic-block control-flow graphs (cfg.py), and prologue-scan function discovery for stripped binaries (discover.py). Adds the 06-stripped-gate challenge. The API now resolves call names, exposes rip targets, and serves /cfg, /xrefs, and discovered functions.
M2 API hardening from a read-only audit: challenge-loader error isolation, a request body-size middleware, session length bounds, CORS, and the disasm gate annotation gated behind solved state so it no longer pre-solves the challenge.
96 backend tests green; frontend typecheck, build, and lint clean.
Group the same story across outlets into clusters, the velocity signal for
ranking. Runs automatically after 'scrape' (recompute-from-scratch, idempotent).
- internal/cluster: connected-components engine (union-find). Two articles join
when, within a time window (Cluster.WindowHours, default 72h), EITHER their
normalized-title token-set Jaccard >= threshold (default 0.6) AND they are from
different outlets, OR they share >= 1 CVE (source-agnostic). The cross-outlet
guard on the title edge kills a real false positive where two distinct CISA
advisories with identical boilerplate titles were merging. Cluster key = the
earliest article by time; SourceCount tracks distinct outlets so the reported
'multi-source' metric is truthful.
- Shared-CVE join reads article_cves, which is empty until M3 CVE extraction, so
that path is a natural no-op today and lights up in M3 with no rework.
- store: ClusterCandidates (effective time = published_at else fetched_at, bounded
by lookback), ArticleCVEMap, ReplaceClusters (single atomic tx: wipe + rebuild).
- config: Cluster.LookbackHours (default 168h) bounds the corpus for O(n^2);
validated as >= window_hours so window edges are never silently dropped.
- Proven live: 215 real articles -> 137 clusters, 2 genuinely cross-outlet
(FortiBleed x3, Scattered Spider x2). 17 cluster tests; suite offline + -race.
One read-only audit agent run; findings (multi-source stat, lookback guard) fixed in-phase.
Clustering is recompute-from-scratch each scrape; safe to re-run.
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.
- internal/fetch: per-host token-bucket rate limiting, conditional GET
(ETag/Last-Modified with 304 handling), retry+backoff that honors
Retry-After and treats timeouts/cancellation as non-retryable, per-source
deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
Client.Allowed() for the future HTML article-scrape path, not on subscribed
feed fetches (user-directed retrieval; several publishers blanket-disallow
generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
-> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
suite is offline and passes under -race. Proven live: cold run 215 new across
7 sources, warm run 0 new (304s + dedup).
Ctrl-C aborts scrape cleanly via signal-aware context.
Go engine that aggregates cybersecurity news and enriches CVEs. This
first commit lays the M0 foundation:
- config: fetch/enrich/cluster/rank/AI settings, YAML + validation,
defaults as named constants, AI opt-in and off by default
- source: Source registry with embedded default feeds (7 verified RSS
sources), external-file override, dedup/URL validation
- store: pure-Go SQLite (modernc), WAL + foreign_keys, forward-only
embedded migrations with schema_migrations tracking and a loud
newer-than-binary guard; ErrDuplicate sentinel for dedup
- cmd: cobra skeleton, version + sources wired, remaining commands
stubbed to their milestones
go vet / gofmt / go test all clean.
Framework-free analysis engine (hex dump, hand-rolled ELF64 parser,
capstone x86-64 disassembly, string extraction, byte patching, challenge
grading) behind a thin read-only FastAPI layer over curated pre-compiled
challenge binaries.
No-execution security posture: the backend only reads and decodes bytes,
never runs a target; patch challenges are graded by static byte diff.
Reveal-after-solve pedagogy: challenge source is returned only on a
correct submission. Input size-caps enforced at the API boundary.
M0 research (ELF format, x86-64/capstone, pedagogy, no-execution) is
cross-checked against a real compiled binary. 63 tests green, including
an ELF parser cross-check against pyelftools and disassembly that matches
objdump instruction-for-instruction.
Carter Perez
dependabot[bot]
surya