Add a watch command (--interval, --once, --no-enrich): a long-lived daemon that re-ingests every enabled source on a ticker and, when a webhook is configured, POSTs genuinely new high-signal stories to it (Slack, Discord, or any JSON endpoint via a text/content/items payload).
internal/watch is a pure, stdlib-only scheduler: the pipeline is injected as a Cycle closure, so the daemon is fully unit-testable with a fake ticker and notifier. Graceful shutdown on SIGINT/SIGTERM returns cleanly; a cycle error is logged and the loop continues (fail-soft).
Notify uses a dedicated fetch-time watermark (store.NewlyFetchedClusters) so a freshly ingested but older-dated advisory is surfaced, which a publish-time filter would drop. Notable stories are capped by watch.notify_max_items and filtered by score threshold or KEV status.
Extract the shared ingest and cluster sequence (pipeline.go) so scrape and watch cannot drift, and route scrape and enrich through cmd.Context() so they honor SIGTERM too.
Add an opt-in ideation layer that turns ranked story clusters into content angles. A Provider interface backs four models: qwen (default, local Ollama), OpenAI, and Gemini share one OpenAI-compatible client; an Anthropic Messages client is written against raw net/http (no vendor SDK, uniform + dep-light). Results persist in ai_notes with a UNIQUE(cluster_id, provider) index (migration 0003), so re-ideation upserts in place.
The ideate command skips any (cluster, provider) that already has a note unless --force is passed; benign classifier refusals get their own counter and never fail the run. In the TUI, i ideates the selected cluster inline and renders an AI IDEAS section in the dossier; it is a no-op unless AI is enabled and only acts in the detail view.
Add a re-runnable interactive wizard that configures an AI provider for the ideation layer: Claude, OpenAI, Gemini, or a local Ollama. Keys persist to ~/.config/nadezhda/credentials (0600, atomic temp+rename) and auto-load on every run; a shell-exported key always wins over the file. A key-name allowlist (NADEZHDA_* / *_API_KEY) blocks env-name injection into spawned subprocesses.
The Ollama path detects a running instance and auto-configures it, else writes an embedded compose file plus one docker command, else prints an install hint. Ollama's host port is remapped to a non-default 39847 (OLLAMA_HOST_PORT) to avoid colliding with an existing Ollama on 11434; the container-internal port stays 11434.
Thread a signal-cancellable root context through cobra (ExecuteContext). Default the Anthropic model to claude-sonnet-4-6.
This is a security-news tool; CVE enrichment is optional garnish, never a
setup tax. Drop the NVD API key from the default path entirely.
- internal/cve/cvelist: keyless CVE Program cvelistV5 client (raw GitHub,
carries CISA-ADP / vulnrichment CVSS). Parses cna and adp metrics with
version precedence v4.0 > v3.1 > v3.0 > v2.0, plus CWE and description;
404 maps to not-found; v2 severity is capped at HIGH. Introduces a
neutral CoreResult and a CVESource interface (NVDResult renamed to
CoreResult; NVDClient still satisfies it).
- enrich: Clients.Core is the CVESource. buildCoreSource picks the keyless
cvelist client by default and NVD only when NVD_API_KEY is set (booster).
- scrape auto-enriches referenced CVEs best-effort after clustering
(time-bounded, non-fatal, --no-enrich to skip); enrich kept as an
optional manual refresh. Default flow is scrape -> tui.
- ranking default weights rebalanced news-first (recency / velocity /
source / keyword 70 percent, cvss / kev / epss 30 percent; was 55 percent CVE).
- KAT fixture testdata/cvelist/CVE-2021-44228.json.
Interactive terminal UI over the ranked story clusters. Rave Maximal
cybercore theming: cyan-to-magenta spectrum score bars, severity-reactive
rows, hot KEV chips, cool-ramp outlet dots, and a scrollable detail
dossier with per-outlet links and full CVE cards (CVSS meter, KEV /
ransomware, EPSS, CWE, vector, description). Reuses store.DigestClusters
and rank.Rank verbatim and preloads full store.CVE records for detail.
- internal/tui: model/update/view state machine with an injected Loader
and opener, so Update is fully unit-testable without a terminal or DB
(nav / clamp / open / back / quit / resize / empty-store / too-small).
- o opens the selected story in the default browser, keyboard-only,
cross-platform, and refuses non-http urls.
- tui command replaces the M5 stub; supports --since.
- deps: charmbracelet bubbletea v1.3.10 + bubbles v1.0.0 + lipgloss v1.1.0.
Turn the aggregated store into a ranked, exportable digest of story clusters.
- internal/rank: pure deterministic Score(Signals, cfg.Rank) — recency
(exp half-life decay), cvss/10, kev, epss, velocity (cluster size/age), source
weight, watchlist keyword. Every signal clamped to [0,1]; all weights +
half_life + velocity_norm from config, no literals. recency/velocity are total
(guard half_life<1 and norm<=0). Rank() scores each cluster and stable-sorts
desc, tiebreak freshest. Research 06's two worked examples are the golden-order
test (A ~0.99 strictly above B ~0.13).
- internal/export: per-cluster (story) digest. Headline = freshest member, distinct
sorted outlets, CVEs sorted KEV-first then CVSS. Markdown + deterministic JSON.
- store.DigestClusters: 3-query per-cluster aggregation (clusters by since, member
articles, union of CVE signals) with explicit ORDER BY; per-cluster CVE dedup
via SELECT DISTINCT.
- digest command: --top/--since/--format md|json/--out. Off the stub list.
- Proven live: scrape -> digest --format md and json render correctly, per-cluster
(FortiBleed shows 3 outlets). Suite offline + -race.
One read-only audit agent; 0 Crit/High/Med, golden-order math verified; Low/Nit
fixed in-phase (top<=0 count, pure-fn guards, ORDER BY, headline sanitize, 2-cluster
attach test).
Digest ranks per-cluster (Carter's choice), not per-article.
The intelligence layer. scrape now extracts CVE IDs (cheap, no network) into
article_cves, which activates M2's shared-CVE clustering. A separate 'enrich'
command does the slow, cached API work (Carter's split: fast scrape, deliberate
enrich).
- internal/cve: regex CVE-ID extraction + three clients (stdlib net/http):
- nvd: API 2.0, CVSS precedence v4.0>v3.1>v3.0>v2 with nullable metrics and v2
metric-level severity fallback, totalResults==0 => not-found, apiKey header
(exact case), token-bucket rate limit (5/30s anon, 50/30s keyed) + retry on
429/5xx, timeouts non-retryable.
- kev: one catalog download -> membership map; knownRansomwareCampaignUse is the
STRING 'Known'/'Unknown' mapped explicitly to bool.
- epss: batched; epss/percentile are QUOTED STRINGS parsed with ParseFloat (a
parse failure is skipped, a legitimate 0.0 is kept); partial batches survive
a transient error.
- internal/enrich: enriches only unenriched/stale CVEs (positive + negative TTL),
KEV once per run, EPSS batched, NVD per-CVE. NVD/EPSS soft per-item (resumable),
KEV fatal by design (enriching without it would persist wrong is_kev for a TTL).
- store: UpsertCVEStub/LinkArticleCVE, CVEsNeedingEnrichment (TTL), UpdateCVEEnrichment
(nullable *float64 -> SQL NULL), GetCVE, ArticlesForCVE, ListArticles (parameterized
--source/--since/--min-cvss/--kev/--keyword via EXISTS subqueries).
- commands: enrich, cve, list (cve/list off the stub list). Trimmed kev-sample.json
(7KB) fixture for KAT tests; nvd/epss fixtures committed.
- KAT proven offline AND live: CVE-2021-44228 -> CVSS 10.0 CRITICAL (3.1), CWE-20,
KEV added 2021-12-10 ransomware yes, EPSS 0.99999. 215 articles -> 82 CVEs, and
shared-CVE clustering lit up (multi-source clusters 2 -> 5). Suite offline + -race.
One read-only audit agent run; 0 Crit/High/Med, Low/Nit fixes applied in-phase.
NVD apiKey header set with exact case to bypass Go header canonicalization.
Group the same story across outlets into clusters, the velocity signal for
ranking. Runs automatically after 'scrape' (recompute-from-scratch, idempotent).
- internal/cluster: connected-components engine (union-find). Two articles join
when, within a time window (Cluster.WindowHours, default 72h), EITHER their
normalized-title token-set Jaccard >= threshold (default 0.6) AND they are from
different outlets, OR they share >= 1 CVE (source-agnostic). The cross-outlet
guard on the title edge kills a real false positive where two distinct CISA
advisories with identical boilerplate titles were merging. Cluster key = the
earliest article by time; SourceCount tracks distinct outlets so the reported
'multi-source' metric is truthful.
- Shared-CVE join reads article_cves, which is empty until M3 CVE extraction, so
that path is a natural no-op today and lights up in M3 with no rework.
- store: ClusterCandidates (effective time = published_at else fetched_at, bounded
by lookback), ArticleCVEMap, ReplaceClusters (single atomic tx: wipe + rebuild).
- config: Cluster.LookbackHours (default 168h) bounds the corpus for O(n^2);
validated as >= window_hours so window edges are never silently dropped.
- Proven live: 215 real articles -> 137 clusters, 2 genuinely cross-outlet
(FortiBleed x3, Scattered Spider x2). 17 cluster tests; suite offline + -race.
One read-only audit agent run; findings (multi-source stat, lookback guard) fixed in-phase.
Clustering is recompute-from-scratch each scrape; safe to re-run.
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.
- internal/fetch: per-host token-bucket rate limiting, conditional GET
(ETag/Last-Modified with 304 handling), retry+backoff that honors
Retry-After and treats timeouts/cancellation as non-retryable, per-source
deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
Client.Allowed() for the future HTML article-scrape path, not on subscribed
feed fetches (user-directed retrieval; several publishers blanket-disallow
generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
-> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
suite is offline and passes under -race. Proven live: cold run 215 new across
7 sources, warm run 0 new (304s + dedup).
Ctrl-C aborts scrape cleanly via signal-aware context.
Go engine that aggregates cybersecurity news and enriches CVEs. This
first commit lays the M0 foundation:
- config: fetch/enrich/cluster/rank/AI settings, YAML + validation,
defaults as named constants, AI opt-in and off by default
- source: Source registry with embedded default feeds (7 verified RSS
sources), external-file override, dedup/URL validation
- store: pure-Go SQLite (modernc), WAL + foreign_keys, forward-only
embedded migrations with schema_migrations tracking and a loud
newer-than-binary guard; ErrDuplicate sentinel for dedup
- cmd: cobra skeleton, version + sources wired, remaining commands
stubbed to their milestones
go vet / gofmt / go test all clean.
- add 1200x630 og-image.jpg plus Open Graph and Twitter Card meta tags so
the live demo renders a rich preview when shared (LinkedIn, etc.)
- omit og:url to avoid crawler cache canonicalizing share URLs back to a
stale entry
- justfile: add redeploy recipe (tunnel down, cached build, up -d with the
backend profile and remove-orphans); strip the exported dev .env overrides
so it deploys under the prod project name and ports
- install.sh: clearer live-capture hint (setcap, then run without sudo)
Add a --report flag to the pcap command that prints one forensic summary of
a whole capture (endpoint inventory, fingerprint distribution, intel verdicts,
detection alerts, and a coverage section with the TLS miss rate and throughput)
instead of the per-event stream, folding in intel and detection automatically
when a seeded database is present. Add a tls_handshakes_fingerprinted counter
and Counters::tls_miss_rate so truncated and multi-segment ClientHellos the
capture clipped are reported rather than mistaken for absence. Add criterion
benches over the vendored captures and the parse/hash hot path. Plus frontend
landing and asset polish.
Passive TLS client and server fingerprinting from packet captures. Two-crate
workspace: tlsfp-core (parsing and fingerprint logic, no I/O, unsafe forbidden)
and tlsfp (clap CLI). Edition 2024, MSRV 1.85.
Core (M0-M4): a hand-rolled bounds-checked TLS parser for ClientHello,
ServerHello, and the TLS 1.2 Certificate chain, plus the full fingerprint
family computed byte-exact against published vectors: JA3/JA3S (MD5), JA4/JA4_r,
JA4S, JA4H, JA4X, and JA4T. GREASE is stripped, wire order and unknown cipher
and extension values are preserved.
Pipeline (M5): a pcap and pcapng file reader behind a PacketSource trait,
etherparse L2-L4 decode (Ethernet, raw IP, BSD loopback, Linux SLL and SLL2,
VLAN and QinQ), and a hand-rolled per-flow TCP reassembler with wrapping serial
arithmetic, out-of-order parking, first-write-wins overlap resolution, and
bounded memory. The tlsfp pcap subcommand fingerprints every handshake in a
capture, as readable lines or NDJSON.
Verified three ways: 94 tests including unit KATs, nine real-pcap full-pipeline
KATs against the FoxIO sample corpus, and property tests that reassemble
arbitrarily segmented and reordered streams. Builds clean on stable and MSRV
1.85; clippy -D warnings and rustfmt clean.
Test captures under testdata/pcap are vendored unmodified from the FoxIO ja4
repository with per-file SHA-256 provenance.
pnpm latest on Node 22 resolves to pnpm 11 which rejects lockfileVersion
9.0 lockfiles with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH. Pinning to pnpm 10
keeps compatibility with existing lockfiles. .npmrc sets
strict-dep-builds=false so build-script warnings don't hard-fail the
frozen install.
Node.js bumped to 22 in lint workflow (pnpm 11.x requires >=22.13).
Extract goconst-flagged string literals in secrets-scanner to named
constants. Carry along monitor-dashboard docker/package fixes.
Closes 36 findings from the project audit. The headline win is correctness:
the daemon now actually does what the README claims, all three audit-log
integrity layers verify, and the rotation control loop terminates.
Critical fixes:
- Orchestrator bumps Credential.last_rotated_at on success and writes a
sealed credential_versions row, so the policy evaluator stops re-firing
the same rotation every tick
- Envelope encryption (AES-256-GCM, KEK-wrapped DEK, AAD-bound) is now
wired into the rotation success path; the credential_versions table is
no longer dead schema
- cre audit verify checks all three layers (hash chain + HMAC ratchet +
Merkle batches) instead of only the chain
- BatchSealerScheduler runs every 300s and on shutdown so audit_batches
actually fills with signed Merkle roots
- Compliance bundle exports the real audit_batches list and covers
public_key.pem under the manifest's checksum
- cre run/watch hard-fail without CRE_HMAC_KEY_HEX + CRE_KEK_HEX rather
than silently using zero defaults
Quality fixes:
- HTTP::Client wrapper with connect/read timeouts + bounded retry-with-
jitter on 408/429/5xx; Telegram error logs redact the bot token
- EventBus dispatch isolates Block subscribers behind select+timeout so
one stuck subscriber can't pin the bus
- RotationWorker checks rotations.in_flight before dispatching to dedupe
duplicate schedules; PG enforces it at the DB via partial unique index
- Commit-step failures transition rotations to Inconsistent (a terminal
state) and raise a critical alert instead of pretending success
- env_file rotator uses per-PID pending paths plus an exclusive flock on
a sibling .lock file so two daemons can't race on the rename
- Versioned migration runner replaces the soup of CREATE IF NOT EXISTS;
SQLite gains BEFORE UPDATE / BEFORE DELETE triggers on audit_events
- AuditSubscriber publishes a critical alert + panics by default when
log writes fail, instead of silently dropping
- Engine.stop drains via an ack channel with a 2s deadline rather than
the previous magic 50ms sleep
- Policy DSL moved into CRE::Policy::Dsl module (consumers `include` it);
multiple matching policies raise PolicyConflictError
- Evaluator uses credentials.overdue() per-max_age group instead of
loading every credential each tick
- AuditRepo gains each_in_range streaming and all_batches enumeration;
bundle export streams instead of buffering
- New cre verify-bundle <zip> CLI re-runs every check the bundle README
documents; new cre tui-demo for synthetic event preview
Tests: 179 unit (up from 159), all passing. New rotation_worker_spec,
new SigV4 regression vector, new orchestrator success-path coverage
(envelope write + last_rotated_at bump + Inconsistent state).
Docs: README + learn/00..04 aligned with the post-audit behavior;
removed /snooze (was a stub) and the AwsIamKey/Database CredentialKind
variants that had no rotators. Added required-env-var documentation.
Crystal's compiler warns when an overriding method's parameter name
differs from the parent's, since named-argument callers would pass
the wrong slot. Underscore-prefix was meant as 'unused' but it changed
the parameter name, which Crystal correctly flagged.
Match the parent signature (s : Domain::NewSecret) and use the explicit
'_ = s' idiom inside the body to mark it intentionally unused. Now
'shards build cre' compiles with zero warnings.
CarterPerez-dev
dependabot[bot]