_PATTERN_RULES list had XXE_INJECTION (0.82) after XSS (0.80) and
CRLF_INJECTION (0.65) after PATH_TRAVERSAL (0.60), placing higher-
scored rules below lower-scored ones. Reordered to strict descending
order and updated the module docstring to match. No change to
scoring behavior — score_request() takes max + boost regardless
of list position.
overrides.vite was pinned to npm:rolldown-vite@7.2.5 while
devDependencies and pnpm-lock.yaml both resolved 7.3.1. The mismatch
can produce ERR_PNPM_OUTDATED_LOCKFILE on --frozen-lockfile installs.
Updated override to 7.3.1 to match the resolved version throughout
the lockfile. No lockfile edits required.
Moved entrypoint.sh copy target from /app/entrypoint.sh to
/entrypoint.sh in the dev Dockerfile. The dev compose bind mount
./backend:/app overlays /app entirely; placing the entrypoint above
that boundary ensures the image-baked script is never shadowed by
host directory contents. No startup failure was reproduced — this
is a preventive hardening change.
urllib.parse.quote(path, safe='/?&=') was re-encoding % to %25 in
already-encoded attack paths (XXE, XSS, traversal payloads). Replaced
with a manual first-? split: path component encoded with safe='/%\'
(preserving existing percent sequences and backslash traversal vectors),
query string passed through untouched. Covers all 53 attack payloads
across 7 simulation modes. Inline comments removed from XXE_PAYLOADS
to match the style of all other payload lists.
The final branch of _XXE_INJECTION was missing % separators between
hex pairs. Pattern %(?:25)?53(?:25)?59... never matched because after
consuming %53 the next sub-pattern expected 5 or 2, not %. Fixed by
inserting % anchors between each pair so both single-encoded
(%53%59%53%54%45%4D) and double-encoded (%2553%2559...) SYSTEM strings
match correctly. All existing XXE branches are unaffected.
This is a security-news tool; CVE enrichment is optional garnish, never a
setup tax. Drop the NVD API key from the default path entirely.
- internal/cve/cvelist: keyless CVE Program cvelistV5 client (raw GitHub,
carries CISA-ADP / vulnrichment CVSS). Parses cna and adp metrics with
version precedence v4.0 > v3.1 > v3.0 > v2.0, plus CWE and description;
404 maps to not-found; v2 severity is capped at HIGH. Introduces a
neutral CoreResult and a CVESource interface (NVDResult renamed to
CoreResult; NVDClient still satisfies it).
- enrich: Clients.Core is the CVESource. buildCoreSource picks the keyless
cvelist client by default and NVD only when NVD_API_KEY is set (booster).
- scrape auto-enriches referenced CVEs best-effort after clustering
(time-bounded, non-fatal, --no-enrich to skip); enrich kept as an
optional manual refresh. Default flow is scrape -> tui.
- ranking default weights rebalanced news-first (recency / velocity /
source / keyword 70 percent, cvss / kev / epss 30 percent; was 55 percent CVE).
- KAT fixture testdata/cvelist/CVE-2021-44228.json.
Interactive terminal UI over the ranked story clusters. Rave Maximal
cybercore theming: cyan-to-magenta spectrum score bars, severity-reactive
rows, hot KEV chips, cool-ramp outlet dots, and a scrollable detail
dossier with per-outlet links and full CVE cards (CVSS meter, KEV /
ransomware, EPSS, CWE, vector, description). Reuses store.DigestClusters
and rank.Rank verbatim and preloads full store.CVE records for detail.
- internal/tui: model/update/view state machine with an injected Loader
and opener, so Update is fully unit-testable without a terminal or DB
(nav / clamp / open / back / quit / resize / empty-store / too-small).
- o opens the selected story in the default browser, keyboard-only,
cross-platform, and refuses non-http urls.
- tui command replaces the M5 stub; supports --since.
- deps: charmbracelet bubbletea v1.3.10 + bubbles v1.0.0 + lipgloss v1.1.0.
M4:
- Add 01-read-the-hex (hex-reading) and 02-find-the-entry (elf-anatomy)
gradeable challenges over the gate binary, completing the five-module
pedagogy
- Swap in-memory progress for a SQLite ProgressStore behind the existing
Protocol; routes untouched, migration-free, thread-safe
M5:
- Self-host Docker: prod (nginx + uvicorn) and dev (nginx + Vite HMR) compose,
justfile, Dockerfiles, and nginx configs; zero-config clone-and-run
- learn/ teaching docs (00-04) and public README
- One-shot install.sh and uninstall.sh
Turn the aggregated store into a ranked, exportable digest of story clusters.
- internal/rank: pure deterministic Score(Signals, cfg.Rank) — recency
(exp half-life decay), cvss/10, kev, epss, velocity (cluster size/age), source
weight, watchlist keyword. Every signal clamped to [0,1]; all weights +
half_life + velocity_norm from config, no literals. recency/velocity are total
(guard half_life<1 and norm<=0). Rank() scores each cluster and stable-sorts
desc, tiebreak freshest. Research 06's two worked examples are the golden-order
test (A ~0.99 strictly above B ~0.13).
- internal/export: per-cluster (story) digest. Headline = freshest member, distinct
sorted outlets, CVEs sorted KEV-first then CVSS. Markdown + deterministic JSON.
- store.DigestClusters: 3-query per-cluster aggregation (clusters by since, member
articles, union of CVE signals) with explicit ORDER BY; per-cluster CVE dedup
via SELECT DISTINCT.
- digest command: --top/--since/--format md|json/--out. Off the stub list.
- Proven live: scrape -> digest --format md and json render correctly, per-cluster
(FortiBleed shows 3 outlets). Suite offline + -race.
One read-only audit agent; 0 Crit/High/Med, golden-order math verified; Low/Nit
fixed in-phase (top<=0 count, pure-fn guards, ORDER BY, headline sanitize, 2-cluster
attach test).
Digest ranks per-cluster (Carter's choice), not per-article.
The intelligence layer. scrape now extracts CVE IDs (cheap, no network) into
article_cves, which activates M2's shared-CVE clustering. A separate 'enrich'
command does the slow, cached API work (Carter's split: fast scrape, deliberate
enrich).
- internal/cve: regex CVE-ID extraction + three clients (stdlib net/http):
- nvd: API 2.0, CVSS precedence v4.0>v3.1>v3.0>v2 with nullable metrics and v2
metric-level severity fallback, totalResults==0 => not-found, apiKey header
(exact case), token-bucket rate limit (5/30s anon, 50/30s keyed) + retry on
429/5xx, timeouts non-retryable.
- kev: one catalog download -> membership map; knownRansomwareCampaignUse is the
STRING 'Known'/'Unknown' mapped explicitly to bool.
- epss: batched; epss/percentile are QUOTED STRINGS parsed with ParseFloat (a
parse failure is skipped, a legitimate 0.0 is kept); partial batches survive
a transient error.
- internal/enrich: enriches only unenriched/stale CVEs (positive + negative TTL),
KEV once per run, EPSS batched, NVD per-CVE. NVD/EPSS soft per-item (resumable),
KEV fatal by design (enriching without it would persist wrong is_kev for a TTL).
- store: UpsertCVEStub/LinkArticleCVE, CVEsNeedingEnrichment (TTL), UpdateCVEEnrichment
(nullable *float64 -> SQL NULL), GetCVE, ArticlesForCVE, ListArticles (parameterized
--source/--since/--min-cvss/--kev/--keyword via EXISTS subqueries).
- commands: enrich, cve, list (cve/list off the stub list). Trimmed kev-sample.json
(7KB) fixture for KAT tests; nvd/epss fixtures committed.
- KAT proven offline AND live: CVE-2021-44228 -> CVSS 10.0 CRITICAL (3.1), CWE-20,
KEV added 2021-12-10 ransomware yes, EPSS 0.99999. 215 articles -> 82 CVEs, and
shared-CVE clustering lit up (multi-source clusters 2 -> 5). Suite offline + -race.
One read-only audit agent run; 0 Crit/High/Med, Low/Nit fixes applied in-phase.
NVD apiKey header set with exact case to bypass Go header canonicalization.
M3 React web app over the existing FastAPI: hex viewer, disassembly pane, section map, strings, challenge runner with reveal-after-solve, and progress. Scaffolded from the open no-auth react-scss template; components kept self-contained for CertGames extraction.
M3.5 static-analysis depth, all still no-execution and KAT-traced to the gate binary: PLT/GOT import resolution (plt.py), RIP-relative cross-references and callers (xref.py), basic-block control-flow graphs (cfg.py), and prologue-scan function discovery for stripped binaries (discover.py). Adds the 06-stripped-gate challenge. The API now resolves call names, exposes rip targets, and serves /cfg, /xrefs, and discovered functions.
M2 API hardening from a read-only audit: challenge-loader error isolation, a request body-size middleware, session length bounds, CORS, and the disasm gate annotation gated behind solved state so it no longer pre-solves the challenge.
96 backend tests green; frontend typecheck, build, and lint clean.
Group the same story across outlets into clusters, the velocity signal for
ranking. Runs automatically after 'scrape' (recompute-from-scratch, idempotent).
- internal/cluster: connected-components engine (union-find). Two articles join
when, within a time window (Cluster.WindowHours, default 72h), EITHER their
normalized-title token-set Jaccard >= threshold (default 0.6) AND they are from
different outlets, OR they share >= 1 CVE (source-agnostic). The cross-outlet
guard on the title edge kills a real false positive where two distinct CISA
advisories with identical boilerplate titles were merging. Cluster key = the
earliest article by time; SourceCount tracks distinct outlets so the reported
'multi-source' metric is truthful.
- Shared-CVE join reads article_cves, which is empty until M3 CVE extraction, so
that path is a natural no-op today and lights up in M3 with no rework.
- store: ClusterCandidates (effective time = published_at else fetched_at, bounded
by lookback), ArticleCVEMap, ReplaceClusters (single atomic tx: wipe + rebuild).
- config: Cluster.LookbackHours (default 168h) bounds the corpus for O(n^2);
validated as >= window_hours so window edges are never silently dropped.
- Proven live: 215 real articles -> 137 clusters, 2 genuinely cross-outlet
(FortiBleed x3, Scattered Spider x2). 17 cluster tests; suite offline + -race.
One read-only audit agent run; findings (multi-source stat, lookback guard) fixed in-phase.
Clustering is recompute-from-scratch each scrape; safe to re-run.
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.
- internal/fetch: per-host token-bucket rate limiting, conditional GET
(ETag/Last-Modified with 304 handling), retry+backoff that honors
Retry-After and treats timeouts/cancellation as non-retryable, per-source
deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
Client.Allowed() for the future HTML article-scrape path, not on subscribed
feed fetches (user-directed retrieval; several publishers blanket-disallow
generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
-> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
suite is offline and passes under -race. Proven live: cold run 215 new across
7 sources, warm run 0 new (304s + dedup).
Ctrl-C aborts scrape cleanly via signal-aware context.
Go engine that aggregates cybersecurity news and enriches CVEs. This
first commit lays the M0 foundation:
- config: fetch/enrich/cluster/rank/AI settings, YAML + validation,
defaults as named constants, AI opt-in and off by default
- source: Source registry with embedded default feeds (7 verified RSS
sources), external-file override, dedup/URL validation
- store: pure-Go SQLite (modernc), WAL + foreign_keys, forward-only
embedded migrations with schema_migrations tracking and a loud
newer-than-binary guard; ErrDuplicate sentinel for dedup
- cmd: cobra skeleton, version + sources wired, remaining commands
stubbed to their milestones
go vet / gofmt / go test all clean.
Framework-free analysis engine (hex dump, hand-rolled ELF64 parser,
capstone x86-64 disassembly, string extraction, byte patching, challenge
grading) behind a thin read-only FastAPI layer over curated pre-compiled
challenge binaries.
No-execution security posture: the backend only reads and decodes bytes,
never runs a target; patch challenges are graded by static byte diff.
Reveal-after-solve pedagogy: challenge source is returned only on a
correct submission. Input size-caps enforced at the API boundary.
M0 research (ELF format, x86-64/capstone, pedagogy, no-execution) is
cross-checked against a real compiled binary. 63 tests green, including
an ELF parser cross-check against pyelftools and disassembly that matches
objdump instruction-for-instruction.
connect-scan: raw non-blocking connect() + poll(POLLOUT) + SO_ERROR via std.os.linux (std.Io.net connect-with-timeout is unimplemented in 0.16); own sized std.Io.Threaded with N io.async workers off a mutex+token-bucket dispenser; --backend connect / --connect, --concurrency, --connect-timeout.
cloud/VM detection: two-socket AF_PACKET self-probe (send on one, observe the tagged frame on a second - single-socket cannot see its own send); --backend auto checks CAP_NET_RAW + egress and auto-falls-back to connect with a notice; forced raw backend disables fallback and hard-fails; zero-response raw scan hints --connect.
raw IPv6 SYN: packet.Addr union result + RFC5952 v6 render; Ipv6Hdr + pseudoChecksum6/tcpChecksum6; 128-bit SipHash cookie (generate6/seq6); parseIpv6, resolveSrcIp6 (/proc/net/if_inet6), defaultGateway6 (/proc/net/ipv6_route); Engine6 (bounded prefix, RFC6890 reserved floor, ::/0 reject, host cap); SynTemplate6; classifyTcp6 + ICMPv6 type1; ndp.zig NDP neighbor resolution; runV6Scan dispatch; connect-path v6 rides the connect engine. v6 scope = TCP SYN.
240 tests Debug+ReleaseSafe (-Dxdp on/off); KAT + two-namespace netns e2e proven on the wire; two read-only audits, 0 Critical/High/Medium, all findings fixed in-phase.
OS-realistic SYN templates (Linux/Windows/macOS/masscan JA4T option chains
plus varying IP-id), Poisson jitter, source-port rotation (RX recomputes off
the reply, zero classify changes), scoped RST-suppression (iptables plus
ambient CAP_NET_ADMIN, self-healing delete-before-insert), decoys (bogon-free
RND, real probe always sent), and FIN/NULL/Xmas/Maimon/ACK/Window flag scans
with per-mode cookie matching plus State.unfiltered.
Dead theater (idle scan, fragmentation, TTL, MAC/source-route spoof, badsum)
omitted and documented as obsolete with citations.
Cursor-based TX emission with token refund plus cold-start pacing; non-stealth
path byte-identical to M7.
Turn the scan from sequential TX-then-RX into two io.concurrent engines on
std.Io.Threaded, fixing the M4 bug where replies arriving during transmit were
dropped and the RX deadline (anchored at socket-open) expired before draining.
The main thread is the non-blocking Io.Queue consumer and dashboard renderer.
- rx: Receiver keyed off a shared tx_done atomic; new pure planDrain anchors the
drain window at TX-completion, not socket-open, killing the quiet-gap early exit
during a slow TX. Hard-cap safety backstop anchored at drain start.
- output (new): cache-line-padded atomic Stats, truecolor/256/none palette
(violet gradient, neon-green, muted-gray), in-place multi-line live dashboard,
Unicode results table, NDJSON to stdout with visuals on stderr, full
NO_COLOR/CLICOLOR_FORCE/COLORTERM/--color chain plus a narrow-terminal fallback.
- tx: wall-clock deadline so a stalled TX ring can no longer hang the scan.
- cli: banner recolored to the violet gradient and routed through the color chain.
- targets: add 192.88.99.0/24 (6to4 relay anycast) to the RFC 6890 exclude floor.
78/78 unit tests pass under Debug and ReleaseSafe; proven end-to-end (open/closed,
dedup, dashboard, NDJSON) in an unshare -r -n netns under ReleaseSafe.
Carter Perez
dependabot[bot]
surya