RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Cybersecurity-Projects/PROJECTS/intermediate/security-news-scraper
History
CarterPerez-dev b41276004f feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 
Concurrent, rate-limited feed ingestion wired end to end: 'scrape' pulls
live security news into the SQLite store, idempotent across re-runs.

- internal/fetch: per-host token-bucket rate limiting, conditional GET
  (ETag/Last-Modified with 304 handling), retry+backoff that honors
  Retry-After and treats timeouts/cancellation as non-retryable, per-source
  deadline, honest User-Agent, 16MB body cap. robots.txt is enforced only via
  Client.Allowed() for the future HTML article-scrape path, not on subscribed
  feed fetches (user-directed retrieval; several publishers blanket-disallow
  generic bots yet serve a public feed).
- internal/parse: gofeed RSS/Atom to a normalized Item; RFC1123Z/RFC822Z/
  RFC3339 time fallbacks (CISA emits RFC822Z, not RFC1123Z as first researched).
- internal/normalize: canonical URL (lowercase scheme+host, drop fragment,
  strip utm_*/gclid/fbclid/ref/mc_cid/mc_eid, drop trailing slash), title
  normalization, goquery HTML strip, sha256 content+title hashes.
- internal/ingest: errgroup fan-out over sources, fail-soft per source, fetch
  -> parse -> normalize -> InsertArticle dedup via store.ErrDuplicate, plus
  fetch_state upsert for conditional GET.
- store: migration 0002 adds articles.title_hash; GetFetchState/UpsertFetchState.
- Trimmed testdata/feeds fixtures (3 items each) back golden parse tests; full
  suite is offline and passes under -race. Proven live: cold run 215 new across
  7 sources, warm run 0 new (304s + dedup).

Ctrl-C aborts scrape cleanly via signal-aware context.
 		2026-07-05 14:51:57 -04:00
..
cmd/nadezhda feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
internal feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
testdata/feeds feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
.gitignore feat(nadezhda): M0 scaffold for security news + CVE aggregator 2026-07-05 13:22:08 -04:00
README.md feat(nadezhda): M0 scaffold for security news + CVE aggregator 2026-07-05 13:22:08 -04:00
go.mod feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
go.sum feat(nadezhda): M1 ingestion core (fetch, parse, normalize, dedup) 2026-07-05 14:51:57 -04:00
justfile feat(nadezhda): M0 scaffold for security news + CVE aggregator 2026-07-05 13:22:08 -04:00

README.md

Nadezhda

A concurrent security-news and CVE aggregation engine, written in Go.

Nadezhda ingests cybersecurity news from reliable RSS feeds, enriches every referenced CVE with authoritative exploit intelligence (NVD, CISA KEV, FIRST EPSS), clusters the same story across outlets, ranks items by real-world significance, and surfaces content angles. It ships as a single static binary with a local SQLite store, a colorful terminal UI, and Markdown/JSON export.

Status

Early development. The scaffold is in place: configuration, source registry, SQLite store with forward-only migrations, and the command skeleton.

nadezhda version     # print version
nadezhda sources     # list configured feeds and persist them to the store

Ingestion, CVE enrichment, ranking, the TUI, and the AI ideation layer land in subsequent milestones.

Build

just build     # -> ./nadezhda
just test

Requires Go 1.25+.

Full documentation lands in learn/ as the project matures.