project-nomad/admin/app/utils
John Cortright 68bb510351 fix(rag): reap orphaned qdrant sources during sync (#1170)
scanAndSyncStorage builds sourcesInQdrant and embeddableFiles in the
same pass but only ever checks one direction - is this on-disk file
already embedded? It never checks the reverse, so points left behind
by a deleted or replaced ZIM just sit in Qdrant forever and keep
showing up as search hits for content that doesn't exist anymore.

Added a reverse sweep that diffs the two sets and purges anything left
in Qdrant with no file backing it. Guarded so it's a no-op when the
disk scan comes back empty, so a filesystem hiccup can't get misread
as "delete everything."

Also has to exclude README.md/docs, since those get embedded by
discoverNomadDocs and live outside the kb_uploads/zim paths the
scanner actually walks. First pass without that exclusion would have
wiped the docs KB on every single sync - caught it by testing against
a live install where those sources were already indexed.

Verified live: found exactly one real orphan out of 29 indexed
sources (an old devdocs_en_bash zim superseded by a newer download),
purged it, and left the other 28 - including all the docs - alone.
2026-08-14 15:24:43 -07:00
..
eval feat(eval): standardized RAG quality evaluation harness (#1233) 2026-08-13 11:10:47 -07:00
affirmative_remedies.ts Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 2026-08-04 03:38:33 +00:00
amd_hsa_override.ts fix(amd): coerce gfx1103 (780M) to HSA_OVERRIDE 11.0.0 so it stays on GPU (#1134) 2026-08-04 03:38:36 +00:00
content_auto_update_backoff.ts feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
content_reindex_decision.ts feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
downloads.ts fix(downloads): don't retry a rejected entitlement for four hours (#1205) 2026-08-04 03:40:02 +00:00
fs.ts fix(rag): add proper .docx text extraction via mammoth (#1100) 2026-08-04 03:37:42 +00:00
hosted_content.ts feat: support gated downloads for self-hosted curated content (#1172) 2026-08-04 03:39:55 +00:00
hosted_content_auth.ts feat: support gated downloads for self-hosted curated content (#1172) 2026-08-04 03:39:55 +00:00
image_disk_preflight.ts feat(supply-depot): opt-in automatic updates for installed apps 2026-06-23 04:46:52 +00:00
kb_ingest_decision.ts feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4) (#894) 2026-05-20 10:16:00 -07:00
kb_job_health.ts feat(KB): status pill + last-activity timestamp on Processing Queue (RFC #883 §5/§10) 2026-05-20 10:16:00 -07:00
kb_orphan_decision.ts fix(rag): reap orphaned qdrant sources during sync (#1170) 2026-08-14 15:24:43 -07:00
kb_ratio_lookup.ts fix(KB): stop partial_stall warning firing on atypical ZIMs (link-out/PDF-heavy) 2026-06-23 04:46:54 +00:00
kb_warning_decision.ts refactor(KB): move FileWarning to shared types/rag following existing convention 2026-05-20 10:16:00 -07:00
misc.ts feat(AI): specify a model for ancillary tasks (#1244) 2026-08-13 21:30:04 -07:00
platform_metadata.ts feat(benchmark): official multi-arch sysbench, resolved digest, platform metadata (#1158) 2026-08-04 03:39:53 +00:00
rag_prompt.ts feat(eval): standardized RAG quality evaluation harness (#1233) 2026-08-13 11:10:47 -07:00
superseded_resource.ts fix(content): remove superseded curated map/ZIM files when a new version installs 2026-06-23 04:47:01 +00:00
update_window.ts feat(supply-depot): opt-in automatic updates for installed apps 2026-06-23 04:46:52 +00:00
version.ts feat: support for updating services 2026-03-11 14:08:09 -07:00
zim_download_resolution.ts feat: support gated downloads for self-hosted curated content (#1172) 2026-08-04 03:39:55 +00:00
zim_filename.ts fix(ZIM): preserve co-existing Wikipedia corpora on cleanup (#884) 2026-05-20 10:16:00 -07:00
zim_html.ts chore(KB): filter non-content sections + render tables in ZIM extraction (#1044) 2026-08-04 03:37:30 +00:00