* feat(drug-reference): offline FDA drug labels, conditions, and remedies
Adds an offline medical-reference feature with three coupled layers:
- Drug Reference: full-text search over openFDA drug-label indications,
a detail page per label, and a side-by-side single-drug comparison view.
A two-phase background pipeline downloads the openFDA label parts to the
storage volume (resumable) and ingests them into the search table.
- Conditions ("When to use what"): a curated spine of first-aid situations
that maps each situation to matching OTC drugs, each linking back to its
Drug Reference detail page.
- Curated remedies: hand-authored natural and home-remedy entries drawn from
US-government public-domain sources (NCCIH, CDC, MedlinePlus, FDA), shown
with their source links and the same safety disclaimers as the rest of the
feature.
The drug-reference and conditions layers are intentionally coupled: the drug
detail page shows the situations a drug treats, and the conditions controller
reads the same drug_labels table.
Safety surfaces ship as written. The amber SafetyBanner ("informational only,
not medical advice, not an FDA endorsement, not a drug-interaction checker, in
an emergency call emergency services") renders on the condition pages and the
search page; the detail and comparison pages carry their own "not a cross-drug
interaction checker" callout; and every page carries the openFDA CC0 source
citation and no-FDA-affiliation footer.
Wiring on this branch:
- start/routes.ts: the /drug-reference and /conditions page GETs plus their
/api/* groups.
- commands/queue/work.ts: the drug-download and drug-ingest queues, both at
concurrency 1. The two drug queues get a per-queue stall override
(lockDuration 1_800_000, maxStalledCount 3) because each part is one long
stream; every other queue keeps the existing 300000 default.
- inertia/pages/home.tsx: Drug Reference and "When to use what" tiles. The
icon and display_order are a starting point, open to change.
- types/kv_store.ts: the two drugReference.* keys the pipeline reads and writes.
- package.json: yauzl and stream-json (plus their @types), used by the ingest
job to stream the label JSON out of the downloaded zips.
The app reads the conditions and remedy data from the compiled TS constants in
app/data/; the repo-root collections/*.json files are the browseable mirrors.
The natural-remedies standalone test reads collections/natural_remedies.json to
assert the two stay in sync, so that file is also a test fixture.
The four standalone tests pass (drug_interactions, drug_ingest_status,
conditions, natural_remedies). tsc reports no errors in the feature code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(drug-reference): drop a fork-internal comment reference
* fix(drug-reference): address review — defer FULLTEXT, fallback remedies prop, strip fork refs
- migration: wrap the FULLTEXT ALTER in this.defer so it runs after the
deferred createTable (was silently swallowed, index never created)
- controller: add remedies:[] to the index() error fallback (required prop)
- strip fork-internal issue/spec references from ported comments
- tsconfig: exclude tests/standalone (node --experimental-strip-types only)
- correct the varchar(768) byte-math comment; extend remedy-spine test
* fix(drug-reference): make the interaction comparison readable at five drugs
The comparison view laid its columns out on an equal-fraction CSS grid
(repeat(N, minmax(0, 1fr))), so each added drug shrank every column; at the
five-drug maximum the FDA interaction text was squeezed into unreadable slivers.
Lay the columns out with flex instead: full-width and stacked on phones, then
fixed-width columns that scroll sideways from the sm: breakpoint up, so they
never shrink below a readable width. Theme the columns with the same palette as
the rest of the page (they were on stock gray), and give the headers a fixed
min-height so columns line up when drug names wrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(collections): route a 'dataset' tier resource to the drug pipeline
Add an optional `type` discriminator to SpecResource ('zim' | 'dataset',
absent == 'zim'), so the tier installer can carry a DB-ingested resource
alongside ZIM files. ZimService.downloadCategoryTier branches on it: a
'dataset' resource dispatches the existing FDA download+ingest pipeline
instead of RunDownloadJob, guarded against duplicate dispatch by the drug
ingest status. Every existing manifest entry has no `type` and keeps the
exact ZIM path.
Widen InstalledResource.resource_type to include 'dataset' and exclude
dataset rows from the ZIM/map catalog-update scan (datasets aren't
filename-versioned; their freshness path is separate). No dataset rows are
written yet: the InstalledResource 'dataset' row on ingest-ready, the
manifest entry, install-gating, and the downloads-aggregator integration
are follow-up commits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): rework into an opt-in medicine-standard tier dataset
Reshapes the offline FDA drug reference from a built-in feature into curated
content installed by selecting the Medicine / Standard tier, per maintainer
direction.
- install-state: the ingest writes an installed_resources 'dataset' row on
ready (version = the openFDA export_date), threaded installer to download to
ingest; the tier-status math and the home-tile gate read it. Manual ingests
write no row, so install-state stays tied to the curated path.
- manifest: declare the dataset in the medicine-standard tier (runtime fetches
the remote manifest, so this also needs to land upstream).
- install-gating: the drug-reference home tiles render only when installed.
- uninstall: DrugReferenceService.uninstall() stops the two drug queues, deletes
the on-disk parts, truncates drug_labels (schema kept), clears the KV markers,
and drops the install row. Best-effort, logged, scoped to drug data only.
- downloads: the download phase reports the canonical {percent, downloadedBytes,
totalBytes} shape as one drug-data card in the Active Downloads aggregator with
cancel/remove; the heavy ingest stays in the IngestStatus surface with an
Indexing handoff on the card.
- auto-update: a daily DrugAutoUpdateJob compares the manifest export_date and
re-downloads when newer, gated on installed + no active job.
typecheck clean; the drug standalone suites pass. Three points are flagged in
code for the maintainer: the InstalledResource 'dataset' approach, the tier
home, and the export_date string format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(downloads): keep the drug download card live across all parts
The Active Downloads card filtered to the deterministic jobId, but the
download's continuations run under auto-generated jobIds (only part 0 uses the
deterministic one). So the card tracked part 0 and then vanished while parts
2..N kept downloading. The queue is concurrency 1, so collapse to whichever
single part is in flight and report the deterministic jobId: one card tracks
aggregate progress through the whole download and cancel/remove still routes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): co-locate a persistent safety note with affirmative remedy guidance
Add RemedySafetyNote at the head of every natural-remedy section — the two on
Drug Reference and the one on "When to use what" — so the "informational only,
not medical advice, seek real medical care in an emergency" framing appears with
the guidance itself, not only in the page-top banner. Replaces the terse
per-section caveat with the same amber alert language as SafetyBanner.
Addresses the upstream #1040 review request that the disclaimer be unmistakable
and present wherever affirmative self-care guidance appears, not a one-time banner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(drug-reference): fold dataset freshness into the content-auto-update path
The drug dataset auto-updated on its own daily cron that ignored the
contentAutoUpdate.* master switch, so it would refresh even with content
auto-update turned off, and it didn't ride the content-update path the way
ZIMs and maps do.
Move the export_date freshness/apply orchestration onto
DrugReferenceService.attemptAutoUpdate(), add
ContentAutoUpdateService.attemptDrugDataset() gated on the same enabled +
window config, and have the hourly ContentAutoUpdateJob drive both. Retire
the standalone DrugAutoUpdateJob. The ZIM/map attempt() path is unchanged.
Addresses the upstream #1040 request to wire the openFDA export_date check
into the content updater so the dataset updates alongside ZIMs and maps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): strengthen the remedy note to consult a clinician before combining with meds
Widen the affirmative-remedy safety note from "talk to a clinician before use"
to explicitly cover using a remedy AND combining one with a medication the user
already takes — the interaction case is the higher-risk path for an off-grid
user self-treating.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): gate affirmative remedy content behind an off-by-default flag
Add drugReference.remediesEnabled (default off), independent of the tier
install. When off, the server emits no remedy data at any boundary — the
drug-reference page prop, conditions show, and the /api/conditions/drugs
situation search — and the "Natural" filter is hidden, so installing the
medicine-standard tier lights up the verbatim FDA label search and the
condition-to-OTC matching but not the hand-authored self-care and herbal
sections. No user-facing toggle: it is flipped on after a clinician content-pass.
Implements the upstream #1040 split-by-risk request: the regulated label content
ships with the tier; the authored remedy guidance stays gated until sign-off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
* feat(creator-packs): backend rail for gated creator content packs
Phase 1 of Creator Packs: the app-side download rail for per-creator
video ZIMs served from the private R2 bucket behind the entitlement
Worker. No UI yet (Phase 2).
- Thread optional requestHeaders through the download util + job so the
ZIM fetch can carry `Authorization: Bearer <APP_KEY>` on both the HEAD
probe and the GET stream (Worker streams the ZIM directly, gated).
- Add resourceMetadata.skip_embedding and gate the RAG/EmbedFileJob
branch on it so video ZIMs are never sent to the knowledge base.
- New creator_packs ManifestType + creatorPacksSpecSchema (display
metadata only, no url) + CollectionManifestService wiring and
getCreatorPacksWithStatus() status join.
- collections/creator-packs.json catalog with the two seed packs.
- CreatorPackService.installPack: ensure Kiwix (auto-install if absent),
resolve the stable Worker URL, dispatch the gated download with
skip_embedding. Typed result codes.
- CreatorPacksController + GET/POST /api/creator-packs routes.
- CREATOR_PACKS_APP_KEY / CREATOR_PACKS_WORKER_BASE env (release-injected
key; app degrades to a clear not_configured error when unset).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(creator-packs): Phase 2 UI — Content Explorer, settings page, wizard step
Adds the three user-facing surfaces for Creator Packs on top of the Phase 1
backend rail. All surfaces HIDE entirely on builds without the release-injected
key (configured=false) so a fork never shows a broken install button.
- Backend: CreatorPackService.isConfigured() + `configured` flag on the
GET /api/creator-packs response; /settings/creator-packs route + controller
action.
- api client + useCreatorPacks hook (shared cached source of configured/packs/
downloads for every surface) + CreatorPackCard (status badges, art fallback,
update pill) + CreatorPacksSection (install-on-click grid + confirm modal with
the license note), reused by the Content Explorer block and settings page.
- Content Explorer: Creator Packs section before Additional Content.
- Settings: new /settings/creator-packs manage page + conditional nav item.
- Easy Setup wizard: new Creator Packs step after Content. Replaces the fragile
hardcoded AI-skip math with a computed activeSteps list so the two optional
steps (Creator Packs + AI) navigate correctly in every on/off combination;
selection state, storage projection, review summary, and finish-install wired.
Browser-tested on NOMAD3 across all surfaces (configured + fork paths), incl.
live install→download→cancel and AI-on/off wizard renumbering.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(creator-packs): use branded pack banners on the cards
Replace the generic colored-block cards with the branded 1060x175 banners we
build into each pack ZIM, shown on both the settings page and the Easy Setup
wizard step (and the Content Explorer block, via the shared card).
- Bundle the seed-pack banners as local webp under admin/public/creator-packs/
so they render offline (no external image dependency, matching the existing
/rogue-support-banner.webp convention).
- CreatorPackCard is now banner-forward: the banner is the hero, with a compact
status/metadata footer (videos · size, Installed/Downloading/Selected/Install,
update pill). Falls back to a simple header if an image is ever missing.
- Prefer an optional catalog `banner_url` (future remote creators) over the
bundled-by-id path; add banner_url to the CreatorPack type + validator.
- Widen the card grids to 1–2 columns since banners are wide.
Browser-tested on NOMAD3: banners render on the settings page and wizard step;
selection/installed/downloading states and storage projection still correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(creator-packs): Phase 3 — uninstall, license, release key-injection
Completes the Creator Packs feature.
- Uninstall: CreatorPackService.uninstallPack reuses ZimService.delete (removes
the ZIM, drops it from the Kiwix library, clears the InstalledResource) behind
DELETE /api/creator-packs/:id. A trash control on installed cards appears only
on the settings "manage" surface (allowUninstall) — never on the Content
Explorer block or the wizard — with a danger confirm modal.
- License: draft "Project NOMAD Creator Pack License v1.0" at
collections/creator-pack-license.md (marked DRAFT — pending legal review;
personal-use, no-redistribution, official-channel-only, creators retain
copyright). The install modal links to it ("View license").
- Release key-injection: Dockerfile ARG/ENV CREATOR_PACKS_APP_KEY in the runtime
stage (empty default → source/CI builds ship unconfigured and hide the UI) +
build-primary-image.yml passes it from the CREATOR_PACKS_APP_KEY CI secret.
NOTE FOR JAKE: add that repo secret = the entitlement Worker's key.
- Update flow was already functional (installed-with-update cards → "Update
pack"); verified.
Browser-tested on NOMAD3: uninstall round-trip (file + DB row + kiwix library
cleaned, then reinstalled), settings-only uninstall control, install-modal
license link, and the "Update available" pill via a temporary catalog bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(downloads): clean up header merge logic
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: jakeaturner <jturner@cosmistack.com>
* feat(rag): add subject/collection organization to knowledge base
- Add nullable collection field to KbIngestState, propagated through the
embed job, RAG service, and Qdrant point payloads (indexed for filtering)
- Add upload-time category selection and per-file collection reassignment
in the Knowledge Base modal, with a filterable Stored Files table
- Add a 'Search in' collection filter to the chat interface, threaded
through to searchSimilarDocuments as an optional Qdrant filter
- Fix .docx extraction: previously routed through raw-text extraction
(garbage output for a ZIP-based XML format); adds a proper mammoth-based
extractor and a dedicated 'docx' file-type case
* feat(rag): support dynamic KB collection creation, rename, and removal
Extends collection organization with a Manage Collections UI: collections
are created on the fly when a file is assigned to a new name, can be
renamed (bulk-updates every tagged file and Qdrant point), and can be
removed (reassigns tagged files back to Uncategorized rather than
deleting anything).
* fix(rag): use dynamic collections query in chat search filter
chat/index.tsx still imported the static KB_COLLECTIONS constant for its
'Search in' dropdown, inconsistent with KnowledgeBaseModal.tsx which already
uses the live getKnowledgeCollections() query. Renamed/added collections
via the new Manage Collections UI weren't reflected in the chat filter.
* feat(rag): broaden preset tags and add creatable collection combobox
Replaces the survival-specific preset list with general-purpose starter
tags (recipes, diy, health, technology, finance, travel, hobbies,
reference, survival, energy) so the Knowledge Base reads well for
home-lab/reference use, not just prepping.
Adds sanitizeCollectionName() (trim, lowercase, length cap) applied on
every write path server-side, and a dependency-free CollectionCombobox
component replacing the plain <select> + window.prompt pattern for
tagging — autocompletes against presets + tags already in use, with a
'+ Create' option for anything new.
* chore(rag): remove .docx fix from this branch, split into #1100
Per review feedback, the .docx extraction fix is unrelated to the
collections feature and can merge independently. Moved to a standalone
PR (Crosstalk-Solutions/project-nomad#1100) off dev.
* chore: remove unrelated diff noise (lockfile, comments, indentation)
---------
Co-authored-by: John Cortright <jcortright@zscaler.com>
Dense source content produces chunks that exceed the embedding model's
context window (nomic-embed-text:v1.5 defaults to 2048 tokens). Two paths
hit this even after the prior pre-cap:
- Older Ollama (e.g. 0.18.1, #944) ignores the num_ctx=8192 we send on
/api/embed, so it stays at the model's 2048 default.
- The OpenAI-compat /v1/embeddings fallback didn't pass num_ctx/truncate
at all, so any Ollama drops to 2048 whenever it lands on the fallback.
When a chunk overflowed, the 400 was swallowed and the chunk was silently
dropped from Qdrant. Worse, the failure propagated to EmbedFileJob, which
re-embeds the entire file on each of its 30 BullMQ attempts — the "endless
queue loop" / "api/embed for weeks" / pegged GPU reported in #944/#959.
Fix:
- OllamaService.embed(): on a context-length error, retry once with an
aggressive 2048-safe cap (EMBED_CONTEXT_SAFE_CHARS = 2000) so the chunk
is embedded (start-of-chunk) instead of dropped. Native-path context
errors now bubble to this retry instead of falling through to the
smaller-context fallback. Split the native+fallback attempt into
_embedWithFallback().
- Pass truncate/num_ctx on the /v1/embeddings fallback too (Ollama's
OpenAI-compat shim forwards them).
- EmbedFileJob: classify "input length exceeds context length" as an
UnrecoverableError so one permanently-oversized chunk can't trigger 30
full-file re-embeds.
- Add OllamaService.isContextLengthError() shared by both.
Graceful degradation: a truncated chunk loses its tail but is kept in the
index, which is strictly better than today's silent drop + retry storm.
Refs #881. Supersedes the #369/#670 symptom closures that never fixed the
fallback path.
On ZIMs that pack one logical article as several sub-pages (e.g. iFixit),
iterByPath yields more entries passing our isArticleEntry() filter than
archive.articleCount reports. The inter-batch progress used nextOffset /
articleCount, so the numerator outran the denominator, the ratio overflowed past
100%, and the UI (which clamps at 99%) pinned the file at 99% for the entire
remaining tail, making it look hung.
Grow the denominator to max(articleCount, nextOffset + ZIM_BATCH_SIZE) once we
pass the reported article count, so progress keeps creeping forward monotonically
instead of freezing, and clamp to 99% so only the genuinely-final batch reports
100%.
This is a graceful heuristic, not exact progress (true accuracy would require a
pre-scan to count isArticleEntry matches up front); it removes the user-visible
"stuck at 99%" symptom with no change to batch semantics.
Closes#903
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a self-update path for the NOMAD admin/core image that runs without a
human in the loop, gated by layered safety checks. Recreation remains the
sidecar's job; this only decides *whether* to update now and requests it.
An hourly job (AutoUpdateJob) evaluates a side-effect-free decision pipeline
(AutoUpdateService.evaluate):
- opt-in: disabled by default
- in a user-configured local-time window (handles midnight wrap)
- an eligible release: same major (major bumps stay manual), strictly newer,
past a configurable cool-off, GA only (no drafts/prereleases), strict semver
- pre-flight: sidecar present, no in-flight system update / downloads / app
installs, and sufficient host disk (estimated from the registry manifest)
When all pass, it drives SystemUpdateService.requestUpdate() with a vetted tag.
Failure backoff auto-disables after 3 genuine update-request failures; transient
release-lookup errors are skips (offline-first appliances are routinely without
connectivity). Re-enabling clears the backoff state.
Settings UI exposes the toggle, window, and cool-off, with live status (eligible
target, in/out of window, last result/error). A `node ace auto-update:dry-run`
command exercises the full pipeline — and a deterministic --scenarios suite the
pure decision logic — without ever triggering an update.
New KVStore keys under autoUpdate.* hold config + last-run state.
The continuation dispatch in EmbedFileJob did not pass the running chunk
count forward, so each batch started with job.data.chunks undefined.
On the final batch, totalChunks collapsed to just that batch's result
and KbIngestState.markIndexed stored a value far below what Qdrant
actually held.
Add chunksSoFar to EmbedFileJobParams and thread it through the
continuation chain so the final markIndexed call reflects the true
total across all batches.
Closes#933
RunDownloadJob's onComplete handler was unconditionally firing
EmbedFileJob.dispatch after every ZIM download, gated only by "is
Ollama installed?". The rag.defaultIngestPolicy KV setting was never
consulted, so users who explicitly set Auto-index to Manual still got
every newly-downloaded ZIM auto-embedded.
RagService.scanAndSync already handles Manual correctly by recording
pending_decision rows instead of dispatching (rag_service.ts:1587-1638
via decideScanAction). The post-download path skipped that gate.
Mirror the same check at the dispatch site: read the policy KV; if
Manual, firstOrCreate a pending_decision row in kb_ingest_state so the
per-file Index affordance from PR #909 surfaces the file the same way
scan-time-discovered Manual files do. firstOrCreate (not create) so a
re-download doesn't demote an existing indexed/failed row — the user
can explicitly re-index from the KB panel if they want fresh content.
Verified on NOMAD3: with rag.defaultIngestPolicy='Manual', every ZIM
downloaded today via Content Explorer (agriculture-essential +
computing-essential, ~62 MB across 7 files) wrote kb_ingest_state
rows with state='indexed' instead of pending_decision. Real bug,
not a hot-patch artifact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each in-flight (or stuck) embedding job gets a colored health pill,
relative-activity timestamp, and chunk counter so users can tell at a
glance whether ingestion is making progress.
## Health states
- **🟢 Active** — last batch < 2 min ago
- **🟡 Slow** — last batch 2-5 min ago (CPU-paced multi-batch ingestion
lives here naturally; not always a problem)
- **🔴 Stalled** — last batch > 5 min ago (likely real problem)
- **⚪ Waiting** — queued, no batch started yet
- **🔴 Failed** — job recorded failed status
## What lands
- New backend util `kb_job_health.ts` with pure `computeJobHealth(input)`
decision function. Time-based thresholds (2 min / 5 min) inlined as
constants. 9 unit tests pin the boundaries.
- `EmbedJobWithProgress` gains `lastBatchAt`, `startedAt`, `chunks` —
already set by `EmbedFileJob.handle` on every batch transition, just
not previously surfaced through `listActiveJobs`.
- Frontend `kb_job_health_display.ts` maps each status to a Tailwind
dot color, label, and aria-label so backend and UI stay in sync.
- `ActiveEmbedJobs.tsx` renders the pill, "last activity Xs ago", and
chunk counter above each progress bar. Adds a manual Refresh button
and "Last updated Xs ago" line — the existing 2s/30s auto-poll
cadence in `useEmbedJobs` is left intact.
- Live tick at 5s keeps the relative timestamps current without
re-fetching from the API.
## Not in scope
- Per-card Cancel / Retry / Un-index — separate Phase 2 PR
- Conditional warnings A/B/C — separate Phase 2 PR
- Computing throughput rate (chunks/min) — needs ratio registry consumer
(Phase 2 follow-up); for now the pill answers the "is it stuck?"
question directly without a rate estimate.
Each continuation batch of a multi-batch ZIM embed runs as a fresh
BullMQ job, so handle() ran the hardcoded `safeUpdateProgress(job, 5)`
even when the file was already 100k articles into a 600k-article ZIM.
The UI gauge briefly dropped to 5% before the per-batch onProgress
callback caught up to the true overall percentage, reading as a
backward jump every time a new batch started.
Compute initialPercent from batchOffset / totalArticles when available,
falling back to 5 for single-batch files (uploaded PDFs, txts) where
totalArticles isn't set. Capped at 99 to leave headroom for the 100%
final-batch marker.
Follow-up to PR #880 (which fixed the 0-100% scaling during a batch
but still had the initial-frame regression).
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.
## What lands
- New table `kb_ingest_state` keyed by `file_path` with enum state column
(`pending_decision | indexed | browse_only | failed | stalled`).
Independent of `installed_resources` so it covers both curated downloads
and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
Registered now but not consumed yet — JIT prompt + wizard step land in
Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
- Success (final batch) → `indexed` + chunks count
- `UnrecoverableError` → `failed` + error message
- Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
decision tree (see `decideScanAction`). Existing installs auto-backfill
on first scan: files with chunks in Qdrant but no state row become
`indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
disappear entirely instead of leaving an orphan that the next scan
would re-dispatch into nothing.
## What does NOT land here
- Ratio registry (separate PR) — needed for partial-stall detection and
cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
/ Reset & Rebuild would also want to set state, but #886 isn't merged
yet; that wiring goes in a follow-up once #886 lands.
## Target
This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.
## Tests
- 9 new unit tests for `decideScanAction` covering all five states plus
the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
- Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
came back `indexed` on first scan
- Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
came back `pending_decision` and was correctly re-dispatched (Bull
deduped to its historical `:completed` jobId — bgauger's #886 fix
drains that)
- Delete hook: deleting a KB upload via `DELETE /api/rag/files`
removed both the disk file and the state row
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
Before this change, the Active Downloads / Processing Queue UI showed the
ingestion progress gauge jumping wildly during multi-batch ZIM ingestion
(e.g. 5% → 88% → 27% → 5% → 56% → 36% over ~60 seconds for cooking SE).
Each continuation batch is a separate BullMQ job, and `EmbedFileJob.handle()`
reported `job.progress` in two different reference frames depending on
where it was in the batch lifecycle:
- During-batch (via the onProgress callback): 5% → 95% scaled across
"% through this batch's chunks"
- End-of-batch (just before dispatching the next): overwritten to
`(nextOffset / totalArticles) * 100` — % through the whole file
- Next continuation batch starts with progress = 5% explicitly, then
climbs through the per-batch range again
`listActiveJobs()` returns the latest active BullMQ job's progress. With
GPU-accelerated ingestion completing a batch every ~4 seconds, the UI
saw the jobId rotate constantly and the gauge whipsaw between the two
reference frames.
`totalArticles` was already wired through the EmbedFileJob params shape
and used end-of-batch — but RagService never actually populated it,
so any frame-scaling that depended on it silently fell back to the
per-batch range. Two fixes together:
1. `ZIMExtractionService.extractZIMContent()` now returns
`{ chunks: ZIMContentChunk[]; totalArticles: number }` instead of a
raw chunks array, surfacing `archive.articleCount` to the caller.
Single caller (rag_service) updated to destructure.
2. `RagService.processZimFile()` includes `totalArticles` in its result
so `EmbedFileJob.dispatch()` can propagate it to the continuation
batch (which the existing code already does via
`totalArticles: totalArticles || result.totalArticles`).
3. `EmbedFileJob`'s onProgress callback scales the service-reported
per-batch percent into the overall-file frame when `totalArticles`
is known: `((batchOffset + (percent/100) * ZIM_BATCH_SIZE) /
totalArticles) * 100`. Capped at 99% to leave room for the explicit
100% set at file completion. Falls back to the original 5-95% range
for single-batch files (uploaded PDFs/txts) where totalArticles is
undefined — the gauge then represents % through the only batch,
which is what the UI expects for one-shot files.
Validated on NOMAD8 (RX 6800, ROCm-accelerated nomic):
- devdocs python (small, ~1500 articles): batch progressions seen
monotonically across continuation jobIds:
1501@30% → 1510@33% → 1514@43% → 1518@52%.
- ifixit (huge, ~100k articles): stays near 3% for the first many
batches at offset 0..3000 — correct, the file is enormous.
- wikipedia_en_medicine (large, ~70k articles): stays near 0-1% for
the first batches — also correct.
- Brief 0-5% blip on continuation handoff (the explicit
`safeUpdateProgress(job, 5)` at batch start, before the first
onProgress callback fires) — visible but quickly resolves to the
overall-frame value. No more 5% ↔ 88% chaos.
Stacks on top of the multi-batch ZIM ingestion fix. After that fix,
multi-batch ZIM ingestion completes correctly — but on installs where
Ollama runs the embedding model on CPU (currently every AMD ROCm
install, since Ollama's ROCm build doesn't accelerate nomic-bert),
the now-correct sustained 100% CPU saturation across all cores can
starve other services hard enough to take the box down. Confirmed
on a Threadripper 3960X + RX 6800 NOMAD: a wikipedia-class ZIM
ingestion pegged 48 threads cleanly enough that sshd lost
banner-exchange responsiveness and the box ultimately required a
power-cycle.
NVIDIA installs aren't affected — nomic-embed-text:v1.5 runs at
100% GPU on RTX 5060 (verified via `ollama ps`).
Detect placement at runtime, pace only when needed:
1. OllamaService.isEmbeddingGpuAccelerated() — queries /api/ps and
returns true if any loaded embedding model reports size_vram > 0.
Fails closed (returns false) if /api/ps is unreachable or no embed
model is loaded yet — over-pacing is safer than crashing.
2. EmbedFileJob.handle() — between batches (hasMoreBatches: true
branch), check placement and `await setTimeout(CPU_BATCH_DELAY_MS)`
when CPU-only. CPU_BATCH_DELAY_MS = 1000 (1s) — enough to give the
OS scheduler a window for sshd/disk-collector/etc., small enough
that total ingestion time isn't meaningfully affected (each batch
is ~60-90s of work).
GPU-accelerated installs see zero behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every static call site instantiated a fresh QueueService (24 call sites
across 8 files). QueueService.getQueue() opens a BullMQ Queue per call
when not cached, and each Queue opens two ioredis connections (one for
commands, one blocking). Because every static call constructed a new
QueueService, its internal `queues` cache was never shared, every call
opened a fresh pair, and none were ever closed.
In normal operation this leaked a few connections per API hit. During
multi-batch ZIM ingestion after PR #872 (where EmbedFileJob.handle()
dispatches the next batch every 50 articles), every batch completion
opened two new connections. On NOMAD3 at ~one batch every 4s sustained,
that's ~1800 leaked connections/hour. Redis hit its 10,000-maxclient
ceiling in ~5 hours and the admin container fell into an EPIPE flood
that required a restart to recover.
Fix: collapse QueueService to a true process-wide singleton with a
private constructor and getInstance() accessor. The existing per-queue
Map is now shared across every dispatch / status / cleanup call, so each
queue's underlying connections are opened exactly once for the lifetime
of the process. close() now clears the map so the singleton can be torn
down cleanly if a graceful-shutdown hook is ever wired up.
Validated on NOMAD3 (RTX 5060, v1.32.0-rc.4 + this patch hot-applied):
under sustained multi-batch wikipedia_en_simple_all_nopic ingestion,
connected_clients held flat at 21-22 across a 5-minute window. Pre-fix
the same scenario climbed to 10,000+ over hours.
EmbedFileJob.dispatch() uses a deterministic per-file jobId
(sha256(filePath).slice(0,16)) for every batch. The parent batch's
handle() calls EmbedFileJob.dispatch({ batchOffset }) before returning,
so the parent is still in `active` state and locked when the
continuation tries to enqueue. BullMQ silently returns the locked
parent instead of creating a new job — and in newer BullMQ versions
it does so without throwing, so the existing
`catch (error.message.includes('job already exists'))` branch never
fires. After the parent completes, its entry stays in the `completed`
ZSET (held by `removeOnComplete: { count: 50 }`), continuing to trip
jobId dedupe for any subsequent re-dispatch attempts.
Result: every NOMAD install since 2026-02-08 (feat: zim content
embedding) with a multi-batch ZIM (wikipedia, cooking SE, ifixit,
lrnselfreliance, etc.) has only the first 50 articles indexed in
qdrant. The RAG feature has been silently degraded for ~3 months —
the user sees the file appear in their KB, qdrant accumulates ~50
articles' worth of vectors, and pagination quietly halts. No error
surfaces anywhere.
Fix: dispatch() skips the deterministic jobId for continuation batches
(batchOffset > 0), letting BullMQ auto-generate a unique one so each
batch stacks as an independent queue entry. Initial dispatches keep
the deterministic jobId so re-triggering an install (UI re-click,
sync rescan) remains idempotent. The existing 'job already exists'
branch is now gated on !isContinuation, since by construction
continuation batches will never hit dedupe.
Validated on NOMAD8 (RX 6800 / Threadripper 3960X, rc.3 + this patch):
devdocs_en_python (~1,500 chunks across multiple batches) correctly
paginates end-to-end. admin.log shows the expected sequence of
"Dispatched embedding job for file: X (continuation @ offset N)"
followed by "Starting embedding process for: X (batch offset: N)"
for each batch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(maps): add regional map downloads via go-pmtiles extract
* address Copilot review feedback on PR #780
- auto-refresh preflight on selection/maxzoom change with 400ms debounce and
requestId stale-safety so the confirm button no longer requires a two-step
"Estimate Size" -> "Start Download" dance
- safeUpdateProgress helper replaces fire-and-forget updateProgress().catch()
pattern so cancelled-job errors (code -1) can't surface as unhandled rejections
- gate world basemap source on worldBasemapReady - when ensureWorldBasemap()
fails we already delete world.pmtiles, so emitting the source was producing
404s on every tile request
- verify go-pmtiles binary SHA256 at image build time; upstream doesn't ship a
checksums file so per-arch hashes are pinned as build args with a regenerate
note when bumping PMTILES_VERSION
Adds a cancel button to in-progress Ollama model downloads and unifies
the Active Model Downloads card layout with the Active Downloads card
used for ZIMs, maps, and pmtiles (byte counts, progress bar, live speed,
status indicator).
Closes#676.
* fix(downloads): increase retry attempts and backoff for large file downloads
* fix download retry config and abort handling
* use abort reason to detect user-initiated cancels
* feat(downloads): rich progress, friendly names, cancel, and live status
Redesign the Active Downloads UI with four improvements:
- Rich progress: BullMQ jobs now report downloadedBytes/totalBytes instead
of just a percentage, showing "2.3 GB / 5.1 GB" instead of "78% / 100%"
- Friendly names: dispatch title metadata from curated categories, Content
Explorer library, Wikipedia selector, and map collections
- Cancel button: Redis-based cross-process abort signal lets users cancel
active downloads with file cleanup. Confirmation step prevents accidents.
- Live status indicator: green pulsing dot with transfer speed for active
downloads, orange stall warning after 60s of no data, gray dot for queued
Backward compatible with in-flight jobs that have integer-only progress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(downloads): fix cancel, dismiss, speed, and retry bugs
- Speed indicator: only set prevBytesRef on first observation to prevent
intermediate re-renders from inflating the calculated speed
- Cancel: throw UnrecoverableError on abort to prevent BullMQ retries
- Dismiss: remove stale BullMQ lock before job.remove() so cancelled
jobs can actually be dismissed
- Retry: add getActiveByUrl() helper that checks job state before
blocking re-download, auto-cleans terminal jobs
- Wikipedia: reset selection status to failed on cancel so the
"downloading" state doesn't persist
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(downloads): improve cancellation logic and surface true BullMQ job states
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Turner <jturner@cosmistack.com>
Model downloads that fail (e.g., when Ollama is too old for a model)
were silently retrying 40 times with no UI feedback. Now errors are
broadcast via SSE and shown in the Active Model Downloads section.
Version mismatch errors use UnrecoverableError to fail immediately
instead of retrying. Stale failed jobs are cleared on retry so users
aren't permanently blocked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Ollama isn't installed, every ZIM download dispatches embedding jobs
that fail and retry 30x with 60s backoff. With many ZIM files downloading
in parallel, this exhausts Redis connections with EPIPE/ECONNRESET errors.
Two changes:
1. Don't dispatch embedding jobs when Ollama isn't installed (belt)
2. Use BullMQ UnrecoverableError for "not installed" so jobs fail
immediately without retrying (suspenders)
Closes#351
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add comprehensive benchmarking capability to measure server performance:
Backend:
- BenchmarkService with CPU, memory, disk, and AI benchmarks using sysbench
- Database migrations for benchmark_results and benchmark_settings tables
- REST API endpoints for running benchmarks and retrieving results
- CLI commands: benchmark:run, benchmark:results, benchmark:submit
- BullMQ job for async benchmark execution with SSE progress updates
- Synchronous mode option (?sync=true) for simpler local dev setup
Frontend:
- Benchmark settings page with circular gauges for scores
- NOMAD Score display with weighted composite calculation
- System Performance section (CPU, Memory, Disk Read/Write)
- AI Performance section (tokens/sec, time to first token)
- Hardware Information display
- Expandable Benchmark Details section
- Progress simulation during sync benchmark execution
Easy Setup Integration:
- Added System Benchmark to Additional Tools section
- Built-in capability pattern for non-Docker features
- Click-to-navigate behavior for built-in tools
Fixes:
- Docker log multiplexing issue (Tty: true) for proper output parsing
- Consolidated disk benchmarks into single container execution
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
caweis
chriscrosstalk
just-jbc
Jake Turner
jakeaturner
Chris Sherwood
Metbcy
0xGlitch
Henry Estela