* feat(drug-reference): compact header + single dashboard tile
Phase 1 of the drug-reference redesign:
- AppLayout gains an opt-in `compact` prop (small inline logo+title) so tool
pages reclaim the ~230px the full branding block costs; drug-reference/index
opts in.
- Consolidate the two dashboard tiles (Drug Reference + When to use what) into a
single Drug Reference tile with a broadened description (/conditions already
redirects to /drug-reference).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(drug-reference): tabbed redesign — search-by-drug + by-situation + FDA data
- Split /drug-reference into three tabs (Headless UI TabGroup): 'Search by
drug', 'By situation', 'FDA data'. Each tab runs only its own direction,
which removes the two-overlapping-sections confusion.
- Search by drug: results grouped by active ingredient (IngredientGroup,
single-ingredient groups first, combos after), drug-first result rows, and a
collapsible de-jargoned filter drawer (Over-the-counter / Prescription, Form,
Sort) that auto-collapses once results land.
- By situation: multi-select symptom chips → an 'Treats all N selected'
intersection section pinned on top (computed client-side from each
situation's result set) + one union group per situation.
- FDA data: the download/ingest control + status moved behind its own tab
(the pre-ingest empty state stays the prominent download prompt).
- DrugResultRow now leads with the active ingredient (drug-first) by default,
or the brand when rendered inside an ingredient group.
- Rename 'Compare interactions' → 'Compare label warnings' to match what it does.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(drug-reference): rank situation matches single-ingredient-first
The raw FDA indication match floods a situation (e.g. Headache) with
many-ingredient homeopathic products, burying real OTC drugs and leaving the
cross-situation intersection empty. Pull a wider result set (200) and sort by
active-ingredient count ascending — not a medical judgement, the same
'single-ingredient first' principle as the drug-search grouping. Now real
drugs (acetaminophen, ibuprofen) surface on top and the 'Treats all N selected'
intersection actually finds the shared OTC options. Per-situation cards cap the
display to the top 25 (ranked), intersection uses the full set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(drug-reference): first-open disclaimer gate + first-group-expanded
- Add a required disclaimer modal on first open of the Drug Reference (Jake's
mechanism): comprehensive not-medical-advice notice the user must acknowledge;
acceptance is saved to the browser's localStorage (versioned key) so it isn't
shown again on that browser, while new browsers/devices see it on first open.
Non-dismissible (no backdrop/Escape) — only the acknowledge button closes it.
- Search-by-drug: expand the first ingredient group by default, collapse all
subsequent groups (IngredientGroup gains an explicit defaultOpen prop,
replacing the size heuristic).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(drug-reference): intersection-first multi-select
Multiple situations now lead with the 'Treats all N selected' intersection and
only break out per-situation sections when the intersection is empty (nothing
treats all) — with a 'No single option treats all N of these' explainer. Keeps
the view combined when there's a shared answer, and only fragments as a fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(drug-reference): compact header on detail/interactions/conditions pages
Apply the compact AppLayout header to the drug detail, interactions, and
condition pages so they match the redesigned index instead of the full-height
branding block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(drug-reference): rename interactions heading to 'Compare label warnings'
Match the page heading + title to the button label, so the name reflects what
the view does (each drug's own FDA-labeled warnings side by side, not a
cross-drug interaction checker).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(drug-reference): offline FDA drug labels, conditions, and remedies
Adds an offline medical-reference feature with three coupled layers:
- Drug Reference: full-text search over openFDA drug-label indications,
a detail page per label, and a side-by-side single-drug comparison view.
A two-phase background pipeline downloads the openFDA label parts to the
storage volume (resumable) and ingests them into the search table.
- Conditions ("When to use what"): a curated spine of first-aid situations
that maps each situation to matching OTC drugs, each linking back to its
Drug Reference detail page.
- Curated remedies: hand-authored natural and home-remedy entries drawn from
US-government public-domain sources (NCCIH, CDC, MedlinePlus, FDA), shown
with their source links and the same safety disclaimers as the rest of the
feature.
The drug-reference and conditions layers are intentionally coupled: the drug
detail page shows the situations a drug treats, and the conditions controller
reads the same drug_labels table.
Safety surfaces ship as written. The amber SafetyBanner ("informational only,
not medical advice, not an FDA endorsement, not a drug-interaction checker, in
an emergency call emergency services") renders on the condition pages and the
search page; the detail and comparison pages carry their own "not a cross-drug
interaction checker" callout; and every page carries the openFDA CC0 source
citation and no-FDA-affiliation footer.
Wiring on this branch:
- start/routes.ts: the /drug-reference and /conditions page GETs plus their
/api/* groups.
- commands/queue/work.ts: the drug-download and drug-ingest queues, both at
concurrency 1. The two drug queues get a per-queue stall override
(lockDuration 1_800_000, maxStalledCount 3) because each part is one long
stream; every other queue keeps the existing 300000 default.
- inertia/pages/home.tsx: Drug Reference and "When to use what" tiles. The
icon and display_order are a starting point, open to change.
- types/kv_store.ts: the two drugReference.* keys the pipeline reads and writes.
- package.json: yauzl and stream-json (plus their @types), used by the ingest
job to stream the label JSON out of the downloaded zips.
The app reads the conditions and remedy data from the compiled TS constants in
app/data/; the repo-root collections/*.json files are the browseable mirrors.
The natural-remedies standalone test reads collections/natural_remedies.json to
assert the two stay in sync, so that file is also a test fixture.
The four standalone tests pass (drug_interactions, drug_ingest_status,
conditions, natural_remedies). tsc reports no errors in the feature code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(drug-reference): drop a fork-internal comment reference
* fix(drug-reference): address review — defer FULLTEXT, fallback remedies prop, strip fork refs
- migration: wrap the FULLTEXT ALTER in this.defer so it runs after the
deferred createTable (was silently swallowed, index never created)
- controller: add remedies:[] to the index() error fallback (required prop)
- strip fork-internal issue/spec references from ported comments
- tsconfig: exclude tests/standalone (node --experimental-strip-types only)
- correct the varchar(768) byte-math comment; extend remedy-spine test
* fix(drug-reference): make the interaction comparison readable at five drugs
The comparison view laid its columns out on an equal-fraction CSS grid
(repeat(N, minmax(0, 1fr))), so each added drug shrank every column; at the
five-drug maximum the FDA interaction text was squeezed into unreadable slivers.
Lay the columns out with flex instead: full-width and stacked on phones, then
fixed-width columns that scroll sideways from the sm: breakpoint up, so they
never shrink below a readable width. Theme the columns with the same palette as
the rest of the page (they were on stock gray), and give the headers a fixed
min-height so columns line up when drug names wrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(collections): route a 'dataset' tier resource to the drug pipeline
Add an optional `type` discriminator to SpecResource ('zim' | 'dataset',
absent == 'zim'), so the tier installer can carry a DB-ingested resource
alongside ZIM files. ZimService.downloadCategoryTier branches on it: a
'dataset' resource dispatches the existing FDA download+ingest pipeline
instead of RunDownloadJob, guarded against duplicate dispatch by the drug
ingest status. Every existing manifest entry has no `type` and keeps the
exact ZIM path.
Widen InstalledResource.resource_type to include 'dataset' and exclude
dataset rows from the ZIM/map catalog-update scan (datasets aren't
filename-versioned; their freshness path is separate). No dataset rows are
written yet: the InstalledResource 'dataset' row on ingest-ready, the
manifest entry, install-gating, and the downloads-aggregator integration
are follow-up commits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): rework into an opt-in medicine-standard tier dataset
Reshapes the offline FDA drug reference from a built-in feature into curated
content installed by selecting the Medicine / Standard tier, per maintainer
direction.
- install-state: the ingest writes an installed_resources 'dataset' row on
ready (version = the openFDA export_date), threaded installer to download to
ingest; the tier-status math and the home-tile gate read it. Manual ingests
write no row, so install-state stays tied to the curated path.
- manifest: declare the dataset in the medicine-standard tier (runtime fetches
the remote manifest, so this also needs to land upstream).
- install-gating: the drug-reference home tiles render only when installed.
- uninstall: DrugReferenceService.uninstall() stops the two drug queues, deletes
the on-disk parts, truncates drug_labels (schema kept), clears the KV markers,
and drops the install row. Best-effort, logged, scoped to drug data only.
- downloads: the download phase reports the canonical {percent, downloadedBytes,
totalBytes} shape as one drug-data card in the Active Downloads aggregator with
cancel/remove; the heavy ingest stays in the IngestStatus surface with an
Indexing handoff on the card.
- auto-update: a daily DrugAutoUpdateJob compares the manifest export_date and
re-downloads when newer, gated on installed + no active job.
typecheck clean; the drug standalone suites pass. Three points are flagged in
code for the maintainer: the InstalledResource 'dataset' approach, the tier
home, and the export_date string format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(downloads): keep the drug download card live across all parts
The Active Downloads card filtered to the deterministic jobId, but the
download's continuations run under auto-generated jobIds (only part 0 uses the
deterministic one). So the card tracked part 0 and then vanished while parts
2..N kept downloading. The queue is concurrency 1, so collapse to whichever
single part is in flight and report the deterministic jobId: one card tracks
aggregate progress through the whole download and cancel/remove still routes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): co-locate a persistent safety note with affirmative remedy guidance
Add RemedySafetyNote at the head of every natural-remedy section — the two on
Drug Reference and the one on "When to use what" — so the "informational only,
not medical advice, seek real medical care in an emergency" framing appears with
the guidance itself, not only in the page-top banner. Replaces the terse
per-section caveat with the same amber alert language as SafetyBanner.
Addresses the upstream #1040 review request that the disclaimer be unmistakable
and present wherever affirmative self-care guidance appears, not a one-time banner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(drug-reference): fold dataset freshness into the content-auto-update path
The drug dataset auto-updated on its own daily cron that ignored the
contentAutoUpdate.* master switch, so it would refresh even with content
auto-update turned off, and it didn't ride the content-update path the way
ZIMs and maps do.
Move the export_date freshness/apply orchestration onto
DrugReferenceService.attemptAutoUpdate(), add
ContentAutoUpdateService.attemptDrugDataset() gated on the same enabled +
window config, and have the hourly ContentAutoUpdateJob drive both. Retire
the standalone DrugAutoUpdateJob. The ZIM/map attempt() path is unchanged.
Addresses the upstream #1040 request to wire the openFDA export_date check
into the content updater so the dataset updates alongside ZIMs and maps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): strengthen the remedy note to consult a clinician before combining with meds
Widen the affirmative-remedy safety note from "talk to a clinician before use"
to explicitly cover using a remedy AND combining one with a medication the user
already takes — the interaction case is the higher-risk path for an off-grid
user self-treating.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): gate affirmative remedy content behind an off-by-default flag
Add drugReference.remediesEnabled (default off), independent of the tier
install. When off, the server emits no remedy data at any boundary — the
drug-reference page prop, conditions show, and the /api/conditions/drugs
situation search — and the "Natural" filter is hidden, so installing the
medicine-standard tier lights up the verbatim FDA label search and the
condition-to-OTC matching but not the hand-authored self-care and herbal
sections. No user-facing toggle: it is flipped on after a clinician content-pass.
Implements the upstream #1040 split-by-risk request: the regulated label content
ships with the tier; the authored remedy guidance stays gated until sign-off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
* feat(AI): per-model thinking toggle with global default (off)
Stop forcing thinking on for every capable model. Adds a global default
(ai.autoThinking, ships OFF) and a per-model override in the chat window,
shown only for thinking-capable models and remembered client-side.
The /v1 (OpenAI-compat) endpoint ignores `think`; reasoning_effort is the
real lever. The controller resolves per-request preference -> global default
-> OFF (gated on capability), and the service maps that to reasoning_effort
('none' to suppress on a capable model, 'medium' for gpt-oss, unset to let a
capable model default thinking on). A new thinkingCapable flag keeps
non-Ollama backends from ever receiving reasoning_effort.
installed-models is enriched with a `thinking` flag (checkModelHasThinking,
now memoized) so the picker knows which models get the toggle.
Stacks on #1078 (reasoning-field read + client-disconnect abort).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(AI): thinking toggle as NOMAD Switch + info tooltip
Address review feedback on the per-model thinking control:
- Use the shared Switch component (matches AI Assistant settings) instead
of a raw checkbox.
- Label "Thinking:" with a colon to match the adjacent "Model:" label.
- Add an InfoTooltip explaining what thinking does and where the default
lives. Extend InfoTooltip with optional `position` ('top'|'bottom') and
`align` ('center'|'right') so it opens downward and expands leftward from
the right-edge header slot instead of being clipped/crushed against the
viewport edge. Defaults preserve existing (benchmark page) behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(creator-packs): backend rail for gated creator content packs
Phase 1 of Creator Packs: the app-side download rail for per-creator
video ZIMs served from the private R2 bucket behind the entitlement
Worker. No UI yet (Phase 2).
- Thread optional requestHeaders through the download util + job so the
ZIM fetch can carry `Authorization: Bearer <APP_KEY>` on both the HEAD
probe and the GET stream (Worker streams the ZIM directly, gated).
- Add resourceMetadata.skip_embedding and gate the RAG/EmbedFileJob
branch on it so video ZIMs are never sent to the knowledge base.
- New creator_packs ManifestType + creatorPacksSpecSchema (display
metadata only, no url) + CollectionManifestService wiring and
getCreatorPacksWithStatus() status join.
- collections/creator-packs.json catalog with the two seed packs.
- CreatorPackService.installPack: ensure Kiwix (auto-install if absent),
resolve the stable Worker URL, dispatch the gated download with
skip_embedding. Typed result codes.
- CreatorPacksController + GET/POST /api/creator-packs routes.
- CREATOR_PACKS_APP_KEY / CREATOR_PACKS_WORKER_BASE env (release-injected
key; app degrades to a clear not_configured error when unset).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(creator-packs): Phase 2 UI — Content Explorer, settings page, wizard step
Adds the three user-facing surfaces for Creator Packs on top of the Phase 1
backend rail. All surfaces HIDE entirely on builds without the release-injected
key (configured=false) so a fork never shows a broken install button.
- Backend: CreatorPackService.isConfigured() + `configured` flag on the
GET /api/creator-packs response; /settings/creator-packs route + controller
action.
- api client + useCreatorPacks hook (shared cached source of configured/packs/
downloads for every surface) + CreatorPackCard (status badges, art fallback,
update pill) + CreatorPacksSection (install-on-click grid + confirm modal with
the license note), reused by the Content Explorer block and settings page.
- Content Explorer: Creator Packs section before Additional Content.
- Settings: new /settings/creator-packs manage page + conditional nav item.
- Easy Setup wizard: new Creator Packs step after Content. Replaces the fragile
hardcoded AI-skip math with a computed activeSteps list so the two optional
steps (Creator Packs + AI) navigate correctly in every on/off combination;
selection state, storage projection, review summary, and finish-install wired.
Browser-tested on NOMAD3 across all surfaces (configured + fork paths), incl.
live install→download→cancel and AI-on/off wizard renumbering.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(creator-packs): use branded pack banners on the cards
Replace the generic colored-block cards with the branded 1060x175 banners we
build into each pack ZIM, shown on both the settings page and the Easy Setup
wizard step (and the Content Explorer block, via the shared card).
- Bundle the seed-pack banners as local webp under admin/public/creator-packs/
so they render offline (no external image dependency, matching the existing
/rogue-support-banner.webp convention).
- CreatorPackCard is now banner-forward: the banner is the hero, with a compact
status/metadata footer (videos · size, Installed/Downloading/Selected/Install,
update pill). Falls back to a simple header if an image is ever missing.
- Prefer an optional catalog `banner_url` (future remote creators) over the
bundled-by-id path; add banner_url to the CreatorPack type + validator.
- Widen the card grids to 1–2 columns since banners are wide.
Browser-tested on NOMAD3: banners render on the settings page and wizard step;
selection/installed/downloading states and storage projection still correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(creator-packs): Phase 3 — uninstall, license, release key-injection
Completes the Creator Packs feature.
- Uninstall: CreatorPackService.uninstallPack reuses ZimService.delete (removes
the ZIM, drops it from the Kiwix library, clears the InstalledResource) behind
DELETE /api/creator-packs/:id. A trash control on installed cards appears only
on the settings "manage" surface (allowUninstall) — never on the Content
Explorer block or the wizard — with a danger confirm modal.
- License: draft "Project NOMAD Creator Pack License v1.0" at
collections/creator-pack-license.md (marked DRAFT — pending legal review;
personal-use, no-redistribution, official-channel-only, creators retain
copyright). The install modal links to it ("View license").
- Release key-injection: Dockerfile ARG/ENV CREATOR_PACKS_APP_KEY in the runtime
stage (empty default → source/CI builds ship unconfigured and hide the UI) +
build-primary-image.yml passes it from the CREATOR_PACKS_APP_KEY CI secret.
NOTE FOR JAKE: add that repo secret = the entitlement Worker's key.
- Update flow was already functional (installed-with-update cards → "Update
pack"); verified.
Browser-tested on NOMAD3: uninstall round-trip (file + DB row + kiwix library
cleaned, then reinstalled), settings-only uninstall control, install-modal
license link, and the "Update available" pill via a temporary catalog bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(downloads): clean up header merge logic
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: jakeaturner <jturner@cosmistack.com>
* feat(rag): add subject/collection organization to knowledge base
- Add nullable collection field to KbIngestState, propagated through the
embed job, RAG service, and Qdrant point payloads (indexed for filtering)
- Add upload-time category selection and per-file collection reassignment
in the Knowledge Base modal, with a filterable Stored Files table
- Add a 'Search in' collection filter to the chat interface, threaded
through to searchSimilarDocuments as an optional Qdrant filter
- Fix .docx extraction: previously routed through raw-text extraction
(garbage output for a ZIP-based XML format); adds a proper mammoth-based
extractor and a dedicated 'docx' file-type case
* feat(rag): support dynamic KB collection creation, rename, and removal
Extends collection organization with a Manage Collections UI: collections
are created on the fly when a file is assigned to a new name, can be
renamed (bulk-updates every tagged file and Qdrant point), and can be
removed (reassigns tagged files back to Uncategorized rather than
deleting anything).
* fix(rag): use dynamic collections query in chat search filter
chat/index.tsx still imported the static KB_COLLECTIONS constant for its
'Search in' dropdown, inconsistent with KnowledgeBaseModal.tsx which already
uses the live getKnowledgeCollections() query. Renamed/added collections
via the new Manage Collections UI weren't reflected in the chat filter.
* feat(rag): broaden preset tags and add creatable collection combobox
Replaces the survival-specific preset list with general-purpose starter
tags (recipes, diy, health, technology, finance, travel, hobbies,
reference, survival, energy) so the Knowledge Base reads well for
home-lab/reference use, not just prepping.
Adds sanitizeCollectionName() (trim, lowercase, length cap) applied on
every write path server-side, and a dependency-free CollectionCombobox
component replacing the plain <select> + window.prompt pattern for
tagging — autocompletes against presets + tags already in use, with a
'+ Create' option for anything new.
* chore(rag): remove .docx fix from this branch, split into #1100
Per review feedback, the .docx extraction fix is unrelated to the
collections feature and can merge independently. Moved to a standalone
PR (Crosstalk-Solutions/project-nomad#1100) off dev.
* chore: remove unrelated diff noise (lockfile, comments, indentation)
---------
Co-authored-by: John Cortright <jcortright@zscaler.com>
A System-Only or AI-Only run is a partial result, not the NOMAD Score
(which is the full-benchmark composite). Two problems addressed:
1. Scoring bug: AI-only runs were NOT renormalized. _calculateNomadScore
always added the system weights (0.60) to the denominator even for an
AI-only run (default-zero system scores), so an excellent AI-only run
scored ~39.8 -- scaled against the full NOMAD 100 where AI is only 40%
-- while system-only already renormalized correctly. Fix: pass
systemScores only when the system benchmarks actually ran, so AI-only
renormalizes to its own 0-100 (39.8 -> ~99.7). Full and System-only
scores are unchanged.
2. Presentation: partial runs were shown with the full "NOMAD Score"
label + big gauge, outweighing the small "Partial" notice. Now partial
runs are relabelled "System Score" / "AI Score" with a PARTIAL badge,
a muted (neutral) gauge + number, and a "run a Full Benchmark for your
NOMAD Score" CTA -- applied to both the persistent score section and
the Phase 3 ScoreReveal via a shared getScoreDisplay() helper. Adds a
`muted` prop to CircularGauge.
Browser-validated on NOMAD3: AI-only now shows "AI Score" + PARTIAL,
muted gauge, 99.7 (was 39.8); Full still shows "NOMAD Score" green.
Implements the display-layer fix from the Score v2 red-team's W2; the
AI-ceiling saturation (W1) remains v2 work.
Part of #1082.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 of the live benchmark run experience.
- ScoreReveal: replaces the abrupt run-view unmount with a deliberate
end-of-run "REPORT" card -- animated NOMAD score gauge + odometer
count-up number, sub-score gauges cascading in, Continue button +
5s auto-dismiss. Takes score scale as a prop so it survives Score v2.
- GPU-util overlay (NVIDIA): during the AI stage, a ~1Hz nvidia-smi poll
inside the Ollama container feeds live GPU utilization + VRAM into the
telemetry frames; shown in the AI hero, hidden when absent (AMD/none).
Poller is side-effect-only and cleared in a finally -- scored numbers
unchanged.
- Disk polish: reset in-test buffers on stage transition so the write
stage no longer briefly shows the carried disk-read value.
Browser-validated on NOMAD3 (RTX 5060): GPU overlay live (1% util,
0.2/8.0 GB VRAM during model load); reveal cascade + count-up; AI-only
score 39.8 (scored path unchanged).
Part of #1082 (tracker stays open).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 of the live benchmark run experience. Streams sysbench
--report-interval=1 interim lines over a Docker attach so the run view
shows authoritative in-test CPU events/sec and disk read/write MiB/s
(overlaying the Phase 1 host-proxy disk numbers), and fills a
"results so far" strip as each stage completes.
- _runSysbenchCommandStreaming: attaches to the container output for
live onLine callbacks, but returns the authoritative output via
container.logs() after exit -- byte-identical to _runSysbenchCommand,
so the SCORED numbers are unchanged (attach 'data' can flush after
container.wait() resolves, which would truncate the final report).
- CPU + disk stages stream interim eps: / reads:/writes: MiB/s into
setStageMetric; memory stays non-streaming (too fast to sample).
- _emitPartialResult broadcasts each finished stage's raw result on the
progress channel; useBenchmarkRun accumulates them; ResultsSoFar
renders the chip strip.
- Frontend: live CPU ev/s readout+sparkline, disk hero switches to
"Benchmark throughput" when in-test numbers arrive.
Browser-validated on NOMAD3 (System-Only): CPU 6536 ev/s live, disk
10399 MB/s benchmark throughput, results strip, final score 68.3.
Part of #1082 (tracker stays open for Phase 3).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 of the live benchmark-run experience. Replaces the opaque (and
in sync mode, simulated) progress bar with a real-time run view driven by
actual host telemetry.
- Async run path: the UI now dispatches to the queue worker and keys off
SSE instead of faking stage progress with client-side timers.
- New BenchmarkTelemetrySampler broadcasts per-core CPU load, CPU temp
(best-effort, hidden when unavailable), and disk MB/s at 1 Hz over a new
benchmark-telemetry SSE channel. Runs in the orchestration process, never
the sysbench container, so it cannot affect scores.
- BenchmarkProgress carries the ordered stage plan + index so the frontend
renders a live stage rail.
- AI benchmark streams /api/generate for live tokens/sec and true TTFT; the
scored numbers still come from Ollama's authoritative final eval fields.
- Frontend: useBenchmarkRun hook owns both subscriptions; self-contained SVG
components (StageRail, CoreGrid, Sparkline, LiveReadout) + BenchmarkRunView,
styled in the desert palette. No chart library added.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface post-upgrade release highlights on the Command Center: a
dismissable banner keyed to the running build's release line (major.minor).
It appears when a user upgrades into the line and every patch within it,
then disappears on the next minor. Dismissal is remembered per-line in
localStorage, so dismissing v1.34 won't suppress a future v1.35 note.
Reuses the existing dismissable Alert component and the dashboard's top
banner slot; reads the running version from the appVersion shared prop.
No schema or migration needed — to surface a new release's highlights,
bump WHATS_NEW.version and replace its highlights.
Initial content: Creator Packs and the offline Medication Reference.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wordmark application filed (USPTO serial 99912179). Mark the
persistent Command Center footer wordmark (both the AppLayout footer
and the settings/docs sidebar footer) and the first mention on the
Legal Notices page. Browser-tab titles and body copy left unmarked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Retire the dotted "N.O.M.A.D." styling everywhere in favor of
"Project NOMAD" (no periods) ahead of the trademark filing, and
remove the "Node for Offline Media, Archives, and Data" backronym
from all copy except a single origin-story line in the About page
(corrected to "Maps" rather than "Media").
Scope is display strings only: docs, admin UI labels/titles, install
script output, Dockerfile labels, package.json description, and issue
templates. Code identifiers, container/service names, env vars, CSS
tokens, URLs, and the project-nomad slug are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously, failed downloads showed only an alert icon and a dismiss (X)
button — the user had no way to retry or reach the resource page without
manually re-adding the download and finding the source URL elsewhere.
This commit adds:
- POST /api/downloads/jobs/:jobId/retry endpoint (controller + service)
- retryDownloadJob() API method on the frontend
- Failed-state UI in ActiveDownloads.tsx now shows:
- Retry button (re-dispatches the original download job)
- 'Download page' external link (when the download URL is an HTTP(S) URL)
- Loading state on the retry button while the request is in-flight
- Screenshots documenting before/after UI
Co-authored-by: eizus <hello@cdr.xyz>
The map_markers backend has accepted a `notes` column since PR #770 and
the popup display path was wired up to render it (commit 6328256), but
the placement UI never got an input. Result: notes are stored,
displayed when present, and impossible to actually enter via the UI.
Add a notes textarea below the name input in the placement popup,
thread the value through `addMarker` and `createMapMarker`, and trim +
null-coalesce on save. Notes display in the marker popup on click is
unchanged and now actually reachable.
- admin/inertia/lib/api.ts: extend createMapMarker request type with
optional notes
- admin/inertia/hooks/useMapMarkers.ts: addMarker accepts and forwards
notes (response already populated notes into local state, so no
display-side change needed)
- admin/inertia/components/maps/MapComponent.tsx: markerNotes state,
textarea after name input, threaded into handleSaveMarker
Edit-mode for existing markers (so users can backfill notes on
already-placed pins) is intentionally out of scope here - selected-marker
popup is still read-only. That's a follow-up PR if there's demand.
Rebuilt on top of dev's RFC #883 state-machine UI rather than the now-defunct
StoredFile shape:
- Extend StoredFileInfo with fileName/size/uploadedAt/isUserUpload
- Populate metadata from on-disk stats in RagService.getStoredFiles
- Add fileSourceSchema validator + getFileContent/downloadFile endpoints
scoped to the uploads directory only (tighter than the original PR — matches
docs_service traversal pattern)
- KnowledgeBaseModal: sortable Size and Uploaded columns; View/Download
buttons on upload-bucket rows; new FileViewerModal for in-browser text
preview. Bucket grouping preserved — sort applies within each bucket.
- Use formatBytes from ~/lib/util rather than redefining
The maps page reset to the default US-wide view on every refresh because
initialViewState was hardcoded. Save the position and zoom to localStorage
on each move-end (key nomad:map-view, matching the existing scale-unit
pattern) and restore it at mount: saved view → default. The restore is
bounds-checked so a corrupt value falls through to the default.
Replaces #815, whose branch had drifted far out of scope (56 files of
stale-base/merged-commit noise plus unrelated map-feature WIP). This is
just the persist-view improvement, ported cleanly onto current dev. The
null-island and coordinate-search parts of #815 targeted URL-param code
that never landed on dev, so they don't apply here.
Let users override an app's "Open" link with a reverse-proxy or
local-DNS address (e.g. https://jellyfin.myhomelab.net). Falls back to
the default host+port when unset. Metadata-only — no container changes.
Tailwind v4 dropped the default `button { cursor: pointer }` preflight
rule. The Always/Manual toggle in the KB modal uses inline <button>
elements (not the StyledButton wrapper that sets cursor-pointer
explicitly), so hovering them shows the default arrow cursor instead of
a pointer like every other clickable in the modal. Reported on NOMAD1
during v1.32.0 dogfooding.
Add cursor-pointer when not pending; keep cursor-not-allowed when the
mutation is in flight.
Since PR #36b6d8e moved tier-installation tracking from a client-side
persistence model to a server-side derive-from-disk model, the card
display only ever updates once every file in a tier is fully on disk.
A user who picks Standard sees a blank card for the duration of the
download (often hours for large tiers like Wikibooks). Worse, if some
files finish before others, the card briefly shows a lower tier (e.g.
Essential) before promoting to the selected tier on completion, which
reads as "the system didn't accept my pick."
Backend: compute a sibling `downloadingTierSlug` by unioning installed
resource IDs with the IDs from active RunDownloadJob queue entries
(waiting + active + delayed, failed deliberately excluded), then
resolving the highest tier whose every resource is in that union. Set
only when it differs from `installedTierSlug` — no point reporting
"downloading Standard" when Standard is already fully installed.
Frontend: unify the prominent corner badge logic in CategoryCard to a
single `badgeTier` derived from selectedTier > downloadingTier >
installedTier. Spinner + "(downloading)" suffix when in flight,
checkmark for installed/selected. The pill row and lime border follow
the same source.
Verified on NOMAD3: backend correctly resolves the downloading tier
from in-flight BullMQ jobs; CategoryCard shows the spinner badge
immediately on Submit and switches to the checkmark variant when
downloads complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related fixes surfaced by armandoescalante in #915 when clicking a
Content Explorer category card (e.g. Medicine) on v1.32.0-rc.6:
1. TierSelectionModal placed a useMemo for freeBytes *after* the
`if (!category) return null` early return (introduced in PR #901's
guardrail integration). When `category` transitioned from null to
non-null on first open, React saw a different hook count between
renders and crashed the entire component tree with "Rendered more
hooks than during the previous render", blanking the modal. Moved
the freeBytes useMemo above the early return so hook order is
constant.
2. `IconLibrary` was used as the icon prop on the Manage Custom
Libraries button in remote-explorer.tsx but never registered in
the DynamicIcon allowlist at admin/inertia/lib/icons.ts. Added it
to both the import block and the icons map so the warning stops
firing and the icon renders.
Closes#915.
Surfaces NOMAD's previously-silent model-stacking behavior and enforces a
"one chat model in VRAM at a time" invariant (the embedding model is
always exempt). Addresses Chris's NOMAD3 testing observation that
switching the dropdown in the chat header was invisibly slow on low-VRAM
hardware because the prior model was never unloaded — Ollama would
either evict it under memory pressure or load the new one on CPU after
the runner choked.
Three integration points all funnel through one new helper:
- **User changes the model dropdown** in an active chat session →
confirm modal "Switch to {newModel}? Switching to {newModel} will
start a new chat. Your current conversation stays available in the
sidebar." On confirm, fire `keep_alive: 0` against the previous chat
model, clear active session, set the new selection. Cancel snaps the
visible dropdown back to the previous value (no popup state leaks
into `selectedModel`).
- **User clicks a session in the sidebar** → no popup (system-initiated).
Restore the session's stored model into the dropdown and fire
`unloadChatModels(targetModel)` so anything that isn't the target
gets the unload hint.
- **Chat page first mount** → page-load normalization. Anything stacked
from a prior session gets the unload hint with the current selected
model as the target-to-preserve. Guarded by a ref so it only fires
once per page lifetime; gated on `selectedModel` being populated.
Backend surface is a single new helper and a single new route:
`OllamaService.unloadAllChatModelsExcept(targetModel: string | null)`
→ queries `/api/ps`, filters out (a) the embedding model name
(hardcoded `nomic-embed-text:v1.5` to avoid the RagService circular
import) and (b) `targetModel`, fires `POST /api/generate` with empty
prompt + `keep_alive: 0` in parallel against everything else.
Returns the names that were hinted. Best-effort: network or Ollama
errors are logged and swallowed so callers don't fail on housekeeping.
`POST /api/ollama/unload-chat-models` → thin wrapper validating
`{ targetModel?: string | null }`.
Why `keep_alive: 0` is safe against in-flight inference: per Ollama's
scheduler semantics, the hint sets the post-completion eviction timer
to zero — the runner is not terminated. If Session A is mid-response
on gemma when Session B fires the unload, gemma stays resident until
A's request completes, then evicts. The user-visible worst case is the
race where A's longer-running request re-extends the timer back to the
default and the unload is no-op'd; the next transition (or page reload)
gets another chance, and Ollama's own LRU catches up under memory
pressure regardless. Robust in-flight tracking deferred to a follow-up
if we see stale-state in the wild.
Base `rc`: v1.40.0 will inherit everything from rc.6 via the backmerge.
Frontend tests deferred to a follow-up PR; existing inertia tsconfig
errors are pre-existing and unrelated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the Manual-mode UX dead-end: after toggling 'Auto-index new content
for AI?' to Manual, a freshly-downloaded ZIM (or any pending_decision file)
had no UI path to opt in for embedding short of the global Sync Storage /
Re-embed All bulk actions. Per RFC #883 §5, each Stored Files row now
carries a state pill and an adaptive single-button action.
State pill (left of any existing warning chips):
- 'Indexed' — green; row had chunks in Qdrant or state row is 'indexed'
- 'Not Indexed' — neutral; state is pending_decision or browse_only
- 'Failed' — red
- 'Stalled' — amber
- admin_docs collapsed row has no pill ('Managed by NOMAD' carries it)
Adaptive action button (paired with the existing Delete button per row):
- pending_decision → 'Index' (force=false)
- browse_only → 'Index' (force=true)
- failed / stalled → 'Retry' (force=true)
- indexed + warning chip → 'Re-embed' (force=true; confirm modal first)
- indexed healthy / null → no action button (bulk Re-embed All covers it)
Backend: GET /api/rag/files now returns
{ files: Array<{ source, state, chunksEmbedded }> }
instead of a flat string[]. State + chunk-count come from a single
KbIngestState query unioned into the existing Qdrant-derived source list
(no new round trips). New POST /api/rag/files/embed validates the source is
known, refuses if any inflight job already targets the same filePath
(prevents double-click duplicate-chunk hazard), pre-deletes Qdrant points
when force=true, then dispatches via the existing _dispatchEmbedJobsFor
helper used by reembedAll.
Per-file Re-embed (force=true on an already-indexed file) routes through a
StyledModal confirmation since it deletes existing vectors before queueing
a fresh job — same destructive-action weight as Delete's inline confirm but
heavier since it affects search until the rebuild finishes.
Folds in PR #907's blank-screen fix because my new render needs the same
generic restored: `<StyledTable<KbFileGroup>>` and `record.displayName`
(instead of the unresolved `sourceToDisplayName(record.source)` that ships
in rc.5 and ReferenceErrors on modal open). PR #907 also adds title
tooltips on the three bulk-action buttons; those tooltips are NOT included
here — let PR #907 land first or independently for that part.
Multi-select bulk-opt-in deferred per discussion: most Manual-mode users
ingest 1-2 files at a time, the existing global toggle covers the bulk
case, and checkboxes would expand scope past what rc.6 should hold. Will
file a follow-up issue for an 'Index N pending files' single-click button
once this lands.
Tests-in-PR scope was limited to keeping `kb_file_grouping.spec.ts` green
after the StoredFileInfo[] signature change (added asInfos() wrapper).
Dedicated unit tests for embedSingleFile (unknown source / inflight refused
/ force=true delete-then-dispatch) and the new state-pill rendering will
land in a follow-up PR alongside Playwright coverage of the row actions.
Verification path: NOMAD3 currently runs project-nomad-admin:integration-
rc6-preview (PRs #907 + #908 atop rc.5). After this branch is built into a
new integration tag, I'll re-run targeted Playwright UAT on the KB modal
covering: state pill rendering per state, Index click on pending_decision
opts in cleanly, Retry on failed re-dispatches successfully, Re-embed
confirmation modal copy + delete-then-dispatch on the military-medicine
partial-stall row, and Delete flow untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Stored Knowledge Base Files render crashed on first open in v1.32.0-rc.5
with `ReferenceError: sourceToDisplayName is not defined`. The table column's
render() called `sourceToDisplayName(record.source)` but the function was
extracted to `lib/kb_file_grouping.ts` in PR #892 and never imported in
KnowledgeBaseModal.tsx. The unhandled error unmounts the entire React tree,
so users see a blank screen ~20s after opening the panel.
Root cause: PR #895 (conditional warnings) rewrote the render() and used
`sourceToDisplayName(record.source)` instead of `record.displayName`, which
KbFileGroup already carries from groupAndSortKbFiles(). PR #895's review
follow-up (cbae48a) compounded this by narrowing the StyledTable generic
from `KbFileGroup` to `{source: string}`, hiding the type drift from tsc.
This restores the post-#892 pattern:
- StyledTable generic back to `KbFileGroup`
- Render uses `record.displayName` (works for both per-file rows and the
collapsed admin-docs row; calling sourceToDisplayName on the synthetic
`__admin_docs_group__` would have rendered that literal as the row name).
Also folds in tooltip copy on the three bulk-action buttons (Reset & Rebuild,
Re-embed All, Sync Storage) so the difference in destructiveness is visible
on hover. Uses native `title` attribute via StyledButton's prop pass-through;
no new component dependency.
Inertia tsconfig catches this regression cleanly (TS2304 + TS2339); the
pre-push hook only runs the backend tsconfig which excludes inertia/**, so
the bug shipped. Tracking the typecheck-coverage gap as a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Disable TierSelectionModal Submit while the embed-estimate query is in
flight, so a fast click can't slip past the guardrail with an undefined
estimate.
- Move KbGuardrailModal out of the outer <Transition> and render it as a
Fragment sibling — Headless UI's Transition expects Transition.Child
descendants, not raw conditional siblings.
One-time confirmation step gating bulk indexing actions that would
consume a substantial amount of disk for embedding storage. Fires only
when the user has policy=Always (i.e., the system would auto-index)
AND the estimate trips either:
- GUARDRAIL_ABSOLUTE_BYTES = 50 GB embedding cost, OR
- GUARDRAIL_FREE_DISK_RATIO = 10% of current free disk space
Under policy=Manual the guardrail is silent because the user has
already opted out of automatic ingestion — the files would just queue
as pending_decision either way.
Pieces
- inertia/lib/kb_guardrail.ts: pure decision helper with two constants
and an evaluateGuardrail() that returns a verdict + reasons. No I/O
on the helper itself so the logic is trivially testable
- inertia/components/KbGuardrailModal.tsx: confirmation dialog. Headless
UI Transition + Dialog, amber 'large operation' header, plain-English
estimate summary, [Cancel] / [Proceed anyway] footer. z-[60] so it
layers above the tier modal underneath instead of replacing it
- inertia/components/TierSelectionModal.tsx integration: handleSubmit
now evaluates the guardrail when policy=Always and embedEstimate is
available; if it trips, we stash the verdict in state and render the
guardrail modal as an overlay. Confirm runs finalizeSubmit (which is
the pre-existing onSelectTier + onClose path); Cancel just closes the
guardrail and leaves the tier modal as-is so the user can change
their tier choice or flip the policy
The disk-free signal comes from the existing useSystemInfo hook +
getPrimaryDiskInfo helper. Passing freeBytes=0 (unknown) skips the
relative-disk check, so the modal still works on hosts whose disk
introspection failed — just relies on the absolute 50 GB threshold
Tests
- 9 cases in tests/unit/kb_guardrail.spec.ts: standard small batch (no
trip), exact absolute threshold trips, over-absolute trips, over 10%
free trips, both-at-once trips with two reasons, freeBytes=0 skip,
freeBytes=0 + over-absolute trip, exact-10% boundary trips, just-
under-both safe. All green.
Stacks on feat/kb-tier-estimate-on-disk (#897) — consumes that PR's
estimate endpoint to compute the verdict input. Auto-rebases to rc
when #897 merges.
Pairs with #894 (policy toggle) and #899 (JIT prompt): together the
three PRs cover the 'how do I avoid surprising the user with auto-
indexing they didn't ask for?' arc.
Out of scope (deferred)
- 6 hr time threshold (RFC §7): needs a per-host chunks-per-second
metric we don't capture yet; would be a follow-up after Phase 4
self-calibration (RFC §15) lands
- Wider integration (KbPolicyPromptBanner 'Index now' button, manual
KB-modal sync): TierSelectionModal is the dominant bulk-decision
surface and the right place to land this first
- KbPolicyPromptBanner: add onError toast to maybeLaterMutation so a
failed policy save surfaces to the user instead of looking like a
broken button (banner would otherwise reappear on next chat open
with no explanation).
- KbPolicyPromptBanner: set staleTime: Infinity on the prompt-state
query. For users who already picked a policy (the vast majority),
the result is effectively immutable per session — the mutations
invalidate the key when it actually changes.
When a user opens AI Chat with content available but no global ingest
policy yet recorded, surface a one-time banner above the chat header
asking how they want new content handled:
- 'Index existing content' -> sets rag.defaultIngestPolicy=Always and
triggers a sync so pending_decision files queue immediately
- 'Maybe later' -> sets policy=Manual; existing and future content
waits in pending_decision until the user opts in from the KB modal
After either button is clicked the banner never reappears, because both
write the policy KV (the same one #894 manages via the KB modal toggle).
There is intentionally no 'dismiss without deciding' X — that would just
re-show the banner forever.
Backend
- New GET /api/rag/policy-prompt-state returns
{shouldPrompt, hasContent, totalFiles}
- RagService.getPolicyPromptState() reads KVStore('rag.defaultIngestPolicy')
and counts kb_ingest_state rows; shouldPrompt is true only when policy
is null AND scanner has seen >=1 file (avoids prompting on empty NOMADs)
Frontend
- New KbPolicyPromptBanner component (~120 LOC) handles the two-button
decision flow with optimistic loading state, success/error toasts, and
invalidates kbPolicyPromptState + ingestPolicy + embed-jobs + storedFiles
on success
- Mounted in components/chat/index.tsx as the first child of the main
content column so it sits above the chat title bar without taking space
when shouldPrompt is false (renders nothing)
- Reads aiAssistantName from Inertia page props so banner copy matches
the user's chosen assistant name
Stacks on feat/kb-policy-toggle (#894) because the policy KV mechanism
it writes through is introduced there. Both can land in rc.5; this PR
auto-rebases to rc once #894 merges.
Existing users on first upgrade to v1.32.0 will see this banner on first
chat visit post-upgrade — an explicit opt-in moment for content that was
already on disk. New users see it the first time they have curated
content downloaded.
useEmbedJobs already polls every 2s while jobs are active (and 30s when
idle) and auto-invalidates Stored Files when the queue drains. The
manual Refresh button was a no-op signal — it just confuses users who
click it and see no change. Per-job 'last activity Xs ago' lines remain
as the live-recency indicator.
Stacks on feat/kb-job-status-pill (#893) since the Refresh button only
exists in that branch.
`computeFileWarnings()` previously caught all errors and returned an empty
map, which the frontend rendered as "every file is healthy" — reintroducing
exactly the silent-failure mode this surface exists to expose.
Return `{ ok, warnings }`; flip `ok: false` from the catch. KB modal renders
an inline amber notice under the Stored Files header when `ok === false`,
leaving per-row warning rendering untouched. Transient failures self-heal on
the next 30s poll; no toast spam.
Surfaces two silent failure modes that the prior binary
"any-chunks-in-Qdrant ⇒ embedded" check could not distinguish from
healthy ingestion:
- **Warning A — Zero-chunk file** (file_size > 100 MB, chunks = 0)
Fires on video-only / image-only ZIMs (`lrnselfreliance_en_all`,
TED talks, etc.) that the pipeline completes "successfully" with no
extractable text. AI Assistant literally cannot reference these.
- **Warning B — Partial-embed stall** (chunks < 50% of expected from
the ratio registry). Surfaces the simple_wiki "266 of 600,000 chunks"
case observed during NOMAD1 ingestion testing — previously these
looked identical to fully-completed embeds in the UI.
Both warnings render only when their condition is met (silent by
default; noisy only on real problems).
Base is `feat/kb-ratio-registry` (#891) because Warning B's "expected
chunks" estimate comes from `KbRatioRegistry.estimateChunks()`. GitHub
fast-forwards to `rc` once #891 merges.
- `app/utils/kb_warning_decision.ts` — pure `decideWarnings(inputs)`
with thresholds (`100 MB`, `0.5×`) as exported constants. 10 unit
tests cover the healthy case, both warnings, the under/at/over
boundary, the registry-miss suppression, and the video-only registry
case (`expectedChunks: 0` correctly skips Warning B).
- `RagService.computeFileWarnings()` — single Qdrant scroll tallies
chunks per source, filesystem walk fills in zero-chunk files,
ratio registry estimates the expectation, decision function emits.
- New endpoint `GET /api/rag/file-warnings` returns
`Record<source, FileWarning[]>` (sources with no warnings are
omitted, so the frontend can `warnings[source] ?? []` for clean
defaults).
- KB modal: warnings render inline under the file name as amber-tinted
pills. Polled every 30s alongside the existing health check.
- Warning C — chunks skipped due to length. PR #890 (#881 fix) prevents
the silent drop at the embed boundary, so the underlying condition
shouldn't fire anymore. If we still want to surface "we truncated
N chunks to fit", that needs separate `skipped_count` tracking in
EmbedFileJob — a Phase 2 follow-up.
- Suppressing Warning B during active mid-ingestion. The user can cross-
reference the Processing Queue to know it's in-flight; suppressing
warnings while a job runs would mask real stalls where the job died
mid-batch. Will revisit when per-card status is wired through.
- Use of `kb_ingest_state.chunks_embedded` (#888) as the chunk count
source. This PR uses Qdrant scroll directly so it can land
independently of #888.
- 10 new unit tests on `decideWarnings`, all pass
- Type-check clean
- Hot-patch + browser smoke test deferred until #891 lands (the ratio
registry needs to exist in the DB for `estimateChunks()` to return
non-null estimates — without it, only Warning A fires which is still
useful but Warning B stays dormant)
When a user picks a tier in TierSelectionModal, show how much additional
disk space the AI Assistant will need if the new ZIMs are indexed, plus
a policy-aware footer explaining whether they'll auto-index (Always) or
wait for opt-in (Manual). Estimates consume #891's KbRatioRegistry via a
new POST /api/rag/estimate-batch endpoint.
Backend
- New POST /api/rag/estimate-batch route + RagController.estimateBatch
- VineJS schema accepting array of {filename, sizeBytes}, capped at 500
- KbRatioRegistry.estimateBatch aggregates via the existing prefix-match
lookup, returns {totalChunks, totalBytes, hasUnknown}
- New BYTES_PER_CHUNK_ON_DISK constant (~8 KB: 3 KB vector + ~3 KB chunk
text + ~2 KB payload/index overhead). Tunable; will be replaced by
Phase 4 self-calibration once we have real measurements.
- Controller normalizes incoming filenames via path.basename so callers
that send full paths or URLs still match registry prefixes correctly.
Frontend
- api.estimateEmbeddingBatch() client method
- TierSelectionModal: when localSelectedSlug is set, resolve the tier's
resources (incl. inherited tiers), POST to /estimate-batch, and render
a new info block with the +~X GB figure + ingest-policy copy. Also
fetches rag.defaultIngestPolicy so the same block surfaces whether
indexing will fire automatically or wait for the user.
- resourceFilename() helper extracts the basename from the resource URL
so the registry lookup hits the right prefix regardless of mirror.
Tests
- 4 new cases in tests/unit/kb_ratio_lookup.spec.ts covering the
estimateBatch aggregator: standard sum, unknown-flagging, video-only
ZIM (0 chunks but known, hasUnknown stays false), empty input.
Stacks on feat/kb-ratio-registry (#891) — consumes the registry table
seeded by that PR. Once #891 merges to rc, this PR auto-rebases.
Out of scope for this PR (deferred to follow-ups):
- Per-batch opt-in checkbox (RFC §1's '☑ Also index these for AI') needs
a per-batch policy override path and is a separate PR
- Guardrail modal at 50 GB / 10% free / 6 hr thresholds (RFC §7) is also
separate; this PR is informational, not gating
- Time-to-embed estimate awaits a chunks-per-second metric per host
* feat(KB): per-file ingest state machine (Phase 1 of RFC #883)
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.
## What lands
- New table `kb_ingest_state` keyed by `file_path` with enum state column
(`pending_decision | indexed | browse_only | failed | stalled`).
Independent of `installed_resources` so it covers both curated downloads
and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
Registered now but not consumed yet — JIT prompt + wizard step land in
Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
- Success (final batch) → `indexed` + chunks count
- `UnrecoverableError` → `failed` + error message
- Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
decision tree (see `decideScanAction`). Existing installs auto-backfill
on first scan: files with chunks in Qdrant but no state row become
`indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
disappear entirely instead of leaving an orphan that the next scan
would re-dispatch into nothing.
## What does NOT land here
- Ratio registry (separate PR) — needed for partial-stall detection and
cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
/ Reset & Rebuild would also want to set state, but #886 isn't merged
yet; that wiring goes in a follow-up once #886 lands.
## Target
This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.
## Tests
- 9 new unit tests for `decideScanAction` covering all five states plus
the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
- Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
came back `indexed` on first scan
- Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
came back `pending_decision` and was correctly re-dispatched (Bull
deduped to its historical `:completed` jobId — bgauger's #886 fix
drains that)
- Delete hook: deleting a KB upload via `DELETE /api/rag/files`
removed both the disk file and the state row
* feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4)
Activates the `rag.defaultIngestPolicy` KV registered in Phase 1
(#888) so users on a fresh install (or anyone who picks Manual mode)
no longer get every new ZIM auto-dispatched to the embed pipeline.
## Stacks on #888
This PR's base is `feat/kb-ingest-state-machine` (#888). The state
machine has to be in place for the decision function to be policy-aware;
GitHub will fast-forward the base to `rc` once #888 merges.
## Backend changes
- `decideScanAction` now takes a `policy: 'Always' | 'Manual'` argument
(defaults to `Always` for backward compatibility).
- New `ScanAction` kind: `create_pending`. Manual mode records that the
scanner has seen a new file (so the UI can surface a per-card Index
affordance later) without dispatching an EmbedFileJob.
- `scanAndSyncStorage` reads the KV and passes it through. The scan-result
log line now includes the active policy and a `waiting on user` count
for Manual-mode hits.
- `rag.defaultIngestPolicy` added to `SETTINGS_KEYS` so it's reachable
through the existing `GET/PATCH /api/system/settings` surface — no new
endpoint.
## Frontend changes
- New section in the KB panel between "Why upload" and "Processing Queue":
"Auto-index new content for AI? [Always | Manual]" — segmented radio
with copy explaining the 5-10× disk multiplier. Default Always.
- `useQuery('ingestPolicy')` reads the current value; clicking the
inactive option mutates and shows a notification confirming the new
behavior.
## Tests
- 14 unit tests on `decideScanAction` (was 9) — split into Always-mode
cases (preserves Phase 1's contract) and Manual-mode cases
(`create_pending`, `pending_decision → skip`, etc.).
- Type-check clean.
- Hot-patch + browser verification deferred until #888 lands; the state
machine smoke-tested cleanly on NOMAD3 in #888's PR, and this PR's
decision-tree changes are exhaustively unit-tested.
## RFC open question §3 — policy-change re-trigger
Switching Manual → Always doesn't auto-dispatch existing `pending_decision`
rows immediately. The next scan re-evaluates and dispatches them under
the new policy. This matches the RFC's "treat the switch as I've-
thought-about-it" instinct for the guardrail; full guardrail
implementation lands in Phase 3 task 14.
---------
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
Each in-flight (or stuck) embedding job gets a colored health pill,
relative-activity timestamp, and chunk counter so users can tell at a
glance whether ingestion is making progress.
## Health states
- **🟢 Active** — last batch < 2 min ago
- **🟡 Slow** — last batch 2-5 min ago (CPU-paced multi-batch ingestion
lives here naturally; not always a problem)
- **🔴 Stalled** — last batch > 5 min ago (likely real problem)
- **⚪ Waiting** — queued, no batch started yet
- **🔴 Failed** — job recorded failed status
## What lands
- New backend util `kb_job_health.ts` with pure `computeJobHealth(input)`
decision function. Time-based thresholds (2 min / 5 min) inlined as
constants. 9 unit tests pin the boundaries.
- `EmbedJobWithProgress` gains `lastBatchAt`, `startedAt`, `chunks` —
already set by `EmbedFileJob.handle` on every batch transition, just
not previously surfaced through `listActiveJobs`.
- Frontend `kb_job_health_display.ts` maps each status to a Tailwind
dot color, label, and aria-label so backend and UI stay in sync.
- `ActiveEmbedJobs.tsx` renders the pill, "last activity Xs ago", and
chunk counter above each progress bar. Adds a manual Refresh button
and "Last updated Xs ago" line — the existing 2s/30s auto-poll
cadence in `useEmbedJobs` is left intact.
- Live tick at 5s keeps the relative timestamps current without
re-fetching from the API.
## Not in scope
- Per-card Cancel / Retry / Un-index — separate Phase 2 PR
- Conditional warnings A/B/C — separate Phase 2 PR
- Computing throughput rate (chunks/min) — needs ratio registry consumer
(Phase 2 follow-up); for now the pill answers the "is it stuck?"
question directly without a rate estimate.
Project NOMAD's bundled docs (`/app/docs/*.md` and `README.md`) each
embed as their own KB source — currently rendering as 12+ individual
rows that swamp user-uploaded content in the Stored Files table.
Collapse them into one informational row:
> Project NOMAD documentation · 12 files · Managed by NOMAD
The admin-docs row hides the Delete button (those files would be
re-embedded on the next sync anyway, so deleting is a footgun). User
uploads and ZIMs keep their existing per-row Delete UX.
Also adds deterministic sort: ZIMs → user uploads → admin docs → other,
alphabetical within each bucket. Pure frontend change — `/api/rag/files`
response shape unchanged.
Decision logic extracted to `kb_file_grouping.ts` with 9 unit tests
covering bucket classification, sort order, count noun pluralization,
and empty-input handling.
Closes#796.
The maps API has accepted and persisted `notes` on map markers since
PR #770, but the marker popup component still rendered name only and
ignored the field. Now the popup shows a notes block beneath the name
when it's populated, with whitespace preserved and long text wrapped.
Threaded `notes` through the read path:
- `api.listMapMarkers` / `api.createMapMarker` response types
- `MapMarker` interface in `useMapMarkers` and the data.map projection
- `MapComponent`'s selectedMarker popup
The create/update UI is unchanged — users still set notes via the API
or DB directly, matching the issue's stated scope. A marker entry with
empty/whitespace-only notes renders the same as before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: Updated the map to show the coordinates as the user moves the cursor over the map. Changed the cursor to a crosshairs to make it easier to place map markers.
* Moved the scale unit control to its own component file for easier maintenance. Enhanced the behavior of the coordinate display on the map to not display when over the on screen controls, and the navigation bar. Added a toggle to turn off the coordinate display if the user doesn't wish to see it. Intentionally left the coordinate display when over a map marker so that the coordinates of the map marker can be estimated. In the future I intend to add the coordinates of a map marker when the map marker is clicked so that behavior may change in the future.
---------
Co-authored-by: Kenneth Brewer <kennethbrewer3@protonmail.com>
Three UX issues from manual testing of #780 on NOMAD3.
1. Slider was unusable for multi-step zoom changes
`setLoading(true)` fired immediately on every selection or maxzoom change,
which disabled the slider until the request returned. Even with the 400ms
debounce delaying the network call, the UI was locked the whole time.
User couldn't drag through zoom levels to find the right one.
Fix: bump debounce to 1500ms, move `setLoading(true)` inside the setTimeout
so it only flips after the debounce expires. Slider stays interactive
throughout the wait. Slider `disabled` now only ties to `downloading`
(active extract dispatch), not `loading` (preflight in flight). The
existing requestId stale-safe pattern handles concurrent changes.
2. Newly-downloaded maps didn't show in Stored Map Files until manual refresh
`props.maps.regionFiles` is rendered server-side and passed through Inertia
props; without a partial reload it stayed stale until the user navigated
away and back.
Fix: watch `useDownloads({ filetype: 'map' })` count via a ref. When the
count drops (a download finished), trigger `router.reload({ only: ['maps'] })`
to refresh just the maps prop. Existing pattern from elsewhere in the
codebase.
3. Country picker didn't surface already-downloaded countries
When a user re-opened "Choose Countries" after downloading UK, UK appeared
unchecked with no indication it was already on disk.
Fix: pass installed pmtiles filenames into the modal as a prop; parse with
regex `^([a-z]{2})_[\w-]+_z\d+\.pmtiles$` to extract country codes from
single-country extracts (matching MapService.buildRegionSlug's iso2 lowercase
slug pattern). Render an "Installed" badge on those countries with a tooltip
explaining they're re-selectable for redownload at a different zoom.
Group / custom multi-country extracts don't reverse-map cleanly from
filename and are skipped here. Could be a follow-up if useful.
Files:
admin/inertia/components/CountryPickerModal.tsx
- SINGLE_COUNTRY_FILENAME_RE: iso2 + flexible date + zoom
- installedFilenames prop with default []
- installedCountrySet derivation via useMemo
- "Installed" badge rendering on country list rows
- Debounce: 400ms -> 1500ms; setLoading inside setTimeout
- Slider disabled: only on `downloading`
admin/inertia/pages/settings/maps.tsx
- import useEffect/useRef
- destructure activeMapDownloads from useDownloads
- useEffect on download count drop -> router.reload({ only: ['maps'] })
- pass installedFilenames to CountryPickerModal
All three fixes tested end-to-end on NOMAD3.
* feat(maps): add regional map downloads via go-pmtiles extract
* address Copilot review feedback on PR #780
- auto-refresh preflight on selection/maxzoom change with 400ms debounce and
requestId stale-safety so the confirm button no longer requires a two-step
"Estimate Size" -> "Start Download" dance
- safeUpdateProgress helper replaces fire-and-forget updateProgress().catch()
pattern so cancelled-job errors (code -1) can't surface as unhandled rejections
- gate world basemap source on worldBasemapReady - when ensureWorldBasemap()
fails we already delete world.pmtiles, so emitting the source was producing
404s on every tile request
- verify go-pmtiles binary SHA256 at image build time; upstream doesn't ship a
checksums file so per-arch hashes are pinned as build args with a regenerate
note when bumping PMTILES_VERSION
Adds a check to RAG health to make sure nomad_qdrant is online, if not
then the user will be blocked from clicking any buttons in the KB modal
until they click the start qdrant button and let the container start
There is a new file qdrant_restart_policy_provider.ts, which tries to
ensure that the restart policy always exists for the nomad_qdrant
container even though the policy should have been there when the
container is created.
chriscrosstalk
Jake Turner
caweis
Andrew Barnes
just-jbc
Chris Sherwood
jarvisxyz
Benjamin Smith
jakeaturner
Henry Estela
Kenneth Brewer
0xGlitch