Activates the `rag.defaultIngestPolicy` KV registered in Phase 1
(#888) so users on a fresh install (or anyone who picks Manual mode)
no longer get every new ZIM auto-dispatched to the embed pipeline.
## Stacks on #888
This PR's base is `feat/kb-ingest-state-machine` (#888). The state
machine has to be in place for the decision function to be policy-aware;
GitHub will fast-forward the base to `rc` once #888 merges.
## Backend changes
- `decideScanAction` now takes a `policy: 'Always' | 'Manual'` argument
(defaults to `Always` for backward compatibility).
- New `ScanAction` kind: `create_pending`. Manual mode records that the
scanner has seen a new file (so the UI can surface a per-card Index
affordance later) without dispatching an EmbedFileJob.
- `scanAndSyncStorage` reads the KV and passes it through. The scan-result
log line now includes the active policy and a `waiting on user` count
for Manual-mode hits.
- `rag.defaultIngestPolicy` added to `SETTINGS_KEYS` so it's reachable
through the existing `GET/PATCH /api/system/settings` surface — no new
endpoint.
## Frontend changes
- New section in the KB panel between "Why upload" and "Processing Queue":
"Auto-index new content for AI? [Always | Manual]" — segmented radio
with copy explaining the 5-10× disk multiplier. Default Always.
- `useQuery('ingestPolicy')` reads the current value; clicking the
inactive option mutates and shows a notification confirming the new
behavior.
## Tests
- 14 unit tests on `decideScanAction` (was 9) — split into Always-mode
cases (preserves Phase 1's contract) and Manual-mode cases
(`create_pending`, `pending_decision → skip`, etc.).
- Type-check clean.
- Hot-patch + browser verification deferred until #888 lands; the state
machine smoke-tested cleanly on NOMAD3 in #888's PR, and this PR's
decision-tree changes are exhaustively unit-tested.
## RFC open question §3 — policy-change re-trigger
Switching Manual → Always doesn't auto-dispatch existing `pending_decision`
rows immediately. The next scan re-evaluates and dispatches them under
the new policy. This matches the RFC's "treat the switch as I've-
thought-about-it" instinct for the guardrail; full guardrail
implementation lands in Phase 3 task 14.
Each in-flight (or stuck) embedding job gets a colored health pill,
relative-activity timestamp, and chunk counter so users can tell at a
glance whether ingestion is making progress.
## Health states
- **🟢 Active** — last batch < 2 min ago
- **🟡 Slow** — last batch 2-5 min ago (CPU-paced multi-batch ingestion
lives here naturally; not always a problem)
- **🔴 Stalled** — last batch > 5 min ago (likely real problem)
- **⚪ Waiting** — queued, no batch started yet
- **🔴 Failed** — job recorded failed status
## What lands
- New backend util `kb_job_health.ts` with pure `computeJobHealth(input)`
decision function. Time-based thresholds (2 min / 5 min) inlined as
constants. 9 unit tests pin the boundaries.
- `EmbedJobWithProgress` gains `lastBatchAt`, `startedAt`, `chunks` —
already set by `EmbedFileJob.handle` on every batch transition, just
not previously surfaced through `listActiveJobs`.
- Frontend `kb_job_health_display.ts` maps each status to a Tailwind
dot color, label, and aria-label so backend and UI stay in sync.
- `ActiveEmbedJobs.tsx` renders the pill, "last activity Xs ago", and
chunk counter above each progress bar. Adds a manual Refresh button
and "Last updated Xs ago" line — the existing 2s/30s auto-poll
cadence in `useEmbedJobs` is left intact.
- Live tick at 5s keeps the relative timestamps current without
re-fetching from the API.
## Not in scope
- Per-card Cancel / Retry / Un-index — separate Phase 2 PR
- Conditional warnings A/B/C — separate Phase 2 PR
- Computing throughput rate (chunks/min) — needs ratio registry consumer
(Phase 2 follow-up); for now the pill answers the "is it stuck?"
question directly without a rate estimate.
Project NOMAD's bundled docs (`/app/docs/*.md` and `README.md`) each
embed as their own KB source — currently rendering as 12+ individual
rows that swamp user-uploaded content in the Stored Files table.
Collapse them into one informational row:
> Project NOMAD documentation · 12 files · Managed by NOMAD
The admin-docs row hides the Delete button (those files would be
re-embedded on the next sync anyway, so deleting is a footgun). User
uploads and ZIMs keep their existing per-row Delete UX.
Also adds deterministic sort: ZIMs → user uploads → admin docs → other,
alphabetical within each bucket. Pure frontend change — `/api/rag/files`
response shape unchanged.
Decision logic extracted to `kb_file_grouping.ts` with 9 unit tests
covering bucket classification, sort order, count noun pluralization,
and empty-input handling.
Closes#796.
The maps API has accepted and persisted `notes` on map markers since
PR #770, but the marker popup component still rendered name only and
ignored the field. Now the popup shows a notes block beneath the name
when it's populated, with whitespace preserved and long text wrapped.
Threaded `notes` through the read path:
- `api.listMapMarkers` / `api.createMapMarker` response types
- `MapMarker` interface in `useMapMarkers` and the data.map projection
- `MapComponent`'s selectedMarker popup
The create/update UI is unchanged — users still set notes via the API
or DB directly, matching the issue's stated scope. A marker entry with
empty/whitespace-only notes renders the same as before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: Updated the map to show the coordinates as the user moves the cursor over the map. Changed the cursor to a crosshairs to make it easier to place map markers.
* Moved the scale unit control to its own component file for easier maintenance. Enhanced the behavior of the coordinate display on the map to not display when over the on screen controls, and the navigation bar. Added a toggle to turn off the coordinate display if the user doesn't wish to see it. Intentionally left the coordinate display when over a map marker so that the coordinates of the map marker can be estimated. In the future I intend to add the coordinates of a map marker when the map marker is clicked so that behavior may change in the future.
---------
Co-authored-by: Kenneth Brewer <kennethbrewer3@protonmail.com>
Three UX issues from manual testing of #780 on NOMAD3.
1. Slider was unusable for multi-step zoom changes
`setLoading(true)` fired immediately on every selection or maxzoom change,
which disabled the slider until the request returned. Even with the 400ms
debounce delaying the network call, the UI was locked the whole time.
User couldn't drag through zoom levels to find the right one.
Fix: bump debounce to 1500ms, move `setLoading(true)` inside the setTimeout
so it only flips after the debounce expires. Slider stays interactive
throughout the wait. Slider `disabled` now only ties to `downloading`
(active extract dispatch), not `loading` (preflight in flight). The
existing requestId stale-safe pattern handles concurrent changes.
2. Newly-downloaded maps didn't show in Stored Map Files until manual refresh
`props.maps.regionFiles` is rendered server-side and passed through Inertia
props; without a partial reload it stayed stale until the user navigated
away and back.
Fix: watch `useDownloads({ filetype: 'map' })` count via a ref. When the
count drops (a download finished), trigger `router.reload({ only: ['maps'] })`
to refresh just the maps prop. Existing pattern from elsewhere in the
codebase.
3. Country picker didn't surface already-downloaded countries
When a user re-opened "Choose Countries" after downloading UK, UK appeared
unchecked with no indication it was already on disk.
Fix: pass installed pmtiles filenames into the modal as a prop; parse with
regex `^([a-z]{2})_[\w-]+_z\d+\.pmtiles$` to extract country codes from
single-country extracts (matching MapService.buildRegionSlug's iso2 lowercase
slug pattern). Render an "Installed" badge on those countries with a tooltip
explaining they're re-selectable for redownload at a different zoom.
Group / custom multi-country extracts don't reverse-map cleanly from
filename and are skipped here. Could be a follow-up if useful.
Files:
admin/inertia/components/CountryPickerModal.tsx
- SINGLE_COUNTRY_FILENAME_RE: iso2 + flexible date + zoom
- installedFilenames prop with default []
- installedCountrySet derivation via useMemo
- "Installed" badge rendering on country list rows
- Debounce: 400ms -> 1500ms; setLoading inside setTimeout
- Slider disabled: only on `downloading`
admin/inertia/pages/settings/maps.tsx
- import useEffect/useRef
- destructure activeMapDownloads from useDownloads
- useEffect on download count drop -> router.reload({ only: ['maps'] })
- pass installedFilenames to CountryPickerModal
All three fixes tested end-to-end on NOMAD3.
* feat(maps): add regional map downloads via go-pmtiles extract
* address Copilot review feedback on PR #780
- auto-refresh preflight on selection/maxzoom change with 400ms debounce and
requestId stale-safety so the confirm button no longer requires a two-step
"Estimate Size" -> "Start Download" dance
- safeUpdateProgress helper replaces fire-and-forget updateProgress().catch()
pattern so cancelled-job errors (code -1) can't surface as unhandled rejections
- gate world basemap source on worldBasemapReady - when ensureWorldBasemap()
fails we already delete world.pmtiles, so emitting the source was producing
404s on every tile request
- verify go-pmtiles binary SHA256 at image build time; upstream doesn't ship a
checksums file so per-arch hashes are pinned as build args with a regenerate
note when bumping PMTILES_VERSION
Adds a check to RAG health to make sure nomad_qdrant is online, if not
then the user will be blocked from clicking any buttons in the KB modal
until they click the start qdrant button and let the container start
There is a new file qdrant_restart_policy_provider.ts, which tries to
ensure that the restart policy always exists for the nomad_qdrant
container even though the policy should have been there when the
container is created.
Adds a cancel button to in-progress Ollama model downloads and unifies
the Active Model Downloads card layout with the Active Downloads card
used for ZIMs, maps, and pmtiles (byte counts, progress bar, live speed,
status indicator).
Closes#676.
Defaults to metric for global audience. Persists choice in localStorage.
Segmented button styled to match MapLibre controls.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add distance scale bar and user-placed location pins to the offline maps viewer.
- Scale bar (bottom-left) shows distance reference that updates with zoom level
- Click anywhere on map to place a named pin with color selection (6 colors)
- Collapsible "Saved Locations" panel lists all pins with fly-to navigation
- Full dark mode support for popups and panel via CSS overrides
- New `map_markers` table with future-proofed columns for routing (marker_type,
route_id, route_order, notes) to avoid a migration when routes are added later
- CRUD endpoints: GET/POST /api/maps/markers, PATCH/DELETE /api/maps/markers/:id
- VineJS validation on create/update
- MapMarker Lucid model
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(downloads): rich progress, friendly names, cancel, and live status
Redesign the Active Downloads UI with four improvements:
- Rich progress: BullMQ jobs now report downloadedBytes/totalBytes instead
of just a percentage, showing "2.3 GB / 5.1 GB" instead of "78% / 100%"
- Friendly names: dispatch title metadata from curated categories, Content
Explorer library, Wikipedia selector, and map collections
- Cancel button: Redis-based cross-process abort signal lets users cancel
active downloads with file cleanup. Confirmation step prevents accidents.
- Live status indicator: green pulsing dot with transfer speed for active
downloads, orange stall warning after 60s of no data, gray dot for queued
Backward compatible with in-flight jobs that have integer-only progress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(downloads): fix cancel, dismiss, speed, and retry bugs
- Speed indicator: only set prevBytesRef on first observation to prevent
intermediate re-renders from inflating the calculated speed
- Cancel: throw UnrecoverableError on abort to prevent BullMQ retries
- Dismiss: remove stale BullMQ lock before job.remove() so cancelled
jobs can actually be dismissed
- Retry: add getActiveByUrl() helper that checks job state before
blocking re-download, auto-cleans terminal jobs
- Wikipedia: reset selection status to failed on cancel so the
"downloading" state doesn't persist
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(downloads): improve cancellation logic and surface true BullMQ job states
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Turner <jturner@cosmistack.com>
The App Installation Activity list on the Easy Setup complete page grew
unboundedly, pushing Active Downloads off-screen. Caps the list at ~8
visible items with overflow scrolling, auto-scrolling to keep the latest
activity visible.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Model downloads that fail (e.g., when Ollama is too old for a model)
were silently retrying 40 times with no UI feedback. Now errors are
broadcast via SSE and shown in the Active Model Downloads section.
Version mismatch errors use UnrecoverableError to fail immediately
instead of retrying. Stale failed jobs are cleared on retry so users
aren't permanently blocked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Failed download jobs persist in BullMQ forever with no way to clear
them, leaving stale error notifications in Content Explorer and Easy
Setup. Adds a dismiss button (X) on failed download cards that removes
the job from the queue via a new DELETE endpoint.
- Backend: DELETE /api/downloads/jobs/:jobId endpoint
- Frontend: X button on failed download cards with immediate refresh
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three bugs caused downloads to hang, disappear, or leave stuck spinners:
1. Wikipedia downloads that failed never updated the DB status from 'downloading',
leaving the spinner stuck forever. Now the worker's failed handler marks them as failed.
2. No stall detection on streaming downloads - if data stopped flowing mid-download,
the job hung indefinitely. Added a 5-minute stall timer that triggers retry.
3. Failed jobs were invisible to users since only waiting/active/delayed states were
queried. Now failed jobs appear with error indicators in the download list.
Closes#364, closes#216
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a "Debug Info" link to the footer and settings sidebar that opens a
modal with non-sensitive system information (version, OS, hardware, GPU,
installed services, internet status, update availability). Users can copy
the formatted text and paste it into GitHub issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a warm charcoal dark mode ("Night Ops") using CSS variable swapping
under [data-theme="dark"]. All 23 desert palette variables are overridden
with dark-mode counterparts, and ~313 generic Tailwind classes (bg-white,
text-gray-*, border-gray-*) are replaced with semantic tokens.
Infrastructure:
- CSS variable overrides in app.css for both themes
- ThemeProvider + useTheme hook (localStorage + KV store sync)
- ThemeToggle component (moon/sun icons, "Night Ops"/"Day Ops" labels)
- FOUC prevention script in inertia_layout.edge
- Toggle placed in StyledSidebar and Footer for access on every page
Color replacements across 50 files:
- bg-white → bg-surface-primary
- bg-gray-50/100 → bg-surface-secondary
- text-gray-900/800 → text-text-primary
- text-gray-600/500 → text-text-secondary/text-text-muted
- border-gray-200/300 → border-border-subtle/border-border-default
- text-desert-white → text-white (fixes invisible text on colored bg)
- Button hover/active states use dedicated btn-green-hover/active vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add custom Markdoc renderers for images, links, paragraphs, code blocks,
inline code, and horizontal rules. Restyle existing heading, table, and
list components to match the desert tactical color palette. Add 8
screenshots to docs with polished image presentation (rounded corners,
shadow, captions). Constrain content width for readability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chris Sherwood
Kenneth Brewer
chriscrosstalk
0xGlitch
Henry Estela
Jake Turner
Jake Turner