Commit Graph

668 Commits

Author SHA1 Message Date
jarvisxyz 4db0010a0d
fix(downloads): add retry button and resource download link for failed downloads (#1059)
Previously, failed downloads showed only an alert icon and a dismiss (X)
button — the user had no way to retry or reach the resource page without
manually re-adding the download and finding the source URL elsewhere.

This commit adds:
- POST /api/downloads/jobs/:jobId/retry endpoint (controller + service)
- retryDownloadJob() API method on the frontend
- Failed-state UI in ActiveDownloads.tsx now shows:
  - Retry button (re-dispatches the original download job)
  - 'Download page' external link (when the download URL is an HTTP(S) URL)
- Loading state on the retry button while the request is in-flight
- Screenshots documenting before/after UI

Co-authored-by: eizus <hello@cdr.xyz>
2026-08-04 03:37:31 +00:00
chriscrosstalk 8ceb2535c2
chore(catalog): sunset orphaned Meshtastic Daemon card (#1049)
Meshtastic Daemon was pulled from DEFAULT_SERVICES because it can't work
without hands-on setup (radio MAC address, etc.). The seeder never deletes,
so every early-access deployment keeps an orphaned nomad_meshtasticd row and
still shows the broken card.

Migration mirrors the legacy-Kolibri sunset: drop the row where installed=0,
flag is_deprecated where installed=1 (keeps it manageable, hides from catalog).
Runs automatically on each box's next update.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 03:37:31 +00:00
chriscrosstalk ab87e29dd5
chore(KB): filter non-content sections + render tables in ZIM extraction (#1044)
Closes #902.

Two gaps in the structured ZIM extraction path:

1. NON_CONTENT_HEADING_PATTERNS was only used by the structure heuristic to
   count meaningful sections, never at section-emit time. Sections under
   "See also" / "References" / "External links" / etc. were still chunked and
   embedded. They're now flagged when the heading opens and dropped.

2. <table> elements were run through cheerio's `.text()`, concatenating every
   cell with no separators ("AgeDoseAdult500mg") into unsearchable word salad.
   New tableToText() joins cells with " | " and rows with newlines so
   row/column structure survives into the chunk.

Refactor: moved extractStructuredContent out of ZIMExtractionService into a
pure, cheerio-only util (app/utils/zim_html.ts) so it can be unit-tested
without the native @openzim/libzim binding. Service delegates to it; behavior
is otherwise unchanged. Adds tests/unit/zim_html.spec.ts (6 tests).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 03:37:30 +00:00
chriscrosstalk ed1fed8d2a
feat(maps): add notes input to map pin placement popup (#926)
The map_markers backend has accepted a `notes` column since PR #770 and
the popup display path was wired up to render it (commit 6328256), but
the placement UI never got an input. Result: notes are stored,
displayed when present, and impossible to actually enter via the UI.

Add a notes textarea below the name input in the placement popup,
thread the value through `addMarker` and `createMapMarker`, and trim +
null-coalesce on save. Notes display in the marker popup on click is
unchanged and now actually reachable.

- admin/inertia/lib/api.ts: extend createMapMarker request type with
  optional notes
- admin/inertia/hooks/useMapMarkers.ts: addMarker accepts and forwards
  notes (response already populated notes into local state, so no
  display-side change needed)
- admin/inertia/components/maps/MapComponent.tsx: markerNotes state,
  textarea after name input, threaded into handleSaveMarker

Edit-mode for existing markers (so users can backfill notes on
already-placed pins) is intentionally out of scope here - selected-marker
popup is still read-only. That's a follow-up PR if there's demand.
2026-08-04 03:37:30 +00:00
chriscrosstalk 885aebd109
content(collections): fix four dead Wikipedia download URLs (#1189)
Four of the five Wikipedia packages pointed at 2025-12 builds that openZIM
has since rolled forward and deleted. Every one returns 404:

  Quick Reference                  wikipedia_en_top_mini_2025-12    404
  Popular Articles                 wikipedia_en_top_nopic_2025-12   404
  Complete Wikipedia (Compact)     wikipedia_en_all_mini_2025-12    404
  Complete Wikipedia (No Images)   wikipedia_en_all_nopic_2025-12   404

Only Complete Wikipedia (Full) still resolved, and at 124 GB that's the
option almost nobody picks. So in practice a user choosing any sensible
Wikipedia size during Easy Setup got a failed download.

Repointed all four at the current 2026-06 builds and corrected every
size_mb from the measured Content-Length. The old figures were estimates
and had drifted by up to 8%.

Sizes are decimal MB, consistent with kiwix-categories.json.

Refs #1171

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 16:22:20 -07:00
chriscrosstalk 66c6676740
ci: fix collection url validation (#1188)
Co-authored-by: jakeaturner <jturner@cosmistack.com>
2026-08-02 16:18:19 -07:00
Chris Sherwood c693cd5cd8 content(collections): rebalance Survival & Preparedness and fix a broken URL
The category was 10.8 GB of which 89% was YouTube channel archives. Nothing
in Essential or Standard could be read without watching a video, which is
the wrong shape for the situations this content is for: low power, low
bandwidth, or just looking something up quickly.

Fixes a live 404. canadian_prepper_preppingfood_en_2025-09.zim no longer
exists upstream, so anyone installing the Comprehensive tier today hits a
broken download (#1171). openZIM renamed it, and because resource_id is
derived from the on-disk filename by parseZimFilename(), the id has to
change with it or the installed file would never match its catalog entry.

Adds 5 GB of searchable text across the three tiers:

  Essential      water treatment, food preparation, knots (144 MB total)
  Standard       Ready.gov, outdoors Q&A, amateur radio Q&A
  Comprehensive  post-disaster library, Hundred Rabbits

Amateur radio in particular filled a gap: communications is a core
preparedness topic and the category had nothing on it.

Text share goes from 11% to 36%. Essential grows by 144 MB, which is 6%,
and stops being video-only.

Deliberately not included: gardening.stackexchange and Gutenberg Agriculture
are already curated under Agriculture & Food, and survivorlibrary.com is the
single most on-mission corpus in the Kiwix library but is 252 GB.

All URLs verified with range requests; all ids verified to match their
filename base.

Refs #1171, #1149

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 12:30:09 -07:00
Chris Sherwood 60f3dd57c0 content(collections): add CD3WD to Survival & Preparedness comprehensive
CD3WD is a compilation of appropriate-technology and development
literature: food production and storage, water and sanitation,
construction, health, and village-scale manufacturing without
industrial supply chains. Requested in #1149.

The survival category was until now almost entirely YouTube channel
archives, with Project Gutenberg military science as the only text
reference. CD3WD adds a substantial practical reference at 581 MB,
which is the smallest resource in the category by a wide margin.

Verified live: HTTP 206 on a range request, 581,165,229 bytes.

Refs #1149

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 12:30:09 -07:00
chriscrosstalk 56cafe564c
catalog: add The Modern Rogue creator pack (#1145)
Adds the Modern Rogue pack (Brian Brushwood) to the Creator Packs catalog.
37 videos, 5177 MB. ZIM (modern-rogue_2026-07.zim) is uploaded to R2 and
verified serveable via the entitlement Worker.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 11:48:27 -07:00
jakeaturner 81667eb45b
chore(collections): add creator-packs.json to main 2026-07-24 03:54:14 +00:00
cosmistack-bot 6a4f02dd46 docs(release): finalize v1.33.0 release notes [skip ci] 2026-06-23 17:52:28 +00:00
cosmistack-bot cf294bf908 chore(release): 1.33.0 [skip ci] 2026-06-23 17:51:38 +00:00
jakeaturner c794d5e4b0
Revert "chore(release): 1.33.0 [skip ci]"
This reverts commit d8cca183f9.
2026-06-23 17:49:54 +00:00
jakeaturner 587600ba78
Revert "docs(release): finalize v1.33.0 release notes [skip ci]"
This reverts commit c9a86790de.
2026-06-23 17:49:36 +00:00
jakeaturner e82a2934c7
Revert "chore(release): 1.33.1 [skip ci]"
This reverts commit 0ee5dda534.
2026-06-23 17:49:17 +00:00
cosmistack-bot 0ee5dda534 chore(release): 1.33.1 [skip ci] 2026-06-23 17:41:25 +00:00
jakeaturner f4af81ef02
fix: lazy-connect + retry for ioredis to avoid blocked startup 2026-06-23 17:40:10 +00:00
cosmistack-bot c9a86790de docs(release): finalize v1.33.0 release notes [skip ci] 2026-06-23 04:49:49 +00:00
cosmistack-bot d8cca183f9 chore(release): 1.33.0 [skip ci] 2026-06-23 04:48:59 +00:00
jakeaturner 76ce725340
chore(release): set version to 1.33.0-rc.1 2026-06-23 04:47:13 +00:00
jakeaturner 9609edc281
chore(docs): update release notes 2026-06-23 04:47:12 +00:00
Jake Turner 0143afe7cc
fix(supply-depot): bump default Ollama and CyberChef image versions (#1036) 2026-06-23 04:47:12 +00:00
Jake Turner 5af27e9904
fix(supply-depot): ensure all curated images pinned to specific versions (#1033) 2026-06-23 04:47:12 +00:00
Jake Turner 88ac4d5ec4
feat(RAG): adds the ability to cancel all embedding jobs (#1034) 2026-06-23 04:47:11 +00:00
jakeaturner f0142b67f8
chore(docs): update release notes 2026-06-23 04:47:11 +00:00
jakeaturner 4a795df793
feat: configurable internet test url override in new Advanced Settings page 2026-06-23 04:47:10 +00:00
Jake Turner 02c9f72bf0
fix(UI): unifies Supply Depot icon and improves loading UX (#1022) 2026-06-23 04:47:10 +00:00
Benjamin Smith 5181637926
feat(KnowledgeBase): add document viewer, download, metadata, and sorting (#721)
Rebuilt on top of dev's RFC #883 state-machine UI rather than the now-defunct
StoredFile shape:

- Extend StoredFileInfo with fileName/size/uploadedAt/isUserUpload
- Populate metadata from on-disk stats in RagService.getStoredFiles
- Add fileSourceSchema validator + getFileContent/downloadFile endpoints
  scoped to the uploads directory only (tighter than the original PR — matches
  docs_service traversal pattern)
- KnowledgeBaseModal: sortable Size and Uploaded columns; View/Download
  buttons on upload-bucket rows; new FileViewerModal for in-browser text
  preview. Bucket grouping preserved — sort applies within each bucket.
- Use formatBytes from ~/lib/util rather than redefining
2026-06-23 04:47:09 +00:00
jakeaturner fad9e30ddc
fix(zim): stabilize uploader state to avoid cancel on tab refocus
Also improves surfacing of upload error messages
2026-06-23 04:47:09 +00:00
Henry Estela a7d05859d4
feat(zim): add zim uploader in content manager
adds a collapsible file uploader to accept zim file uploads into kiwix.
2026-06-23 04:47:08 +00:00
Lorenzo Galassi fe6735fdcb
fix(queue): share one ioredis connection across BullMQ queues and workers (#1009)
BullMQ instantiates a fresh ioredis client per Queue/Worker when handed a
plain {host, port} config object, and under sustained ZIM ingestion the
embed pipeline leaked ~1 client/sec until Redis maxclients was exhausted.
Pass a single shared ioredis instance (maxRetriesPerRequest: null, as
required by BullMQ) so all queues and workers reuse one client pool.
Workers still duplicate the connection once for their blocking client,
which is expected and bounded.

Closes #885
2026-06-23 04:47:08 +00:00
Jake Turner b507b8bd4e
chore(deps): bump mysql2 in admin (#1021) 2026-06-23 04:47:07 +00:00
Jake Turner 8982d93a31
feat: replace legacy Kolibri image default with latest v19 image (#1019)
* feat: replace legacy Kolibri image default with latest v19 image
* feat(supply-depot): add content migration instructions for Edu Platform Gen 1 to 2
2026-06-23 04:47:07 +00:00
jakeaturner 1b040c8b9a
fix: pin default meshcore web image 2026-06-23 04:47:07 +00:00
jakeaturner 475781ffa2
build: ensure openssl installed in admin container 2026-06-23 04:47:06 +00:00
Chris Sherwood bd65c885be
feat(supply-depot): add MeshCore Web with self-signed HTTPS
Adds the MeshCore web client to the Supply Depot catalog (host port 8500),
alongside the existing Meshtastic apps. Uses aXistem's prebuilt image of Liam
Cottle's MeshCore client (MeshCore is a sibling LoRa mesh project to Meshtastic).

The image is stock nginx serving a static Flutter build over HTTP, but the
client reaches radios via Web Bluetooth / Web Serial, which browsers only allow
from a secure (HTTPS) context. So we serve it over HTTPS: a new preinstall hook
generates a self-signed cert + a small SSL nginx config into storage/meshcore-web,
both bind-mounted into the container (the config over the image's default.conf),
publishing 443. Same one-time browser-warning approach as Vaultwarden, whose
openssl cert generation is refactored into a shared _ensureSelfSignedCert helper.

Also adds a NOMAD-specific docs section + Manage>Docs anchor, and registers the
IconAntenna icon. Meshtastic Web left unchanged.

Validated on NOMAD3 (v1.33.0-rc.1): the image + SSL config + self-signed cert
serves the MeshCore Flutter app over HTTPS 200 with working SPA fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 04:47:06 +00:00
johno10661 83576ec33d
feat(supply-depot): add uninstall for curated apps (#1006)
Curated catalog apps could be installed, stopped, and force-reinstalled,
but never removed — the only path off a device was manual docker + DB
surgery. Custom apps already had delete; this adds the equivalent for
curated apps.

POST /api/system/services/uninstall stops and removes the app's
container (optionally its image, same best-effort semantics as custom
app delete) and flips the record back to not-installed so the card
returns to the available catalog. Host bind-mount data is deliberately
left on disk, so a later reinstall picks the app back up where it left
off — unlike force-reinstall, which clears volumes.

Guards: custom apps are rejected (use delete), dependency services are
rejected, and uninstalling a not-installed app is a 409.

UI: installed curated cards get an Uninstall action in the card menu,
with a confirm modal that explains data is preserved and offers the
same remove-image checkbox as custom app delete.
2026-06-23 04:47:05 +00:00
Chris Sherwood e2beb7ebeb
docs: document Supply Depot + auto-updates in README
Bring the GitHub-facing README in line with the v1.33 feature set:

- Add Supply Depot (one-click app catalog + bring-your-own custom
  Docker containers) and Automatic Updates to the "Built-in
  capabilities" list and the "What's Included" table.
- Fix the Offline Maps row, which claimed "navigation" — NOMAD maps
  are download/view only, with no routing. Reword to "offline viewing
  and search."

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 04:47:05 +00:00
Chris Sherwood 536ad49277
docs: update in-app docs for v1.33 Supply Depot + auto-updates
Refresh the in-app Markdoc docs for the v1.33 feature set:

- Repoint dead /settings/apps links to the Supply Depot (/supply-depot)
  across home, getting-started, and faq; reword "Apps page" / "Settings
  -> Apps" to "Supply Depot". The old /apps route now redirects to the
  Supply Depot.
- Expand supply-depot-apps.md with a "Managing your apps" section (Docs/
  Edit/Logs/Stats/Update/Remove, version + update-available visibility,
  custom launch URLs, per-app auto-update toggle) and a "Bringing your
  own app" section for custom Docker containers.
- Add a new "Updates" doc (updates.md) covering the auto-update trilogy
  (core/apps/content), manual updates, and the Early Access channel;
  wire it into DOC_ORDER and cross-link from home, getting-started, faq.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 04:47:04 +00:00
jakeaturner 61ef8f3697
chore: force bump main version 2026-06-23 04:47:04 +00:00
jakeaturner 5eb208fb50
docs: update release notes 2026-06-23 04:47:03 +00:00
jakeaturner 37ad684fbd
fix(bullmq): bump to 5.77.6 and update set calls w new args shape 2026-06-23 04:47:03 +00:00
jakeaturner d4972445d9
chore(deps): bump autoprefixer 2026-06-23 04:47:02 +00:00
jakeaturner b6a08058e8
chore(deps): bump react and react-dom 2026-06-23 04:47:02 +00:00
jakeaturner 6a2d4c2bf6
feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
akashsalan 9de6473f6a
fix(system): prevent false offline reports when Cloudflare endpoint is unreachable 2026-06-23 04:47:01 +00:00
Chris Sherwood bbd62d8ed1
fix(content): remove superseded curated map/ZIM files when a new version installs
Only Wikipedia had version cleanup; every other curated map and non-Wikipedia
ZIM left its prior version on disk when a newer one installed, so users silently
accumulated orphaned content (potentially hundreds of GB). (#634)

The install paths already record each resource via InstalledResource
{resource_id, resource_type, version, file_path}, so the authoritative old-file
path for a resource is known. On install of a new version we now capture the
prior row before updateOrCreate repoints it, then delete the old file — gated
behind a pure, fully unit-tested decision function with strict safety rails:

  - tracked-only: requires a prior InstalledResource row for the same
    resource_id, so sideloaded/untracked files are never touched
  - genuine replacement: old and new file paths must differ
  - new-file-verified: the new file must be confirmed on disk first
  - strictly-newer: a re-install or downgrade can't wipe a newer file
  - within-storage-dir: the old path must resolve under the content store

ZIM cleanup deletes the old file directly (NOT via this.delete(), which would
drop the InstalledResource row by resource_id that updateOrCreate just
repointed) and rebuilds the Kiwix library only if a file was actually removed,
so its XML never references a deleted ZIM. Maps need no library step. Wikipedia
keeps its own existing cleanup path. All deletions are best-effort and logged;
a failure never breaks the install.

Closes #634

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 04:47:01 +00:00
John Onysko 2ae30a4abb
fix(chat): prefer selected model for suggestions, fall back to smallest
`getChatSuggestions` previously picked the largest installed model by file
size, on the assumption that bigger models give better suggestions. This
is unsafe: if any installed model exceeds available VRAM (e.g.
llama3.1:405b on a 96 GB GPU), Ollama spends minutes trying to load it
and the request 500s — making the chat page unusable for anyone who
happens to keep a flagship-sized model on disk.

Chat suggestions are short prompts that don't benefit from a flagship
model anyway. Prefer the user's selected `chat.lastModel` when set, and
fall back to the smallest installed model otherwise. `OllamaService.getModels()`
already excludes embedders, so the fallback always picks a chat model.
2026-06-23 04:47:00 +00:00
jakeaturner e5565d1e95
refactor(supply-depot): extract version and subtitle helpers instead of IIFE for readability 2026-06-23 04:47:00 +00:00
Chris Sherwood 4ece29b6d4
feat(supply-depot): show installed version on cards + make Update pill stand out
Two card tweaks for the update workflow:

- Show the installed image tag next to the powered-by name (e.g. "Kiwix ·
  3.7.0"), so the running version is visible at a glance. Only rendered for
  installed apps; falls back to just the version when powered_by is unset.
- Change the "Update available" pill from a muted light-green tint to a solid
  desert-orange fill with white text, so an available update actually draws
  the eye instead of blending into the card.
2026-06-23 04:46:59 +00:00