26 KiB

Raw Blame History

Offline (Air-Gapped) Installation

Project NOMAD is built to run without an internet connection, but a normal installation still needs one: the installer downloads Docker, host packages, container images and its own helper scripts. That leaves a gap for anyone who wants to prepare a machine now and deploy it somewhere with no connectivity — or rebuild a machine after connectivity is gone.

Offline artifact mode closes that gap. You build a bundle once on a connected machine, carry it to the target on a USB drive, and run the ordinary installer against it.

# On a connected build machine, from a Project NOMAD checkout (needs only Docker):
./install/build_offline_bundle_docker.sh --target ubuntu:26.04 --output /media/usb/NOMAD

# On the disconnected target, from inside the bundle directory:
sudo bash ./install_nomad.sh --artifacts .

There is only one installer. install_nomad.sh with no --artifacts argument behaves exactly as it always has.

What the bundle covers
Building a bundle
Installing on a disconnected target
Verifying a bundle
Bundle layout
How artifact mode stays offline
GPU support
Bundling Supply Depot apps
Easy Setup with no internet
Updating an air-gapped installation
Optional images and pre-staged content
Known limitations
Troubleshooting
Verified offline behaviour

What the bundle covers

A bundle contains everything needed to bring up the core NOMAD management stack — the Command Center and its supporting services — on a clean machine:

Docker Engine, CLI, containerd, Buildx and Compose v2, plus the host utilities the installer uses, as .deb packages with their dependencies
every container image referenced by install/management_compose.yaml
the management compose file, the start/stop/update helper scripts, and the uninstall script
the exact install_nomad.sh from the checkout the bundle was built from

Supply Depot applications can be added with --with-apps. Offline content — ZIM files beyond a small Wikipedia sample, maps, AI models, Kolibri channels — is fetched from remote catalogs at runtime and is not covered; see Known limitations.

Building a bundle

The build machine needs internet access and Docker — that's the whole list. The entire build runs in containers, so no git, bash version, or GNU coreutils is required on the host, and the machine does not need to resemble the target: the package closure is resolved inside a container of the target distribution, so any build machine produces the same bundle.

Linux, macOS, WSL2 and Windows Git Bash all work. On Windows the wrapper hands the daemon Windows-form mount sources and mounts them inside the build container at /run/desktop/mnt/host/<drive>/…, the path Docker Desktop's VM resolves to the same directory — the build starts further containers, and their bind mounts are resolved by the host daemon, so a path has to mean the same thing on both sides.

Docker Desktop shares fixed drives; removable and exFAT volumes usually read as an empty directory rather than failing, which would otherwise yield a bundle that builds, passes its own verification, and contains nothing.

The builder probes each directory up front and works around what it finds:

Checkout on an unshared drive — the working tree is copied to a mountable staging directory and built from there, .git and uncommitted changes included, so the manifest still records the right commit.
--output on an unshared drive — the bundle is built in staging and then copied across with ordinary file I/O, which removable media accepts perfectly well. Writing a bundle straight to the USB stick it will travel on is the normal case and is fully supported.
Neither mountable — one clear error naming the fix, before any work starts.

Set TMPDIR if the default staging location is itself on an unshared volume.

git clone https://github.com/Crosstalk-Solutions/project-nomad.git
cd project-nomad

./install/build_offline_bundle_docker.sh --target ubuntu:26.04

With no --output, the builder asks where to write the bundle, offering the directory you are running from, any connected removable drives (so you can write straight to the USB stick it will travel on), and your home directory. To skip the question, pass --output DIR, or --no-prompt to accept the default. Prompting is disabled automatically when stdin is not a terminal, so scripts and CI never hang.

build_offline_bundle_docker.sh is a thin wrapper: it mounts the checkout, the output directory and the Docker socket into a container and runs build_offline_bundle.sh there, passing every option straight through. Call build_offline_bundle.sh directly only if you are already inside a Linux environment with git and coreutils available.

This produces /media/usb/NOMAD/project-nomad-offline-ubuntu-26.04-amd64-<commit>/.

The dependency closure is resolved inside a pristine container of the target release — nothing is installed there first, because APT only downloads packages it does not already have, and a contaminated container silently produces an incomplete bundle. Before finishing, the builder re-runs the target's exact isolated APT installation in a clean container with --network none. If the bundle cannot satisfy its own package list offline, the build fails rather than handing you a bundle that breaks in the field.

Option	Purpose
`--repo PATH`	Source checkout to build from (default: parent of the script)
`--target OS:VERSION`	Target operating system (default: `ubuntu:26.04`)
`--arch ARCH`	Target architecture (default: `amd64`)
`--output DIR`	Where to create the bundle (default: `./dist`)
`--without-nvidia-toolkit`	Omit the NVIDIA Container Toolkit packages
`--extra-image-list FILE`	Also pull and bundle the image references in `FILE`
`--extra-image-archive FILE`	Include an existing `docker save` archive
`--with-apps LIST`	Bundle Supply Depot app images (`default`, `all`, or a list)
`--list-apps`	Print the installable app names and exit
`--use-local-images`	Bundle images already in the local Docker daemon instead of pulling them
`--content-dir DIR`	Include pre-staged NOMAD storage content
`--archive`	Also produce a `.tar.gz` of the finished bundle

Why the build runs in a container

Every path-valued option is resolved to an absolute host path and mounted into the build container at that same location. This matters because the build starts further containers, and their bind mounts are resolved by the host Docker daemon — a path that existed only inside the build container would silently fail to mount. The wrapper handles this for --output, --repo, --content-dir, --extra-image-archive and --extra-image-list.

Two environment variables are available if your setup is unusual:

Variable	Purpose
`NOMAD_BUILDER_IMAGE`	Build container image (default `docker:cli`)
`NOMAD_DOCKER_SOCKET`	Docker socket path (default `/var/run/docker.sock`)
`NOMAD_NO_PROMPT`	Set to `1` to never ask where to write the bundle

Building onto FAT/exFAT removable media

Writing a bundle straight to a USB stick is supported and expected. Note that macOS deposits AppleDouble sidecars (._name) and .DS_Store files on such filesystems. These are OS metadata rather than bundle content, so the builder removes them and excludes them from SHA256SUMS, and the installer ignores them when loading image archives. Without that, a ._core-images.tar sidecar would both break checksum verification and be handed to docker load.

Testing an unreleased Command Center

management_compose.yaml pins the published ghcr.io/crosstalk-solutions/project-nomad:latest, and the builder pulls it. A bundle therefore carries the released Command Center, not whatever is in your checkout — so a change to the admin application cannot be tested air-gapped until it ships in an image.

--use-local-images closes that loop. Build the image from your checkout, tag it as the reference the compose file uses, then build the bundle:

docker build -t ghcr.io/crosstalk-solutions/project-nomad:latest .
./install/build_offline_bundle_docker.sh --target ubuntu:26.04 --use-local-images

Any image already present in the daemon is taken as-is; anything missing is still pulled, so the flag can never quietly produce a bundle with a gap in it.

Bundles built this way record USED_LOCAL_IMAGES=1 in the manifest and carry a note in README.txt, because they may contain unreleased code and should not be distributed.

Bundles are specific

A bundle is valid for one operating system, one version, and one architecture. The package closure is resolved for that exact combination, so an Ubuntu 26.04 bundle will not install on Debian 12, and the installer refuses to try. Build one bundle per target platform.

The bundle records the git commit it was built from and carries that commit's installer, so the target runs the same code the bundle was built against.

Installing on a disconnected target

The target needs a clean, supported OS install and nothing else. In particular, do not install Docker first — it comes from the bundle:

Normal install	Offline install
Install a supported OS	Same
Connect to the internet	Not needed
`apt-get install curl`	Not needed — provided by the bundle
`curl … install_nomad.sh`	Not needed — the bundle carries the installer
Installer fetches Docker from `get.docker.com`	Docker installs from the bundle
Installer pulls images from GHCR / Docker Hub	Images load from the bundle
Installer downloads helper scripts from GitHub	Provided by the bundle

Copy the bundle directory to the target (USB drive, external disk, whatever moves bytes), then:

cd /media/usb/NOMAD/project-nomad-offline-ubuntu-26.04-amd64-<commit>
sudo bash ./install_nomad.sh --artifacts .

The environment variable form is equivalent:

sudo NOMAD_ARTIFACT_PATH=/media/usb/NOMAD/project-nomad-offline-... \
  bash ./install_nomad.sh

If both are supplied, the --artifacts argument wins.

Installation proceeds as usual from there — same prompts, same license acceptance, same result. When it finishes, the Command Center is available at http://localhost:8080, and at http://<device-ip>:8080 if the machine has a LAN address.

A completely disconnected machine may have no LAN address at all. That is fine: artifact mode falls back to localhost instead of failing, and simply doesn't advertise a LAN URL.

Verifying a bundle

Before transferring a bundle — or on the target before installing — check it:

./install/verify_offline_bundle.sh /media/usb/NOMAD/project-nomad-offline-...

This confirms every required file is present, verifies the checksums, and prints the manifest. It runs entirely offline and installs nothing.

The installer performs the same validation automatically before it touches the system.

Bundle layout

project-nomad-offline-<os>-<version>-<arch>-<commit>/
├── install_nomad.sh              # the installer from the source commit
├── manifest                      # bundle metadata (data, never sourced)
├── README.txt
├── SHA256SUMS                    # covers every other file in the bundle
├── packages/
│   └── apt/                      # flat local APT repository
│       ├── Packages
│       ├── Packages.gz
│       └── *.deb
├── images/
│   ├── core-images.txt           # image references in the bundle
│   ├── core-image-metadata.tsv   # image IDs and repo digests
│   ├── core-images.tar           # docker save archive
│   └── optional-images.tar       # optional
├── payload/
│   └── nomad/
│       ├── management_compose.yaml
│       ├── compose.artifact.yml  # generated pull_policy: never override
│       ├── start_nomad.sh
│       ├── stop_nomad.sh
│       ├── update_nomad.sh
│       └── uninstall_nomad.sh
└── content/                      # optional pre-staged storage content

The manifest

BUNDLE_FORMAT_VERSION=1
NOMAD_COMMIT=<full git sha>
TARGET_OS=ubuntu
TARGET_VERSION=26.04
TARGET_ARCH=amd64
WITH_NVIDIA_TOOLKIT=1
USED_LOCAL_IMAGES=0
CREATED_AT_UTC=2026-08-14T16:00:00Z

The manifest is plain data. The installer reads individual keys from it and never executes it as shell.

Checksums

SHA256SUMS covers every regular file in the bundle except itself, and is verified before any package is installed or image loaded.

This proves the bundle transferred intact. It is not a publisher signature and does not establish who produced the bundle — verify provenance separately if that matters to you.

How artifact mode stays offline

Artifact mode is fail-closed. Once an artifact path is given, the installer never falls back to the network to fill a gap. A missing package, image, payload file, manifest field or checksum stops the installation with an actionable error.

Concretely, in artifact mode the installer does not use curl or wget to acquire anything, does not run docker pull, and does not contact GitHub, GHCR, Docker Hub, the Docker convenience script, or NVIDIA's package repository.

Host packages are installed from the bundle's flat APT repository through a temporary source that excludes the machine's own configuration:

deb [trusted=yes] file:/tmp/nomad-artifact-apt.XXXXXX/repo ./

APT is invoked with Dir::Etc::sourcelist pointing at that file, Dir::Etc::sourceparts=- to exclude /etc/apt/sources.list.d, Dir::State::Lists pointing at a private directory so the host's package lists are left alone, APT::Get::List-Cleanup=0, and Acquire::Retries=0. The machine's configured remote repositories take no part in dependency resolution, which is why a missing dependency fails instead of quietly downloading.

Container images are loaded with docker load. Because management_compose.yaml legitimately uses pull_policy: always for online installs, the builder generates compose.artifact.yml setting pull_policy: never for every service. Artifact mode layers both files:

docker compose -p project-nomad \
  -f /opt/project-nomad/compose.yml \
  -f /opt/project-nomad/compose.artifact.yml \
  up -d --pull never

The canonical compose file is never rewritten just to make an offline install work.

A practical consequence: a bundle tested on a machine that happens to be online behaves identically once the cable is pulled. If it works connected, it works disconnected.

GPU support

Artifact mode keeps the installer's existing non-blocking GPU behaviour — GPU problems warn, they never fail the install.

NVIDIA hardware is detected with the same logic as an online install.
If the bundle includes nvidia-container-toolkit (the default; disable with --without-nvidia-toolkit) and a working host NVIDIA driver is present, the Docker runtime is configured locally.
If the host driver is missing, you get a warning and the install continues.
NVIDIA's package repository is never contacted.

Host NVIDIA drivers are out of scope for this version of the bundle format. Bundling kernel drivers means matching kernel versions and handling DKMS, which is a materially larger problem. Install the driver on the target separately if you need GPU acceleration offline.

AMD GPU detection and the marker files the Command Center reads are unchanged.

Bundling Supply Depot apps

A core bundle gets you the Command Center, but the Supply Depot will be unable to install anything without a network. --with-apps carries the app images too:

./install/build_offline_bundle_docker.sh \
  --target ubuntu:26.04 \
  --with-apps default

Value	Meaning
`default`	A useful starter set: Kiwix, CyberChef, IT-Tools, FlatNotes, Excalidraw, File Browser, Stirling PDF
`all`	Every app in the catalog (large — includes Ollama, Kolibri, Jellyfin)
`kiwix,cyberchef,…`	An explicit comma-separated list

Run ./install/build_offline_bundle.sh --list-apps to see the available names. An unrecognised name fails the build rather than quietly producing a bundle without the app you asked for.

This works because the app catalog is not fetched from the internet: it is seeded into the Command Center's database from a file baked into the admin image, with pinned image tags. The installer then skips the registry pull for any image already present locally. Loading the images from a bundle is therefore enough for the Supply Depot to install those apps offline.

The image references are read from that same seeder at build time, so a bundle cannot drift from the catalog the UI will offer.

Apps with extra requirements

Most apps need nothing beyond their image. Two exceptions are worth knowing:

Kiwix refuses to start without at least one ZIM file, and its pre-install step fetches a sample from GitHub. When Kiwix is selected, the builder includes the small Wikipedia sample from the checkout as pre-staged content, and the pre-install step skips the download when storage already holds a ZIM.

That skip is a change to the admin application, so it only takes effect once it ships in a released ghcr.io/crosstalk-solutions/project-nomad image. Against an older admin image, installing Kiwix on an air-gapped host fails with getaddrinfo EAI_AGAIN github.com even when the ZIM is already in storage, because the downloader's idempotency check sits behind a HEAD request.
The AI Assistant (Ollama) ships as an image, but models are downloaded separately from the Ollama registry at first use. Bundling the image does not make model downloads work offline.

On a host with an AMD GPU the installer switches Ollama to the ROCm image (ollama/ollama:rocm), which --with-apps does not carry — it bundles the catalog's pinned tag. An air-gapped AMD host therefore needs that tag added explicitly with --extra-image-list, or AMD acceleration turned off (KV key ai.amdGpuAcceleration set to false) so the bundled CPU image is used.

Content that is fetched from remote catalogs at runtime — ZIM files beyond the bundled sample, maps, Kolibri channels, AI models — is not covered. Use --content-dir to pre-stage such files if you already have them in NOMAD's storage layout.

Easy Setup with no internet

The Command Center's Easy Setup wizard is the first thing an operator sees after an install, and it works air-gapped — for the part of it that can.

Apps install offline when their image is already on the machine. The wizard asks Docker which images are present and offers exactly those. An app whose image came from a --with-apps bundle is selectable and installs without touching a registry; one that would need a pull is shown greyed out with a Needs internet badge, so nothing fails halfway through.

Content downloads stay unavailable. Map regions, curated content tiers, creator packs, AI models and Wikipedia all come from remote catalogs. Offline, those steps explain themselves and their cards can't be selected. You can walk through every step, skip them, and finish the wizard with just the apps.

If the wizard is offline and no app images are on the machine, it says so plainly rather than offering a set of installs that would all fail. That is the signal to re-run the installer against a bundle built with --with-apps.

Two consequences of a truly air-gapped first boot are worth knowing:

The wizard shows no Wikipedia selector. The catalog that populates it is fetched from GitHub and cached; a machine that has never been online has no cached copy, so the section is omitted. It appears on the Content step (and in the ZIM manager) the first time the machine reaches the internet.
The Kiwix app still needs at least one ZIM to start. A bundle that includes Kiwix pre-stages the small Wikipedia sample for exactly this reason.

Updating an air-gapped installation

Re-running the installer with a newer bundle updates in place. This is the supported offline update path — build a new bundle on a connected machine, carry it over, and run the same command:

sudo bash ./install_nomad.sh --artifacts .

Re-running is safe and repeatable. When an existing installation is detected the installer:

keeps the database, installed apps, settings and content;
reuses the credentials the database was initialised with, rather than generating new ones (MySQL only applies those on first startup, so regenerating is precisely what would force wiping the data directory);
preserves the URL the instance is already reachable at;
loads any new or updated images from the bundle and recreates the changed containers;
adds new pre-staged content without overwriting files you have changed.

Use it to move to a newer NOMAD version, or simply to add apps: rebuild with a wider --with-apps set and re-run.

Note that update_nomad.sh — the online update path — still pulls from a registry and will not work air-gapped. Use a new bundle instead.

Optional images and pre-staged content

Two further extension points exist. Both are transport mechanisms, not guarantees.

Additional images

On a machine with NOMAD already installed and apps set up:

./install/capture_installed_images.sh --output /tmp/nomad-extra-images

Then include the archive:

./install/build_offline_bundle.sh \
  --target ubuntu:26.04 \
  --extra-image-archive /tmp/nomad-extra-images/optional-images.tar \
  --output /media/usb/NOMAD

This moves the Docker images to the target. It does not by itself make those applications installable offline — Supply Depot install flows and catalog metadata may have their own network dependencies. Test any app you care about individually before relying on it.

Pre-staged content

./install/build_offline_bundle.sh \
  --target ubuntu:26.04 \
  --content-dir /srv/nomad-storage-seed \
  --output /media/usb/NOMAD

The directory is copied into /opt/project-nomad/storage before the stack starts. It must already match NOMAD's storage layout.

Known limitations

Be precise about what offline artifact mode does and does not claim:

Apps need --with-apps; content is mostly not covered. A core bundle installs the Command Center only. App images can be bundled, but offline content (ZIMs beyond the bundled sample, maps, AI models, Kolibri channels) is fetched from remote catalogs at runtime and is not covered.
update_nomad.sh still needs the internet. Update an air-gapped install by building a new bundle and re-running the installer against it.
Host NVIDIA drivers are not bundled. See GPU support.
Checksums are integrity, not provenance. See Checksums.
One bundle, one platform. OS, version and architecture must match.
x86_64 only, matching the installer's supported architecture.
The Command Center still probes for connectivity at runtime. It shows an offline status and falls back to cached or bundled data rather than failing, but those requests are attempted.
Easy Setup offline installs apps, not content. See Easy Setup with no internet. Only apps whose images are already on the machine are offered; every download-backed step is disabled with an explanation.

Troubleshooting

"This bundle targets ubuntu 26.04, but this host runs 24.04." Bundles carry a package set resolved for one exact OS version. Build a bundle for the target's version.

"The offline artifact bundle failed checksum verification." The copy is corrupt or incomplete — a common outcome of pulling a USB drive early. Copy it again and re-run verify_offline_bundle.sh.

"Failed to install host dependencies from the offline artifact bundle." The bundle is missing a package this machine needs. The builder resolves the dependency closure inside a pristine base image of the target release and verifies offline that it resolves, so this normally means the target is more minimal than a stock install of that release. Note which package APT names and rebuild the bundle for the target's exact OS version.

"The bundle lists image X but it is not present after loading." The image archive is incomplete. Rebuild the bundle on a connected machine.

The installer says it cannot find the bundle. Pass a path to the bundle directory (the one containing manifest), and remember sudo — the path must be readable by root. Mount points containing spaces are handled.

Verified offline behaviour

What has actually been exercised on a disconnected Ubuntu 26.04 host, with all egress denied:

Step	Result
Fresh install from a core bundle	Works — all six management containers healthy
Reboot with egress still denied	Works — stack returns on its own
Docker installed from the bundle	Works — no repository added to the host
Re-running with a newer bundle	Works — database, apps, settings and content preserved
Installing CyberChef / IT-Tools offline	Works — installed and serving
Installing Kiwix offline	Requires the pre-install fix above in the admin image
Downloading content (ZIMs, maps, models)	Not supported offline by design

26 KiB Raw Blame History