--- name: nvidia-datacenter-bringup description: Bring up NVIDIA HGX/DGX datacenter GPU hosts on Ubuntu 24.04 LTS — air-gapped or connected, Secure Boot enabled. Covers B300/B200/H100/A100/L40S/L4 driver+fabricmanager+NVLSM+DOCA-OFED install order and exact package set from NVIDIA CUDA repo + DOCA repo. Triggers on B300/B200/HGX/DGX install, "fabricmanager won't start", "system not yet initialized" / cudaErrorSystemNotReady, NVLSM missing, ib_umad not loading, DOCA-OFED before NVIDIA driver, nvidia-driver-pinning-XXX, nvlink5-XXX, nvidia-open vs cuda-drivers, "Blackwell requires open kernel modules", ConnectX-7/8 bridge device, FM exact-version-match, gpu-operator cuda-validator CrashLoopBackOff, B300 PCI ID 0x3182, air-gap CUDA + DOCA mirror, three-tier DOCA GPG key, MOK enrollment, DKMS sign, Dell PowerEdge XE9780/XE9785 baseboard firmware v1.4.30, iDRAC Redfish virtual AC cycle DellOemChassis.ExtendedReset, generic "install nvidia driver ubuntu 24.04 datacenter". when_to_use: Use for bringing up an NVIDIA datacenter GPU host (HGX, DGX, or inference-card servers) on Ubuntu 24.04 LTS. Covers the full air-gapped path from clean OS install to the gpu-operator cuda-validator pod passing. Trigger when fabricmanager fails to start, nvidia-smi sees no GPUs on B200/B300, the cuda-validator loops with "system not yet initialized", a host needs coherent DOCA-OFED + NVLSM + NVIDIA driver install, MOK enrollment for Secure Boot is needed, or a Dell XE9780/XE9785 chassis needs baseboard firmware updated. --- # nvidia-datacenter-bringup Opinionated greenfield recipe for **NVIDIA datacenter GPUs on Ubuntu 24.04 LTS** — get from a clean OS install to a healthy host where `nvidia-smi` reports all GPUs, `nvidia-fabricmanager` is `active (running)`, and the gpu-operator `cuda-validator` pod passes. Air-gap is the primary case; connected sites use the same packages from the same upstream URLs. ## Decision tree | Question | Answer | Read | |---|---|---| | Has Blackwell silicon (B300/B200/B100)? | Yes | Open kernel modules **mandatory** — proprietary is unsupported [[open-modules-transition]] | | Grace Hopper (GH200)? | Yes | Open kernel modules **mandatory** (same as Blackwell). Otherwise 8-GPU SXM path [[hopper-recipe]] | | HGX 8-GPU SXM with 3rd-gen NVSwitch (H100/H200/H800 in XE9680 or similar)? | Yes | Open recommended (not mandatory). Use `cuda-drivers-fabricmanager-` meta; **skip** `nvlink5-`, NVLSM, DOCA-OFED entirely. Min driver 525+ for H100, 535+ for H200 [[hopper-recipe]] | | HGX 4-GPU SXM (H100 in XE8640, A100 4-GPU)? | Yes | **No NVSwitch on this baseboard** — direct NVLink mesh between 4 GPUs. **Skip fabricmanager entirely** + DOCA + NVLSM. Three-package install: driver + container-toolkit [[hopper-recipe]] | | HGX A100 8-GPU (2nd-gen NVSwitch)? | Yes | Same as 8-GPU SXM H100 path. Min driver 450.xx. ALI not available — FM trains NVLinks at boot | | L40S, L40, L4, H200 NVL, or other PCIe-only? | Yes | Driver + container-toolkit only. **Skip** DOCA-OFED, fabricmanager, NVLSM entirely. Validation reduces to `nvidia-smi` clean — no fabric registration to check | | 4th-gen NVSwitch (B200/B300/B100)? | Yes | Must install `nvlink5-` meta (or `nvlsm + libnvsdm + libibumad3 + infiniband-diags` separately). FM talks to switches via CX7 bridge — DOCA-OFED is non-optional [[fabric-manager-guide]] §"Additional Steps for B200/B300" | | Dell XE9780 or XE9785 chassis? | Yes | **Step 0 is firmware ≥ v1.4.30**. Use Redfish `DellOemChassis.ExtendedReset`, NOT BIOS Full Power Cycle [[dell-firmware]] | | Dell XE9680 / XE9640 / XE8640? | Yes | Enable **iDRAC Direct USB Port in BIOS** before GPU baseboard FW updates (KB 000308105). Same `DellOemChassis.ExtendedReset` activation pattern [[dell-firmware]] | | Secure Boot enabled? | Yes | Enroll one DKMS MOK — signs both NVIDIA and DOCA-OFED modules [[secure-boot]] | | Air-gapped? | Yes | Mirror three repos. Total budget ~3 GB. Three-tier GPG keys for DOCA [[airgap-mirror]] | | gpu-operator on top? | Yes | Pre-installed driver mode: `driver.enabled=false`, `toolkit.enabled=false`. Know about B300 issue #2231 [[gpu-operator]] | ## The 10-step install order Order matters: DOCA before driver, MOK before any DKMS build, firmware before everything. ``` 0. Dell baseboard firmware ≥ v1.4.30 + Redfish virtual AC cycle [[dell-firmware]] 1. OS prep: kernel headers, blacklist nouveau [[recipe]] 2. Secure Boot: generate + import MOK (once, signs both stacks) [[secure-boot]] 3. Add repos: NVIDIA CUDA + DOCA (+ Ubuntu archive subset) [[airgap-mirror]] 4. Install DOCA — modules-load.d generated, ib_umad autoloads [[recipe]] 5. Pin NVIDIA driver branch: apt install nvidia-driver-pinning-580 [[packages]] 6. Install driver: apt install nvidia-open-580 (open is mandatory) [[packages]] 7. Install fabric stack: apt install nvlink5-580 [[packages]] 8. Install nvidia-container-toolkit (lives in same CUDA repo) [[packages]] 9. Validate: nvidia-smi clean, FM active, ib_umad loaded, Fabric Completed/Success ``` `nvlink5-580` is the **compute-only** fabric meta — pulls `nvidia-fabricmanager + nvlsm + libnvsdm + libnvidia-nscq + nvidia-imex + collectx-bringup + mft{,-oem,-autocomplete} + nvidia-dkms-580-open + libnvidia-compute-580` (and transitively `libibumad3` + `infiniband-diags`). It does NOT pull `nvidia-utils-580` / `nvidia-smi` — that's why step 6 installs `nvidia-open-580` (full userland) AS WELL. Dependency tree verified live from the NVIDIA CUDA repo for Ubuntu 24.04 amd64 — see [[packages]]. ## Pitfalls catalogue The mistakes that cost hours, ordered by frequency in the wild: 1. **Proprietary modules on Blackwell.** `nvidia-driver-580-server` (no `-open`) installs cleanly but the kernel module silently doesn't bind to B300 silicon. `nvidia-smi` reports no GPUs. NVIDIA's open-modules transition blog: proprietary is **unsupported** on Blackwell and Grace Hopper. Cure: use `nvidia-open-` from the CUDA repo. See [[open-modules-transition]]. 2. **DOCA installed AFTER the driver.** `ib_umad` does not autoload at boot; `nvidia-fabricmanager.service` fails. NVIDIA forum 353369. Cure: install DOCA first. If already broken, reinstall DOCA OR `echo ib_umad | sudo tee /etc/modules-load.d/ib_umad.conf && sudo modprobe ib_umad && sudo systemctl restart nvidia-fabricmanager`. See [[troubleshooting]]. 3. **Dell BIOS Full Power Cycle.** The iDRAC web-UI "Full Power Cycle" and the standard Redfish `Chassis.Reset` `FullPowerCycle` do **not** reliably activate the GPU baseboard subcomponents (HMC, NVSwitch, CX bridge firmware). Symptom: firmware update "succeeded" but old version still reports. Cure: Dell-specific `DellOemChassis.ExtendedReset`. See [[dell-firmware]]. 4. **Mixing Ubuntu archive `nvidia-driver-XXX-server` with CUDA-repo `nvlsm`.** Ubuntu archive's FM version lags NVIDIA's by patches; NVLSM is only in CUDA repo. Rip out Ubuntu archive NVIDIA packages and go single-source CUDA repo. See [[packages]]. 5. **MOK enrolled for NVIDIA but not for DOCA-OFED (or vice versa).** Both stacks are DKMS-built. One `/var/lib/dkms/mok.pub` signs both — enroll once, covers everything. There is no Canonical-signed precompiled HWE module for ConnectX, so MOK is unavoidable on B300 + Secure Boot. See [[secure-boot]]. 6. **`cudaErrorSystemNotReady` blamed on driver.** This is the textbook FM-not-running signature on any NVSwitch system. The fix is always: confirm `nvidia-fabricmanager.service` is `active (running)` and its version matches the loaded driver. See [[troubleshooting]] and [[gpu-operator]]. 7. **gpu-operator B300 PCI ID warning loop (#2231).** Cosmetic, not fatal. `nvidia-operator-validator` logs `unable to get device name: failed to find device with id '3182'`. Open in upstream since 2026-03-18; NVIDIA acknowledged "B300 should be supported" but waiting on must-gather. Symptom is benign — pods still work. See [[gpu-operator]]. 8. **Driver 570.158.01 + operator-managed driver mode.** FM start script broken (gpu-operator #1595). Cure: pin ≥570.172.08 OR move to 580+. Or just use pre-installed-driver mode and let the host's systemd unit own FM. See [[gpu-operator]]. ## Validation After step 9 these should all be true. Expected output on a healthy 8-GPU B300: ```bash $ nvidia-smi --query-gpu=name,driver_version --format=csv,noheader NVIDIA B300 SXM6 270GB, 580.126.20 NVIDIA B300 SXM6 270GB, 580.126.20 # ... 8 identical lines $ systemctl is-active nvidia-persistenced nvidia-fabricmanager nvidia-nvlsm active active active $ lsmod | grep -E '^nvidia|^nvidia_uvm|^nvidia_peermem|^ib_umad' | awk '{print $1}' nvidia_peermem nvidia_uvm nvidia ib_umad $ nvidia-smi -q -i 0 | grep -A 2 Fabric Fabric State : Completed Status : Success $ nvidia-smi nvlink --status -i 0 | head -3 GPU 0: NVIDIA B300 SXM6 270GB (UUID: ...) Link 0: 200 GB/s Link 1: 200 GB/s ``` If any value is missing or wrong (e.g. `State: Not Initialized`, `lsmod` missing `ib_umad`, `systemctl is-active` reports `inactive`), see [[troubleshooting]] keyed by the symptom. On a fresh 8-GPU B300, FM takes ~30–90 s to finish fabric registration on first boot. The gpu-operator validator pod may CrashLoopBackoff transiently during this window — only sustained crashes >3 min are real failures. ## Hard out-of-scope - Windows, workstation/Optimus, non-Ubuntu (RHEL/SLES handled by NVIDIA's own guides) - vGPU / MIG runtime config (separate concern from bring-up) - DGX OS (NVIDIA's preinstalled bundle — different workflow, see DGX OS docs) - gpu-operator helm-values for full cluster config (host-side stops at validator passing) - Multi-node NVLink Switch (NVL72-style); single-chassis NVL8 only - Data-plane CX-8 NIC config (bridge role only — those are the management PFs) - Vendor firmware for non-Dell chassis (Supermicro/HPE/Lenovo — defer to vendor docs) ## References | Topic | File | |---|---| | Full install recipe with commands | [[recipe]] | | Exact apt package matrix (verified live) | [[packages]] | | Air-gap mirror setup + GPG keys + sizing | [[airgap-mirror]] | | Secure Boot + MOK + DKMS pipeline | [[secure-boot]] | | Dell baseboard firmware + Redfish AC cycle | [[dell-firmware]] | | gpu-operator host-contract + cuda-validator triage | [[gpu-operator]] | | Symptom → cause → fix playbook | [[troubleshooting]] | | Dated URL index | [[sources]] | | Open follow-ups / ceiling findings | [[improvement-backlog]] | | NVIDIA Fabric Manager User Guide (offline copy) | [[fabric-manager-guide]] | | NVIDIA Driver Install Guide — Ubuntu chapter | [[driver-install-ubuntu]] | | NVIDIA Driver Install Guide — Kernel modules | [[driver-install-kernel-modules]] | | NVIDIA Driver Install Guide — Advanced Options | [[driver-install-advanced-options]] | | DOCA Host Installation Guide — Ubuntu 24.04 | [[doca-install-ubuntu]] | | NVIDIA blog: open kernel modules transition | [[open-modules-transition]] | | Dell HGX B300 SXM6 firmware release notes (v1.4.30) | [[b300-firmware-release-notes]] | | Hopper recipe — H100 / H200 / GH200 simplified install paths | [[hopper-recipe]] | | One-shot host health check (`bash`) | `scripts/health-check.sh` |