Proxmox¶

Everything runs on two Proxmox VE hosts:

• Services host. The NAS, the Docker host and the workstation are VMs, and small services such as the UniFi controller are LXC containers.
• Backup host. Separate hardware with a backup NAS VM, Proxmox Backup Server and the second DNS server (AdGuard Home + Unbound) as LXCs.

Keeping backups on their own machine means a failed disk controller, a bad Proxmox upgrade or a mistake on the services host can't take the backups down with it. Backups themselves are covered in Files → Backups.

VM or LXC?¶

My rule: one service and no special hardware → LXC; everything else → VM.

For LXCs, the Proxmox VE Helper-Scripts are a quick start. They're one-line installers that create a ready-made, unprivileged container for a given app. Read the script before piping it into a root shell.

GPU passthrough¶

The Docker host VM gets the whole NVIDIA GPU, which is used by Immich, Frigate, Whisper and Ollama. The steps on an AMD host:

1. Turn on IOMMU (also called AMD-Vi or VT-d) in the BIOS, then on the kernel command line. Proxmox with LVM boots via GRUB: edit /etc/default/grub, run update-grub and reboot.

   /etc/default/grub

   GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt amd_iommu=on"
   

   For Intel, use intel_iommu=on. iommu=pt only maps devices that are passed through, which is faster for the host.

2. Load VFIO and keep the host's own drivers off the GPU:

   /etc/modules

   vfio
   vfio_iommu_type1
   vfio_pci
   

   /etc/modprobe.d/blacklist.conf

   blacklist nouveau
   blacklist nvidia*
   blacklist nvidiafb
   

   Then update-initramfs -u -k all and reboot.

3. Check the IOMMU groups. A device can only be passed through together with everything else in its group:

   iommu.sh

   #!/bin/bash
   for d in /sys/kernel/iommu_groups/*/devices/*; do
     n=${d#*/iommu_groups/*}; n=${n%%/*}
     printf 'IOMMU Group %s ' "$n"
     lspci -nns "${d##*/}"
   done
   

   The GPU and its audio function should be alone in a group. If they share it with other devices, pcie_acs_override=downstream on the kernel command line splits the groups. It's a workaround that weakens isolation between those devices, so only add it if you need to. My board needs it.

4. Configure the VM. Use machine type q35, OVMF (UEFI) BIOS and CPU type host. Then add the GPU under Hardware → Add → PCI Device with All Functions and PCI-Express ticked:

   qm config <vmid> (excerpt)

   machine: q35
   bios: ovmf
   cpu: host
   balloon: 0
   hostpci0: 0000:81:00,pcie=1
   

   Turn memory ballooning off (balloon: 0). A VM with a passed-through device has to keep all its RAM pinned anyway.

5. Inside the VM, install the normal NVIDIA driver and the NVIDIA Container Toolkit so Docker containers can use the GPU.

Passing disks through¶

The NAS and the Docker host get whole physical disks. Pass them by their stable /dev/disk/by-id/ name, never /dev/sdX, because those letters can change between boots:

ls -l /dev/disk/by-id/ | grep -v part
qm set <vmid> -scsi1 /dev/disk/by-id/ata-<model>_<serial>,backup=0,iothread=1

backup=0 keeps the disk out of the nightly vzdump job, since its data is backed up file by file with restic instead (see Backups). Add discard=on for SSDs.

The alternative is passing through a whole SATA or HBA controller as a PCI device. The VM then sees the disks directly, including SMART data and spin-down. That's worth doing if a NAS VM needs proper disk health monitoring.

Small things worth doing¶

• qemu-guest-agent in every VM (apt install qemu-guest-agent, then tick QEMU Guest Agent under Options). Proxmox can then shut the VM down cleanly, show its IP, and freeze its filesystems for consistent snapshot backups.
• iothread=1 with the VirtIO SCSI single controller gives each disk its own I/O thread.
• Start order under Options → Start/Shutdown order: start the NAS first and the Docker host a minute later, so the SMB mounts are there when containers start.