ZFS on NixOS

2023-07-24, 2023-12-17, 2024-01-06

I’ve recently set up a new home server to act as a NAS and fulfil other misc responsibilities. It has 4x6TB HDDs in a raidz configuration. Here’s how I got ZFS set up on NixOS.

Install NixOS and enable ZFS

Install NixOS as usual, then add ZFS support to your configuration.nix.

# configuration.nix

boot.supportedFilesystems = [ "zfs" ];
boot.zfs.forceImportRoot = false;
networking.hostId = "6d778fb4"; # head -c4 /dev/urandom | od -A none -t x4

Create ZFS pool and datasets

Once rebuilt, find the names of the drives you want to add to your pool.

$ ls /dev/disk/by-id

 ata-ST6000VN001-2BB186_ZRxxxxxx
 ata-ST6000VN001-2BB186_ZRxxxxxx
 ata-TOSHIBA_HDWG460_32xxxxxxxxxx
 ata-TOSHIBA_HDWG460_33xxxxxxxxxx

Now create the pool with zpool create. The ArchWiki says to always set ashift=12 so let’s just do that.

$ zpool create -f -o ashift=12 -m /mnt/<poolname> <poolname> \
    raidz \
    ata-ST6000VN001-2BB186_ZRxxxxxx \
    ata-ST6000VN001-2BB186_ZRxxxxxx \
    ata-TOSHIBA_HDWG460_32xxxxxxxxxx \
    ata-TOSHIBA_HDWG460_33xxxxxxxxxx

I’m chosing ’taskpool’ for the pool name as the machine is called Taskmaster

Then create the datasets you want. For now, I’m just creating one for various services running on the server. I’ll add more as I go.

$ zfs create poolname/services

The pool and dataset you created can be seen under the /mnt directory

$ ls /mnt

To ensure the pool and datasets are mounted at boot you need to add the required lines to your hardware-configuration.nix. You can auto generate a config with:

$ nixos-generate-config  # Add --show-hardware-config if you're not using
                         # /etc/nixos/hardware-configuration.nix

Or you can manually add the lines yourself

# hardware-configuration.nix

fileSystems."/mnt/taskpool" =
  {
    device = "taskpool";
    fsType = "zfs";
  };

fileSystems."/mnt/taskpool/services" =
  {
    device = "taskpool/services";
    fsType = "zfs";
  };

Before rebuilding the config you’ll need to set the mountpoint to legacy

$ zfs set mountpoint=legacy <poolname>

If you don’t set mountpoint=legacy you’ll likely see the following error and get dropped in to emergency recovery mode on the next boot.

A dependency job for local-fs.target failed. See 'journalctl -xe' for
details. Job for sysinit.target canceled

Getting out of recovery mode doesn’t seem to be a problem, just run that command and then reboot. All should be well again.

Encrypted datasets

One of the server’s jobs will be to store all of our photos using something like Immich. This data is quite personal, so I want the supporting datasets to be encrypted.

An encrypted dataset needs encryption keys. These keys are created by ZFS but are encrypted with a wrapping key that you supply. Generally this can be supplied interactively or from a file. As I want the dataset to be mounted at boot time, I’ll use the file method. I couldn’t decide where best to put the file, so I’m going to let Nix handle it as an agenix secret.

Creating the wrapping key file

I followed LGUG2Z’ instructions for getting agenix set up and secret encrypted (see there for the full config).

First, define the secret file and the keys that can read it.

# secrets/secrets.nix

let
  personal_key = "ssh-rsa AAA...";  # my public key (contents of ~/.ssh/id_rsa.pub)
  taskmaster_key = "ssh-ed25519 AAAA...";  # ed25519 public key for the server (found from `ssh-keyscan taskmaster`)
  keys = [ personal_key taskmaster_key ];
in {
  "zfs.key.age".publicKeys = keys;
}

Then generate the encrypted file.

$ cd secrets
$ nix run github:ryantm/agenix -- -e zfs.key.age

This will open an editor where you can enter the passphrase you want to use to unlock the keys. It’s a good idea to save this in your password manager. After saving and exiting the editor the encrypted zfs.key.age file will be created for you to commit.

After wiring in agenix to your Nix config, you can make sure an unencrpyted version of the file exists on the machine.

$ git add secrets/zfs.key.age  # agenix/nix expects this to be tracked by git

# configuration.nix

age.secrets."zfs.key".file = "./secrets/zfs.key.age";

Once rebuilt, you can see this as root at /run/agenix/zfs.key

Creating the encrypted dataset

Creating an encrypted pool is not much different to unencrypted, it just needs a few extra properties set.

$ zfs create -o encryption=on -o keyformat=passphrase -o keylocation=file:///run/agenix/zfs.key taskpool/photos

This uses the default encryption which is currently aes-256-gcm. Once the dataset is created, you can then mount it via your Nix config as before.

# hardware-configuration.nix

fileSystems."/mnt/taskpool/photos" =
  {
    device = "taskpool/photos";
    fsType = "zfs";
  };

Maintaining pool health

As noted in the Nix docs it’s recommended to regularly scrub the pools. Googling around suggested doing it weekly for consumer grade disks, or monthly for enterprise. The following will run it on the 1st and 15th of the month (local time).

services.zfs.autoScrub = {
  enable = true;
  interval = "*-*-1,15 02:30";
};

Snapshots and backups

I’m going to use Sanoid to create local snapshots for recovery when things get delete/corrupted but disks remain intact. I’ll then use Syncoid to push snapshots to somewhere like zfs.rent or rsync.net for offsite backup.

Configuring regular snapshots is very simple. Create a template for frequency and retention, then apply that to the relative pools.

services.sanoid = {
  enable = true;
  templates.backup = {
    hourly = 36;
    daily = 30;
    monthly = 3;
    autoprune = true;
    autosnap = true;
  };

  datasets."taskpool/services" = {
    useTemplate = [ "backup" ];
  };
};

One applied this will run on the hour and create the first/hourly/monthly snapshots, then auto delete the older ones as time goes on. You can view all current snapshot.

$ zfs list -t snapshot

return 12; // good enough for now