bhankas

Deploying NixOS to Raspberry Pi from laptop

There's a little Raspberry Pi 4B ticking along in my home, quietly reducing adverts within my network, backing up important stuff, serving my music collection to guests and generating pretty graphs while at it.

Thanks to NixOS and its amazing options, the machine checks for updated at 4AM every day, and if I have made any changes to its config, including updating to newer version of NixOS, it proceeds to build the system with latest changes. Then, if the updates require, and only if the updates require, it reboots the system. On average, the whole process 4-10 minutes.

So far so good, this machinery has kept the system going strong for more than a year, and I've barely kept any checks on it. NixOS simply refuses to build if there are any update failures, It Just Works™.

But, I'm a tinkerer at heart, and when I'm mucking about with my Pi, say for example trying to get Zabbix running1, I want my Pi to build and deploy my config changes instantly, and not wait for next 4 AM. And I want to do this withoug SSHing in, because reasons.

You'd think it must be easy with NixOS, and you'd be right, of course.

Almost.

You see, Nix, in its ever-glorious genius, has no 'official' mechanism for doing this kind of DevOps2. There are bunch of these thingies sprouted like mushrooms after rain. So challenge #1 is actually selecting one that fits my needs. Thankfully, Solene has already done the survey work! I won't go into details, but for me it came down to Colmena and deploy-rs. I chose to go with the latter for few reasons:

  1. It reuses the already defined nixOsConfigurations attribute from my flake
  2. It does not need a separate hive.nix or anything, just little modification to existing flake.nix
  3. Its written in Rust3 :p

Colmena does have an edge for deploying secrets outside the nix-store, but I can live with that for now.

With that said and done, I set about the mission of deploying to my Pi with a single command4. There were a few lessons along the way:

  1. SSH key used by deploy-rs MUST NOT be password protected. It will keep on rolling forever, never being able to complete.
  2. If the CPU architecture of target system(aarch64, in this case) differs from source(x86_64, my laptop, because not everyone has monies for fancy fruits), then the source machine must have binfmt emulation enabled:

    1boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
  3. Thats it!

After all, the entire config ended up pretty small:

 1{
 2  inputs = {
 3    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
 4    deploy-rs.url = "github:serokell/deploy-rs";
 5  };
 6
 7  outputs =
 8    { self, nixpkgs, deploy-rs, ... }:
 9    {
10      nixosConfigurations = {
11        # My Pi
12        pi = nixpkgs.lib.nixosSystem {
13          system = "aarch64-linux";
14        };
15      };
16
17      deploy.nodes = {
18        pi = {
19          hostname = "pi";
20          fastConnection = true;
21          remoteBuild = true;
22          profiles.system = {
23            sshUser = "bhankas";
24            user = "root";
25            path = deploy-rs.lib.aarch64-linux.activate.nixos self.nixosConfigurations.pi;
26          };
27        };
28      };
29
30      # This is highly advised, and will prevent many possible mistakes
31      checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks self.deploy) deploy-rs.lib;
32    };
33}

Mind the 'aarch64-linux' at deploy.nodes.pi.profiles.system.path, it was easy to miss, and quite crucial

The checks thing at the bottom is pretty nifty too. If the deployment and/or subsequent activation fails for some reason, then the system is automatically rolled back to previously known working state. Pretty nice :)


1

Even though I already have a nice looking SystemD+Prometheus+Grafana already running :p

2

nixops gets close, but oh my gawd I'm not using that thing. Yet.

3

Colmena is also written in Rust, but its slightly more complex than deploy-rs to configure.

4

Yes, I know, nixos-rebuild can do the same, and the syntax is not that complex, and I have used it in past and it works. But then, I'd have to admit on wasting a Sunday and playing with cool toys is what this whole thing is about.