Personal Infrastructure as code
This repo demonstrates my use of the Infrastructure as Code philosophy using NixOS, Ansible, and Chocolatey.
Clone the repo and run direnv allow to load the environment.
I used to use an UnRaid server at home, that I use as Network Attached Storage (NAS) server. It also run many docker containers, and several virtual machines (Home Assistant). It became a server that I had confidence in, but couldn't recreate or adjust easily. I've since moved to a NixOS server, and use Ansible to manage it.
Why Ansible? I've used Ansible for a few years now, and I'm comfortable with it. It allows me to template docker compose easilty and it's vault is familiar to me. There are ways to use NixOS entirely manage the server, but I'm not there yet (and don't want to have to create the secrets files manually). I haven't seen how tools on NixOS handle secrets without manually making the file after ssh'ing into the server.
To me, I see the NixOS server as a way to manage the server and it's configuration, packages, networking, and anything to do with the host. On top of that extremely stable base, the Ansible playbooks are a tried and true way to manage the configuration of the containers.
There are ways to use NixOS to manage the containers, but there are currently more questions than paved road solutions. I'm not sure how to handle the secrets files (/nix/store is world readible). I haven't seen how tools on NixOS handle secrets without manually making the file after ssh'ing into the server. To me it seems like a good idea to use Ansible to manage the containers, and NixOS to manage the host.
When I change the ansible files for the NixOS configuration, it will fail to build if I have an issue that impacts the host.
Full stop! That is powerful. The only thing to avoid is to make a change that removes tailscale...which would remove my ability to ssh into the server. I could go grab the server, plug in a keyboard and monitor, and revert with GRUB. That is a pain, but is a nice physical backup plan.
The first setup:
- copy the configuration.nix file directly into /etc/nixos/configuration.nix
- adjust the hostname
- delete the jinja2 template rows
- run
sudo nixos-rebuild switch - then I can run
sudo tailscale up --sshto connect to the server to tailscale - then run the ansible playbook to configure the containers and adjust the nix configuration (rebuilding and switching to the new configuration as needed).
I don't believe it is a good idea to open any ports from that server up to the internet, from my local network.
I use Tailscale to connect back to these systems and am protected by WireGuard and no exposed ports to the internet.
The docker-compose files all use a Tailscale sidecar to connect to the Tailscale network. This allows me to connect to the containers from my local machine, and from the server, but not from the internet. It also sets up HTTPS for the containers, using Let's Encrypt.
See Contain your excitement: A deep dive into using Tailscale with Docker, by Alex Kretzschmar.
I allow some local ports to be exposed to the local area network, but only ones needed to configure my Wireless Access Points (to facilitate there adoption).
The default is to not expose a port, and use HTTPS with my Tailnet.
I bought a Macbook Air and use it for development, catching up with friends, and when I just need to get some personal things done. Since Nix runs very well on Apple Silicon Macs, I have started to do a lot of configuration with Nix. This removes the need for Development Containers and any virtualization...speed!
I originally started down a path of using dotfiles and ansible to manage my (Ubuntu LTS) desktop configuration, iancleary/ubuntu-dev-playbook.
Since then, I've discovered the Nix package manager, and the NixOS Operating System.
I still use a dotfiles repo, iancleary/dotfiles, but have switched from yadm to chezmoi, as the latter supports Windows, while yadm is Unix only. I recommend both, but Chezmoi fit my changing needs.
At work (RF Systems Engineering on Communications Satellites), the ubiquitous operating system is Windows.
I've dabbled with Ubuntu LTS as a host operating system, but the Microsoft ecosystem can be high friction, depending on what kind of tasks you do, day to day. I went to try Windows Subsystem for Linux, but experienced issues while connecting to the work VPN and DNS issues, when using WSL2. There are probably ways to get that to work now (at the time of writing this) or in the future. However, that's not where I want to spend my time. So, for now, I've focused on how to cultivate all the tools and systems I'd like, while using Windows as the host Operating System.
I explored various options in Virtual Machines, while on a Windows host: Hyper-V, VirtualBox. The performance of Hyper-V was intriguing to me, but the lack of static IP Addresses was frustrating. Again, there might be ways to setup Hyper-V to consistently address VMs by IP Address, across reboots, but the only workflow I found was to use PowerShell to get the IP Address after reboot. That worked fairly well with Ubuntu guests, but I experienced intermittent issues recently. See iancleary/local-ssh-config for a Python package that helps script the update process (get IP Address, update ssh config files).
I've landed on Virtual Box, with a few tricks to make SSH consistent across reboots. Port Forwarding!
TBD to add a repo, but search online for "Virtual Box Port Forwarding +SSH" and you'll find the solution. It maps a host port to a guest's port 22 (or whatever port the guest uses for the openssh server).
So my solution, Virtual Box, allows me to use NixOS, have an immutable file system with rollbacks via GRUB entries, that I can pause/resume as needed. I really enjoyed learning Ansible, and it is a fantastic tool, but the discover of the recovery and rollback features of NixOS were a *illuminating discovery for me. The build with fail if it causes issues with other parts of the configuration.
I have dabbled with NixOS as a desktop Operating System and can say it is fantastic.
My NixOS Configuration is in the iancleary/nixos-config repo.
There are some excellent write up on Nix and NixOS, so I'll defer to them rather than repeat them (poorly) here:
-
Home Manager, which appears to be a preference on whether you care for it and want to use it.
-
Declarative management of dotfiles with Nix and Home Manager. Nix is a declarative package manager with rising popularity. One of the best ways to actually learn it is to use it, and what better way to learn it than using it every day to manage your dotfiles? 10 min read, by Ole Kristian Pedersen