Soap boxing here but I feel these kinds of use cases is what NixOS is built for.
Declarative config to setup the system, users, and apps.
Declarative and customizable impermanence exactly how you want it.
I use Tails as well but NixOS is my daily driver. Anything not marked explicitly to persist is dropped each reboot. I’m the only user so I keep the last 30 days of non persisted data for like a power outage but that’s something I had to go out of my to setup for my use case.
The original used XI where it was 9 or 11 depending on the side.
edit: Nope I was wrong. That post links this one, lol.
For our lower environments we use rsync like the author but skip the pipeline altogether. The servers have a watch script to restart when files are rsynced. We then have a local watch script that rsyncs on file changes.
Relatively instant deploy (2-5s) whenever a file is saved.
Good call on a simulated failure. When I first set it up, it was LVM/BTRFS or ZFS as my top choices. It was a coin toss at the time because I hadn’t built this sort of setup before.
I use immutable nixos installs. Everything to redeploy my OS is tracked in git including most app configurations. The one exception are some GUI apps I’d have to do manually on reinstall.
I have a persistence volume for things like:
I have 30 days (or last 5 minimum) of system rollbacks using BTRFS volumes.
The personal files are backed up hourly to a local server which then backs up nightly to B2 Backblaze using rclone in an encrypted volume using my private keys. The local server has a mishmash of drives in a mirrored LVM setup. While it works well for having mixed drives, I’ll warn I haven’t had a drive failure yet so I’m not sure the difficulty of replacing a drive.
My phone uses the same flow with RoundSync (rclone + GUI).
Git repos are backed up in git.
Logs aren’t backed up. I just persist them for debugging and don’t want them lost after every reboot.
Caches/Games are persisted but not backed up. Nixos uses symlinks and BTRFS to be immutable. That paradigm doesn’t work well for this case. The one exception is a couple game folders are part of my personal files. WoW plugin folder, EvE online layouts, etc.
I used to use Dropbox (with rclone to encrypt). It was $20/mo for 2Tb. It is cheaper on paper. I don’t backup nearly that much. Backblaze started at $1/mo for what I use. I’m now up to $2/mo. It will be a few years before I need to clean up my backups for cost reasons.
The local server is a PC in a case with 8 drive bays plus some NVME drives for fast storage. It has a couple older drives and for the last couple years I typically buy a pair of drives on sale (black Friday, prime day, etc). I have a little over 30TB mirrored, so slightly over 60TB in total. NVME is not counted in that. One NVME is for the system, the others are a caching layer (monero node) or temp storage (transcoding as it also my media server).
I like the case, but if I were to do it again, I’d probably get a rack mountable case.
For the networking I found some repos with Nix and Gluetun (OCI containers). I don’t see them in my bookmarks, so it was probably a day project when I set up and didn’t keep the references.
That part is still in docker / podman. So any docker network guide just needs to be translated to nix.
Best resource I’ve found is searching GitHub.
My setup closely follows https://github.com/Misterio77/nix-config.
For servarr I just translated someone else’s docker compose setup to nix. There are some ready made nix ones you can look at like https://github.com/rasmus-kirk/nixarr/tree/main/nixarr.
The complex networking I just picked up over time once I knew my way around a little bit.
GitHub is your best resource. lang:nix search terms.
I wouldn’t run NixOS in a container. With native nix containers I’m pretty sure they share the store. For docker I’d use images built with nix (doesn’t run nix itself) or pull from docker hub.
OS: NixOS (high learning curve but its been worth it). Nix (the config language) is a functional programming language, so it can be difficult to grok. Documentation is shit as its evolved while maintaining backwards compatibility. If you use the new stuff (Nix Flakes) you have to figure what’s old and likely not applicable (channels or w/e).
BYOD: Just using LVM. All volumes are mirrored across several drives of different sizes. Some HDD volumes have an SSD cache layer on top (e.g., monero node). Some are just on an SSD (e.g., main system). No drive failures yet so can’t speak to how complex restoring is. All managed through NixOS with https://github.com/nix-community/disko.
I run stuff on a mix of OCI containers (podman or docker, default is podman which is what I use) and native NixOS containers which use systemd-nspawn.
The OS itself I don’t back up outside of mirroring. I run an immutable OS (every reboot is like a fresh install). I can redeploy from git so no need to backup. I have some persistent BTRFS volumes mounted where logs, caches, and state go. Don’t backup, but I swap the volume every boot and keep the last 30 days of volumes or a min of at least 10 for debugging.
I just use rclone for backups with some bash scripts. Devices back up to home lab which backs up to cloud (encrypted with my keys) all using rclone (RoundSync for phone).
Runs Arrs, Jellyfin, Monero node, Tor entry node, wireguard VPN (to get into network from remote), I2C, Mullvad VPN (default), Proton VPN (torrents with port forwarding use this), DNS (forced over VPN using DoT), PiHole in front of that, three of my WiFi vlans route through either Mulvad, I2C, or Tor. I’ll use TailsOS for anything sensitive. WiFi is just to get to I2C or Onion sites where I’m not worried about my device possibly leaking identity.
Its pretty low level. Everything is configured in NixOS. No GUIs. If its not configured in nix its wiped next reboot since the OS is immutable. All tracked in git including secrets using SOPS. Every device has its own master key setup on first install. I have a personal master key should I need to reinstall which is tracked outside of git in a password manager.
Took a solid month to get the initial setup done while learning NixOS. I had a very specific setup of LVM > LUKS encryption /w Secure Boot and Hardware Key > BTRFS. Overkill on security but I geek out on that stuff. Been stable but still tinkering with it a year later.
I saw that documentary. “The Wolf of Wall Street” or something? Maybe that was actually late 80s-early 90s.
On a daily basis I consume enough drugs to sedate Manhattan, Long Island, and Queens for a month. I take Quaaludes 10-15 times a day for my “back pain”, Adderall to stay focused, Xanax to take the edge off, pot to mellow me out, cocaine to wake me back up again, and morphine… Well, because it’s awesome.
You need downers to ride the uppers and get that perfect drug fueled circadian rhythm going.
Energy drinks during the day and a nice indica bong/dab rip, edible, or blunt in the evening.
Warning: If things have escalated to cocaine/meth/adderall to go up and opiates and a handy from the local masseuse to go down, you’re probably riding the rhythm too hard.
/s please take care of yourself!
My work laptop is a Dell Precision. It was a “data science” model that came with Ubuntu. Wiped Dell’s modified Ubuntu and put vanilla Ubuntu on it and now running Nixos. Works great. There was a weird period when using triple monitors with their dock had an intermittent issue on boot where resolutions and monitors were not being detected. Cause was Nvidia drivers. It eventually got resolved and it was easy enough to rollback the drivers to one that worked.
Won’t auto update but you could add the upgrade command to a login script or something.
Won’t lie, nix has a high learning curve to get the most out of it, but installing a single app is pretty simple.
Most startups I’ve applied to are Linux friendly.
I currently work for a fortune 100 and managed to get a Linux machine purchased as a “lab” machine.
I’m fully in control. IT doesn’t even know it exists. I’m not allowed on the corporate network, but I managed to get some internal corporate access through another department’s lab network (IT sanctioned) that has a VPN with a few routes to things like ticketing, time cards, and our internal wiki. Most of the stuff I need to do my job is in AWS and we are allowed to add home IPs to the security groups.
IT still gives me a MacBook. I use it like once every 6 months.
nixos-unstable is the only thing I will use currently.
I’m running bleeding edge stuff like the latest kernel, Hyprland nightly, my own “shell” built from Gnome components and lots of custom stuff using GJS (Gnome JavaScript).
If you get one, and you are free to do whatever on it, encrypt your drives like your job depends on it. I have a memorized passphrase, pin protected hardware key, and a key in TPM. No biometrics.
As far as other nice things to have:
I work in software dev as FYI. For the few issues I have, my team has more issues getting stuff working consistently on macOS for our project. I used that as a justification when requesting the laptop: my dev environment should closely match our runtime environment. Most of that is moot now since we use Nix flakes in our repos for local dev envs.
Immutable Nixos. My entire server deployment from partitioning to config is stored in git on all my machines.
Every time I boot all runtime changes are “wiped”, which is really just BTRFS subvolume swapping.
Persistence is possible, but I’m forced to deal with it otherwise it will get wiped on boot.
I use LVM for mirrored volumes for local redundancy.
My persisted volumes are backed up automatically to B2 Backblaze using rclone. I don’t backup everything. Stuff I can download again are skipped for example. I don’t have anything currently that requires putting a process in “maint mode” like a database getting corrupt if I backup while its being written to. When I did, I’d either script gracefully shutting down the process or use any export functionality if the process supported it.
I use rclone and the Round Sync Android client.
Supports a ton of back ends, self hosted, and commercial options. You can transparently encrypt with private keys you control.
I personally use B2 Backblaze for storage.
My phone backs up every night and Round Sync pushes them to B2. On my desktop I can mount as a volume. I can also access my storage from my phone going the other direction.
I’ve done the same using SFTP if I don’t want the overhead of persistent file storage.
It does not support indexing or previews for searching or finding say a photo. You can put whatever you want for data. So I have caches, indexes, and thumbnails that work in Linux. I can’t really make use of those on my phone though.
Rclones bisync feature is also a bit dangerous when I tried to use it a year ago. I more than once “deleted” everything. B2 doesn’t delete by default, just hides, so I was able to recover. I now do unidirectional syncs from my machines to different buckets until I’m motivated to investigate a proper 3-way merge solution.
Oh nice! I’ll have to dig into that. Wonder if its an implementation issue across vendors. I was always under the impression that DHCPv6 was the common convention if not static.
Ok. So a device didn’t get a dhcp address? No problem… It creates it’s open IP address and starts talking and try to get out on internet on its own…
Its not that different from a conceptual point of view. Your router is still the gate keeper.
Home router to ISP will usually use DHCPv6 to get a prefix. Sizes vary by ISP but its usually like a /64. This is done with Prefix Delegation.
Client to Home Router will use either SLACC, DHCPv6, or both.
SLACC uses ICMPv6 where the client asks for the prefix (Router Solicitation) and the router advertises the prefix (Router Advertisement) and the client picks an address in it. There is some duplication protection for clients picking the same IP, but its nothing you have to configure. Conceptually its not that different from DHCP Request/Offer. The clients cannot just get to the internet on their own.
SLACC doesn’t support sending stuff like DNS servers. So DHCPv6 may still be used to get that information, but not an assigned IP.
Just DHCPv6 can also be used, but SLACC has the feature of being stateless. No leases or anything.
The only other nuance worth calling out is interfaces will pick a link local address so it can talk to the devices its directly connected to over layer 3 instead of just layer 2. This is no different than configuring 169.254.1.10/31 on one side and 169.254.1.11/31 on the other. These are not routed, its just for two connected devices to send packets to each other. This with Neighbor Discovery fills the role of ARP.
There is a whole bunch more to IPv6, but for a typical home network these analogies pretty much cover what you’d use.
No. Its all text file config. You wouldn’t use live CD mode. You define your own how you want it to work.
Its a steep learning curve so if looking for off the shelf solutions, don’t use nix. If you need something custom but through a single config paradigm, nix is awesome.