I’m using CheckMK to monitor my hypervisor, physical hardware like disks, CPU etc. and SNMP-capable hardware like my pfSense firewall via a CheckMK instance in docker. It either works in docker or on a few different linux based OS like ubuntu and debian (see CheckMK download page).

There’s a free and open source version (called raw edition, GitHub Link) which I am using. It comes with a lot of checks / plugins for monitoring stuff out of the box and if there’s something it doesn’t ship, you can easily create your own check in whatever language your server is capable of executing a binary of. Or you could look up if there’s a user-contributed plugin on the official CheckMK Exchange Platform.

The whole configuration of this is based on rules with a lot of predefined rules and sane defaults already set.

To have an example for your use-case: You can monitor docker-logfiles and let CheckMK warn you, if specific keywords are or are not in a logfile. You will then be able to view the offending lines in the monitoring UI.

Why do I use this?

• We use it at work
• FOSS
• docker makes updating this easy
• can send mails, teams notifications, …
• very customizable and expandable

my docker compose file

# docker-compose.yml

services:
  monitoring:
    image: checkmk/check-mk-raw:2.4.0-latest
    container_name: monitoring
    restart: unless-stopped
    environment:
      - CMK_PASSWORD=changeme
    ports:
      # WEB UI port
      - "5000:5000"
      # agent communication port
      - "8000:8000"
      # used for SNMP
      - "162:162/udp"
      - "514:514/tcp"
      - "514:514/udp"
    volumes:
      - "./monitoring:/omd/sites"
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env

I did not run OPNSense, but I have a direct comparison for pfSense as VM on Proxmox VE vs pfSense on a ~400€ official pfSense physical appliance.

I do not feel any internet-speed or LAN-speed differences in the 2 setups, I did not measure it though. The change VM -> physical appliance was not planned.

Running a VM-firewall just got tiring fast, as I realized that Proxmox VE needs a lot more reboot-updates than pfsense does. And every time you reboot your pfSense-VM-Hypervisor, your internet’s gone for a short time. Yes, you’re not forced to reboot. I like to do it anyway, if it’s been advised by the people creating the software I use.

Though I gotta say, the pfSense webinterface is actually really snappy and fast when running on an x86 VM. Now that I have a Netgate 2100 physical pfSense appliance, the webinterface takes a looooong time to respond in comparison.

I guess the most important thing is to test it for yourself and to always keep an easy migration-path open, like exporting firewall-settings to a file so you can migrate easily, if the need arises.

[EDIT] - Like others, I also would advice heavily against using the the same hypervisor for your firewall and other VMs. Bare-Metal is the most “uncomplicated” in terms of extra workload just to have your firewall up and running, but if you want to virtualize your firewall, put that VM on its own hypervisor.

If you don’t want to have any freedom until you have it all, you’ll be slave forever.

You’re letting perfect get in the way of good enough.

I don’t know about tailscale, but it seems pihole has got you covered with local DNS, if you’re willing to set the local DNS records manually.

I use pihole as selfhosted DNS server for all my servers and clients. I don’t have many local DNS records (only 2), so if you handle a great amount of ever-changing DNS records, this might not be for you.

To me it seems like:

• you want to do a lot of stuff yourself on arch
• but there’s quite some complicated stuff to learn and try

I’d try Proxmox VE and, if you’re also searching for a Backup Server, Proxmox Backup Server.

I recommend these because:

• Proxmox VE is a Hypervisor, you can just spin up Arch Linux VMs for every task you need
• Proxmox VE, as well as Proxmox BS are open source
• you can buy a license for “stable updates” (you get the same updates, but delayed, to fix problems before they get to you)
• includes snapshots, re-rolls, full-backups, a firewall (which you can turn on or off for every VM), …

I personally run a Proxmox VE + Proxmox BS setup in 3 companies + my own homelab.

It’s not magic, Proxmox VE is literally Debian 13 + qemu + kvm with a nice webui. So you know the tech is proven, it’s just now you also get an easy to use interface instead of virsh console commands or virt-manager.

I personally like a stable infrastructure to test and run my important and experimental tuff upon. That’s why I’m going with this instead of managing even the hypervisor myself with Arch.

Thank you very much. I sent this to my coworker who expressed interest in switching to vim :)

It seems nobody really tested 11 to 13, or maybe any kind of major-version-skipping and you won’t find direct experiences here.

Your best bet is to follow the official procedure, so 11 -> 12 -> 13. I’ll leave you with the official upgrade guide for 11 to 12 and 12 to 13.

It seems longer than it is, as not every step is actually required for every system. When upgrading VMs, a snapshot pre-upgrade can also help you skip backup-steps in the guides.

Proxmox Virtual Environment (PVE, Hypervisor), my beloved. Especially in combination with Proxmox Backup Server (PBS).

My homelab would not exist without Proxmox VE, as I’m definitely not going to use Nutanix or VMWare. I love working with linux and Proxmox VE is literally debian with a modified kernel and a Management Webinterface on top.

I first learned about Proxmox VE in my company, while we still had VMWare for us and all of our customers. We gradually switched everyone over to Proxmox VE and now I’m using it at home too. Proxmox is an Austrian (my country) company, so I was double hyped about this software.

A few things I like most about Proxmox VE

• Ease of access to the correct part of the documentation you currently need (*)
• Open Source
• Company resides in my country (no US big tech walled garden)
• Linux / Debian based, so no learning new OS’s and toolchains
• Free version available
• Forum available and actually used

(*) What I mean by ease of access to the correct part of the documentation is: Whenever you’re in the WebUI and need to decide on some settings, there’s a button somewhere on the same page which is going to lead you directly to the portion of the documentation you need right now. I don’t know why this seems like such a great luxury, every software should have something like this.

Next steps

My “server” (some mini PC with spare parts I already had) is getting too weak for the workload I put it through, so I’m going to migrate to a better “server”. I already have a PC and most of the necessary parts, I just need some SSDs and an AMD CPU.

Even migrating from PVE (old) -> PVE (new) couldn’t be easier:

• PVE (old): create last backup to PBS, shut down PVE (old)
• PVE (new): add PBS, restore Backups
• ???
• profit

I think it’s great to have a series posting about personal achievements and troubles with selfhosting. There’s so much software out there, you always get to see someone doing something you didn’t even know could be done or using a software you didn’t realize even existed. Sharing is caring.

Of course you can always build a good PC or server.

I could have done that too, but I wanted my first real homelab-do-it-all-yourself setup to be a little more on the cautiously small side. I didn’t want to have too much noise in my apartment and also didn’t want to stress my electricity-bill and wallet too much, so I opted to build small and reuse what I had lying around.

I already had 2 Mini-PCs and a raspberry pi from earlier experiments with selfhosting. I just bought some disks and RAM. If you don’t have any mini-PCs, they’re relatively cheap in comparison with full PCs. Or you could use some older PC you still have but do not use.

My motto more or less was you can always spend more money and build bigger later

The final Hardware

• Mini-PC: Zotac ZBox CI665 nano
• RAM: 32 GB DDR4 RAM (according to specs, CI665 cannot go beyond 32GB sadly)
• SSD: 1x 2TB Samsung SATA SSD
• external USB HDD (6TB)

What I host on my Proxmox VE

• CheckMK Monitoring
• GitLab Runner
• pihole (network-wide, DNS-based adblocker)
• Nextcloud (production and staging instance for testing new updates before rolling out to production with my data)
• nginx proxy manager
• syncthing server

The 2nd Mini PC (some old intel NUC with 4 cores and 16 GB RAM) + a USB HDD is my Proxmox Backup Server for all this. And what’s really important (my data from nextcloud + some configs) gets backed up to my Hetzner Storage Box with restic.

The raspberry pi is now my WiFi Access Point :)

Conclusion

Homelab doesn’t need to be big or small, it can be whatever you want it to be or whatever you can afford or are willing to have and maintain. From my experience, if you’re not hosting anything CPU-intensive, older or smaller machines will do just fine.

For example, my nextcloud could easily use more resources than the whole Zotac ZBox could house, if there were more users. But as my services are only used by me, most of them are idle most of the time.

Tip at the end about your opsense-VM on Proxmox

I tried letting Proxmox host my pfSense too, but that got old pretty fast. Whenever Proxmox needed a Reboot, my internet was gone too for that time, as the pfSense VM on Proxmox was the gateway to my ISP-modem. In the end, I just bought a real Netgate pfSense appliance.