Out of curiosity, what software is normally being run on your clusters? Based on my reading, it seems like some companies run clusters for business purposes. E.g. an engineering company might use it for structural analysis of their designs, or a pharmaceutical company might simulate the interactions of new drugs. I assume in those cases they’ve bought a license for some kind of high-end software that’s been specifically written to run in a distributed environment. I also found references to some software libraries that are meant to support writing programs in this environment. I assume those are used more by academics who have a very specific question they want to answer (and may not have funding for commercial software) so they write their own code that’s hyper focused on their area of study.

Is that basically how it works, or have I misunderstood?

This actually came up in my research. Folding@Home is considered a “grid computer” According to Wikipedia:

Grid computing is distinguished from … cluster computing in that grid computers have each node set to perform a different task/application. Grid computers also tend to be more heterogeneous and geographically dispersed (thus not physically coupled) than cluster computers.

The primary performance disadvantage is that the various processors and local storage areas do not have high-speed connections. This arrangement is thus well-suited to applications in which multiple parallel computations can take place independently, without the need to communicate intermediate results between processors.

I’ll have to look a little more into the AI stuff. It was actually my first thought, but I wasn’t sure how far I’d get without GPUs. I think they’re pretty much required for Stablediffusion. I’m pretty sure even LLMs are trained on GPUs, but maybe response generation can be done without one.

I’m not sure what you’d want to run in a homelab that would use even 10 machines, but it could be fun to find out.

Oh yeah, this is absolutely a solution in search of a problem. It all started with the discovery that these old (but not ancient, most of them are intel 7th gen) computers were being auctioned off for like $20 a piece. From there I started trying to work backwards towards something I could do with them.

I was looking at HP mini PCs. The ones that were for sale used 7th gen i5s with a 35W TDP. They’re sold with a 65W power brick so presumably the whole system would never draw more than that. I could run a 16 node cluster flat out on a little over a kW, which is within the rating of a single residential circuit breaker. I certainly wouldn’t want to keep it running all the time, but it’s not like I’d have to get my electric system upgraded if I wanted to set one up and run it for a couple of hours as an experiment.

Slightly off topic, but as long as we're ranting about DNS…

Proxmox handles DNS for each container as a setting in the hypervisor. It's not a bad way of simplifying things, but if, hypothetically, you didn't know about that, then you could find yourself in a situation where you spend an entire afternoon trying every single one of the million different ways to edit DNS in Linux and getting increasingly frustrated because the IP gets overwritten every time you restart the container no matter what you do, until eventually you figure out that the solution is just like three clicks and a text entry box in the Proxmox GUI!

…Hypothetically, of course.