NGINX is a bit more hands on than some other options but it’s mature, configurable and there’s a huge amount of information out there for setting it up for various use cases.
in my case, its what I set up when i was first getting into this and it works, so I don’t want to go through setting up anything else.
can you run something like iperf3 or openspeedtest between the server and client to prove its a network throughput issue?
do you have a network switch you can add to avoid switching through your router (if it is indeed bad?)
Have you ensured you arent unknowingly using wifi at either end?