I'm also having this issue. I've tried 3 different firewalls (sophos, pfsense and unifi) all have the same issue with random latency. I've tried fiddling with MTU/MSS to no avail but I did notice when I used my business VPN (going to digitalocean) I ...
