Hello,
I just spent the last 1h45 with a support rep and not any closer to finding a solution.
I'm curious to see if anyone else has experienced something similar.
Here is some context for reference: I have around 25+ yrs in IT/Infosec experience developing network infrastructures. I run a number of servers at home for both professional and personal reasons.
I currently have a 3Gbps Bell Fibe connection, this is running through the bridge port on the HH4k into an opnsense system using the 10G WAN port going into a 10G port on the firewall. The internet uplink is achieved using PPPoE (with Bell login info) on the 10Gb WAN port.
Up to here, everything works flawlessly:
My speed tests on the opnsense box are consistently, which matches the modem speed tests. This shows I am able to access the full 3G most of the time, clocking at at around 3.2Gbps for download and 2.8-3.2Gbps on upload.
For professional reasons along my line of work, I wanted to download a fairly large data set from one of our private AWS S3 servers (using s5cmd to speed up the process, which is able to max out a 40Gbps connection if available). The download is done on a file server with a 10Gbps line plugged into a 10Gbps switch, which is also connected to the opnsense firewall (thus providing internet access).
The first time I did it, the download ran for about 7-10 min, then all of a sudden the entire connection dropped - The internet went down through the PPPoE uplink/bridge.
The uplink was showing green on both the modem and opnsense (both different PPPoE lines). I was seeing around 400-425Mb/s (roughly the full 3Gbps in download) during the download process until the drop (I was able to access the modem through a separate ethernet port on my system).
Two minutes later, the internet returned by itself without any action. When I attempted the process again and every attempt after, the download went for around 4-5min consistently then same thing, connection drop. 2 Min down, then back online.
In my experience, this looks a lot like some kind of throttling or IPS/active defense blocking the connection after a certain sustained download threshold for size and time.
When I discussed this with 3 different reps, and after trying to explain the layout of the network, they kept insisting that I was having a hardware issue between the modem and the firewall. This was unlikely given I never lost connection to the upstream.
Eventually I managed to talk to someone from Level 2, which took another 30 min of explaining. After making me restart the modem 3-4 times, reboot the firewall and consistently encountering the problem 3-4 more times, he decided to contact his colleagues at SME Service, who were able to look at the packets and confirmed the connection was dropping.
The rep ultimately said that his SME Service colleague noticed the drops and mentioned the modem could be bad (noting that this is a replacement I received 3 days ago). They are shipping me a replacement.
After the call I attempted limiting the download bandwidth, to around 2.5Gbps on that download, but I encountered the same result.
I am not convinced another modem will make any difference; I feel like this is some kind of distribution center firewall/ips rule that blocks the kind of transfer that I need to perform at this point - This test transfer was around 500Gb, the full set is around 60Tb.
I'm not too happy with the fact that this is not working, especially since the rep kept repeating that there shouldn't be any limitations to bandwidth, duration or amount of data being transferred. This assumes I should be able to leverage the full pipe when available without restrictions.
I frequently transfer files around 10-15Gb without any problems at around 60-90Mbps. This issue is pretty strange.
I know this is a very particular use case, but I was wondering if anyone else was experiencing the same kind of issues that I am seeing with sustained full pipe transfers. Or if anyone could provide any insight on something config related that I could have missed that could be causing this problem.
Any thoughts, ideas appreciated.
Thanks!
Hi there @MommySciku
Thank you for your post and welcome to the Community.
We'd like to take a closer look into this and have sent you a private message.
Please check your Messages within your profile avatar in the top right corner.
Unsure how to check or send a private message? See How to send a private message
