Home Hub 4000 bridge drops internet when using the full 3Gbps pipe

[ninja0n3]

Hello,

I just spent the last 1h45 with a support rep and not any closer to finding a solution.

I'm curious to see if anyone else has experienced something similar.

Here is some context for reference: I have around 25+ yrs in IT/Infosec experience developing network infrastructures. I run a number of servers at home for both professional and personal reasons.

I currently have a 3Gbps Bell Fibe connection, this is running through the bridge port on the HH4k into an opnsense system using the 10G WAN port going into a 10G port on the firewall. The internet uplink is achieved using PPPoE (with Bell login info) on the 10Gb WAN port.
Up to here, everything works flawlessly:

My speed tests on the opnsense box are consistently, which matches the modem speed tests. This shows I am able to access the full 3G most of the time, clocking at at around 3.2Gbps for download and 2.8-3.2Gbps on upload.

For professional reasons along my line of work, I wanted to download a fairly large data set from one of our private AWS S3 servers (using s5cmd to speed up the process, which is able to max out a 40Gbps connection if available). The download is done on a file server with a 10Gbps line plugged into a 10Gbps switch, which is also connected to the opnsense firewall (thus providing internet access).

The first time I did it, the download ran for about 7-10 min, then all of a sudden the entire connection dropped - The internet went down through the PPPoE uplink/bridge.

The uplink was showing green on both the modem and opnsense (both different PPPoE lines). I was seeing around 400-425Mb/s (roughly the full 3Gbps in download) during the download process until the drop (I was able to access the modem through a separate ethernet port on my system).

Two minutes later, the internet returned by itself without any action. When I attempted the process again and every attempt after, the download went for around 4-5min consistently then same thing, connection drop. 2 Min down, then back online.

In my experience, this looks a lot like some kind of throttling or IPS/active defense blocking the connection after a certain sustained download threshold for size and time.

When I discussed this with 3 different reps, and after trying to explain the layout of the network, they kept insisting that I was having a hardware issue between the modem and the firewall. This was unlikely given I never lost connection to the upstream.

Eventually I managed to talk to someone from Level 2, which took another 30 min of explaining. After making me restart the modem 3-4 times, reboot the firewall and consistently encountering the problem 3-4 more times, he decided to contact his colleagues at SME Service, who were able to look at the packets and confirmed the connection was dropping.

The rep ultimately said that his SME Service colleague noticed the drops and mentioned the modem could be bad (noting that this is a replacement I received 3 days ago). They are shipping me a replacement.

After the call I attempted limiting the download bandwidth, to around 2.5Gbps on that download, but I encountered the same result.

I am not convinced another modem will make any difference; I feel like this is some kind of distribution center firewall/ips rule that blocks the kind of transfer that I need to perform at this point - This test transfer was around 500Gb, the full set is around 60Tb.

I'm not too happy with the fact that this is not working, especially since the rep kept repeating that there shouldn't be any limitations to bandwidth, duration or amount of data being transferred. This assumes I should be able to leverage the full pipe when available without restrictions.

I frequently transfer files around 10-15Gb without any problems at around 60-90Mbps. This issue is pretty strange.

I know this is a very particular use case, but I was wondering if anyone else was experiencing the same kind of issues that I am seeing with sustained full pipe transfers. Or if anyone could provide any insight on something config related that I could have missed that could be causing this problem.

Any thoughts, ideas appreciated.

Thanks!

[Vanadiel]

If it would be the modem, it would drop on all machines. Are all these devices on WiFi, or are they wired or a mix of wired and WiFi?

[Sonia Brock]

All WiFi and the machine that keeps dropping carrier is sitting right next to the modem.

[Vanadiel]

Something to try next time it fails, is to do speed test from within the modem.

This will rule out any issues with the modem itself: Speed test 

[Sonia Brock]

For reasons I do not understand the dropped connection problem has resolved itself but, just in case, I have also added an Ethernet cable directly to the modem/router in case the problem reoccurs.

Thank you for your response.

[dks]

Thanks for the update. If it is possible, a wired connection is always better than a wireless connection.  

[Sonia Brock]

In time I lost all Internet connection. After getting a new modem there was still no Internet. A call to the Bell help number established that there may have been a problem with the wires.  Bell tech appeared and in rain and wind climbed up and fixed the broken wire in a very professional manner.  His remark to me that "It is never the modem." proved to be absolutely true 🙂

[shrodi]

OK, so there's a lot going on here! I've quickly read most of the posts, and here is my 2¢ as an experienced network technician. I will not speculate wether Bell is throttling or not, but on my part I never noticed anything that would suggest such a thing with my 3Gbps service on my 10/2.5/1Gb hybrid home network.

I've read quickly most of the posts, and it's obvious to me that there are probably many different causes to the quite diverse problems and symptoms that are reported. Also, I don't think everyone has done all the tests to make sure where the problem lies exactly, especially a stability test as decribed below.

I just recently successfully debugged my moderately complex home network (three routers, four subnets and about ten switches) that was experiencing intermittent Internet drops on a Bell Fibe 3Gbps subscription. The first thing one should do is to test wether there is stability at the modem.

I've seen a lot of ping and speed tests in the posts, but those tests don't tell the whole story. The only speed test that matters to check wether you're getting what you're paying for is the one you can do directly on the modem (within the modem administration panel): this test will tell you if you're getting what you're paying for in terms of bandwidth ("speed"). Otherwise, usually, what you want to do is a stability test over a decent period of time (1-2 hours), like the one freely provided at https://test.vsee.com/. Do this on an unburdened network, and with an ethernet cable (wired connection). Also, make sure your computer stays awake and the test page stays visible at all time in the foreground. This kind of test will tell you if any packets are dropped over time. If there are, the next step will be to determine where the packets are dropped. If your problem is exclusively with Wi-Fi, then see below for some hints.

STEP 1: do a stability test at the modem with a short CAT6 or CAT6A ethernet cable. A perfect stability test should show no fluctuations at all, but a few slight fluctuations is no big deal. If the stability test shows a lot of fluctuations, especially if they are important, then test with another cable to make sure. If the fluctuations persist, then it could be the modem, but it is more likely a problem with the fiber that goes to your home. In my case, the stability test was perfect at the modem.

STEP 2: if the wired stability test at the modem is peachy, then the problem is your setup. You must test at different places on the network to see where the fluctuations are introduced. In my case, it happens that I could identify three culprits: a defective SFP+ module was the main source of fluctuations, but then I also found out that two cables were giving some problems. After replacing them three, I could get a perfect stability test at any point within my network, and no more choppy Internet.

FOR WI-FI PROBLEMS: first, make sure it's only Wi-Fi by doing the wired stability tests above: Wi-Fi can't do better than a wired connection. If everything is peachy while wired, then because the 2.4Ghz band is saturated and very prone to interferences, make sure to priorize 5Ghz connections. Turn off 2.4Ghz on your modem (or any other acces point (AP)) and do a stability test while standing close to the Wi-Fi source. If the test fails, test again with another wireless device to make sure. If the stability problem persists, then your modem/AP is most likely to blame, or it could be a wireless network saturation or an AP crowding problem (see below). But if the stability is good, then your Wi-Fi is OK and the problems you encounter are most likely due to a bad reception.

• Bad reception: Don't forget that the 5Ghz bands, although less crowded than the 2.4GHz ones, are less penetrating, meaning that they are much more easily blocked by physicial objects like walls, and that the power of any electromagnetic wave will decrease proportionally to the square of the distance from the source of the signal (which means it drops very quickly over even short distances). You can use an app like "WiFi Analyzer" on Android or "WifiInfoView" on Windows (and a bunch of others) to test both the signal strenght and the crowdedness for different bands at different locations in your house. If you find that the signal is too weak at some places, think about setting up a mesh network (best solution, but more expensive) or adding repeaters (less expensive, but less performant, although they usually do fine enough).
• Wireless network saturation: If you have a lot of wireless devices around your house, especially IoT devices like cameras, sensors, etc., but only one main wired AP, then it could be that you're saturating your wireless network; in that case, even if the reception is good and the bands are not crowded, your AP just won't be able to keep up with the demand and will produce choppy connections. In that case, adding more wired APs should help.
• AP crowding: Sometimes too much is like not enough. Too many APs in the same area will interfere with each other and decrease the signal quality. Make sure there is a reasonable space between all APs, taking into account that 2.4GHz bands travel further than the 5GHz ones (so you would turn off 2.4GHz on some APs).