Home Hub 4000 bridge drops internet when using the full 3Gbps pipe

[ninja0n3]

Hello,

I just spent the last 1h45 with a support rep and not any closer to finding a solution.

I'm curious to see if anyone else has experienced something similar.

Here is some context for reference: I have around 25+ yrs in IT/Infosec experience developing network infrastructures. I run a number of servers at home for both professional and personal reasons.

I currently have a 3Gbps Bell Fibe connection, this is running through the bridge port on the HH4k into an opnsense system using the 10G WAN port going into a 10G port on the firewall. The internet uplink is achieved using PPPoE (with Bell login info) on the 10Gb WAN port.
Up to here, everything works flawlessly:

My speed tests on the opnsense box are consistently, which matches the modem speed tests. This shows I am able to access the full 3G most of the time, clocking at at around 3.2Gbps for download and 2.8-3.2Gbps on upload.

For professional reasons along my line of work, I wanted to download a fairly large data set from one of our private AWS S3 servers (using s5cmd to speed up the process, which is able to max out a 40Gbps connection if available). The download is done on a file server with a 10Gbps line plugged into a 10Gbps switch, which is also connected to the opnsense firewall (thus providing internet access).

The first time I did it, the download ran for about 7-10 min, then all of a sudden the entire connection dropped - The internet went down through the PPPoE uplink/bridge.

The uplink was showing green on both the modem and opnsense (both different PPPoE lines). I was seeing around 400-425Mb/s (roughly the full 3Gbps in download) during the download process until the drop (I was able to access the modem through a separate ethernet port on my system).

Two minutes later, the internet returned by itself without any action. When I attempted the process again and every attempt after, the download went for around 4-5min consistently then same thing, connection drop. 2 Min down, then back online.

In my experience, this looks a lot like some kind of throttling or IPS/active defense blocking the connection after a certain sustained download threshold for size and time.

When I discussed this with 3 different reps, and after trying to explain the layout of the network, they kept insisting that I was having a hardware issue between the modem and the firewall. This was unlikely given I never lost connection to the upstream.

Eventually I managed to talk to someone from Level 2, which took another 30 min of explaining. After making me restart the modem 3-4 times, reboot the firewall and consistently encountering the problem 3-4 more times, he decided to contact his colleagues at SME Service, who were able to look at the packets and confirmed the connection was dropping.

The rep ultimately said that his SME Service colleague noticed the drops and mentioned the modem could be bad (noting that this is a replacement I received 3 days ago). They are shipping me a replacement.

After the call I attempted limiting the download bandwidth, to around 2.5Gbps on that download, but I encountered the same result.

I am not convinced another modem will make any difference; I feel like this is some kind of distribution center firewall/ips rule that blocks the kind of transfer that I need to perform at this point - This test transfer was around 500Gb, the full set is around 60Tb.

I'm not too happy with the fact that this is not working, especially since the rep kept repeating that there shouldn't be any limitations to bandwidth, duration or amount of data being transferred. This assumes I should be able to leverage the full pipe when available without restrictions.

I frequently transfer files around 10-15Gb without any problems at around 60-90Mbps. This issue is pretty strange.

I know this is a very particular use case, but I was wondering if anyone else was experiencing the same kind of issues that I am seeing with sustained full pipe transfers. Or if anyone could provide any insight on something config related that I could have missed that could be causing this problem.

Any thoughts, ideas appreciated.

Thanks!

[ninja0n3]

Well that's the thing, it's not really an "up to..." problem, it's a "your connection drops and stops for 5 minutes if you download too much too fast". There is nothing in their agreements indicating that the connection will drop. I would understand speed dropping based on network load and node capacity on a residential line.

However this is different, the external connection is simply disabled at the edge center (while the modem is still online) and internet becomes inaccessible.

When asked, Bell will say that there are no restrictions and network should not drop. I've had this conversation a number of times with them.

[dks]

I would agree with you, but the wonderfully open words "up to..." leave any explanation or lack of explanation in Bell's hands. Your point could be argued until the cows come home (and it has, on many occasions before) but Bell appears to be saying, "You can't run your fibre internet connection wide open, full throttle, all the time because we don't guarantee that under the words "up to..." . The same was said in the days of DSL.  So here we sit. 

[ninja0n3]

For sure, it's a very long thread. Bell should take the time to read it however...

We discovered this as a solution about 8 months ago when running all kinds of tests to see if I was losing my mind. For most people however, it's not a solution. The equipment and configuration required to achieve that solution is out of the reach of most clients; if you're relying on your modem alone, then this is not feasible.

In order to implement the multi-WAN failover solution like this, you'd need a linux-based with a 2x 10G NIC (one going to the modem [WAN] and one or more going to your internal network) that will initiate multiple PPPoE connections. Microtik boxes and/or RouterOS are probably the best solution to implement something like this.

[ZaneP]

"...if you're relying on your modem alone, then this is not feasible."

For the majority of us who are in the retail/consumer space, this may be the only WAN-LAN hardware we have. So we can only hope that Bell will take steps to resolve the connection-dropping problem. 

[Anton]

first of all i would like to say thank you all.. especially to @ninja0n3 

"We discovered this as a solution about 8 months ago when running all kinds of tests to see if I was losing my mind" - lol you are not alone 

i do have requiered hardware, will test it next week 

i recorded 2 conversations with Bell support, maybe i should upload it for some giggles

[Anton]

your support laughed at me and told me there is nothing wrong with the network
and i'm the one to blame because i have no idea what im doing
seeng 10 pages just discussing this problem makes me think otherwise

[ninja0n3]

@Anton I've gone through this process over the course of over 2 months, back and forth; literally spent 60h discussing the issue with all levels of Bell support. You're walking the same road I was on a year ago, which I've documented on the DSL forums. I ended where I began, the problem still remains. I've learned to work around it.

As much as I hope you can reach someone in my experience, they don't. I doubt that they will dedicate any effort to resolve the issue unless it becomes a PR problem for them that gets real attention.

Save yourself 2 months of headaches and having to repeat 50 times what the problem is and how it is unlikely to be the "website" that's failing... If you can try alternative options such as multi-WAN mitigation.

You may try to make your point with a ton of logs, hard proof, documentation and plain common sense; in the end what I've experienced is like speaking to a 5 year old child. They only see what they want to see, everything else is irrelevant.

When you're the only player in town, you don't really have to care about what customers want or the problems they face, because ultimately, they have nowhere else to go.

[cd75]

I'm getting the same issue, really annoying! the worst, I just upgraded to 3gbs because this was a issue with the gpon from the hh3000 and my opnsense box. If I download at max speed a large file for too long. The connection drop for a while 2-5 min, making a new ppoe connection seem to work after a reboot. 

Same as everyone, my pppoe is still up.. just no internet! it's definitivly a IPS rules that kick in at Bell.. 
Ridiculous... 

[ZaneP]

"it's definitivly a IPS rules that kick in at Bell.. "

^ This.

Given that your WAN IP address essentially initiated the traffic, you'd think there would be some way of coding for exceptions, so Bell's IPS would recognize a valid user?

But I'm not a network engineer, so what do I know.

[bimmer]

Having this issue aswell. This is ridiculous. Bell get your stuff together.

[Vanadiel]

I would try directly from the modem, without any third party equipment attached. Just to rule out equipment/configuration issues.

I noticed a lot of these issues are from people in bridge mode, or using third party firewalls...

[ninja0n3]

There's about 12 pages of documentation about this issue over the course of the last 18 months on DSLReports: https://www.dslreports.com/forum/r33444694-Internet-Home-Hub-4000-bridge-drops-internet-at-3Gbps

This is not an equipment issue or a third party firewall problem, this is an upstream service issue that Bell refuses to acknowledge and fix.

[Vanadiel]

There are a lot of discussions in that thread. Have you ever tried pinging the server during a drop out?

I would start the troubleshooting from a direct connection between the file download machine and the modem, just to rule out any oddball issues with equipment after the modem.

[ZaneP]

Users who post in that thread have stated that they can duplicate this issue. It is unrelated to Bell modems, as the issue can be replicated when a modem is completely bypassed.

It also doesn't happen to all connections (a post on DSLR forum thread):

"Last I remember, [user=JAMESMTL] has demonstrated that the issue doesn't affect all connections and has seen it occur or not occur within sequential IPs of the same subnet."

There is speculation that the volume of data downloading is triggering Bell's IPS, being interpreted as a DDOS attack.

This was posted on the  DSLR Bell forum thread, August 14th:

"the only known mitigations at this time are to limit download speed or use multiple pppoe sessions either in series or parallel to minimize the impact of potentIal failures",

and @ninja0n3 replied:

"As @JAMESMTL mentioned, there is no official solution other than leveraging multiple failover PPPoE active connections."

From what has been posted, Bell flatly refuses to acknowledge there is any problem at all. Have a look at @ninja0n3 posts on this forum and the DSLR forum.

[Vanadiel]

There is a lot of speculation and possibilities discussed, that's why I thought it was a good idea and start fresh with a direct single connection from computer to modem and run tests. Especially since firmware upgrades on the modem have been performed since the start of the thread.

Once you start adding pfsense, switches, direct PPPOE connections into the mix it becomes hard to troubleshoot.