Home Hub 4000 bridge drops internet when using the full 3Gbps pipe

[ninja0n3]

Hello,

I just spent the last 1h45 with a support rep and not any closer to finding a solution.

I'm curious to see if anyone else has experienced something similar.

Here is some context for reference: I have around 25+ yrs in IT/Infosec experience developing network infrastructures. I run a number of servers at home for both professional and personal reasons.

I currently have a 3Gbps Bell Fibe connection, this is running through the bridge port on the HH4k into an opnsense system using the 10G WAN port going into a 10G port on the firewall. The internet uplink is achieved using PPPoE (with Bell login info) on the 10Gb WAN port.
Up to here, everything works flawlessly:

My speed tests on the opnsense box are consistently, which matches the modem speed tests. This shows I am able to access the full 3G most of the time, clocking at at around 3.2Gbps for download and 2.8-3.2Gbps on upload.

For professional reasons along my line of work, I wanted to download a fairly large data set from one of our private AWS S3 servers (using s5cmd to speed up the process, which is able to max out a 40Gbps connection if available). The download is done on a file server with a 10Gbps line plugged into a 10Gbps switch, which is also connected to the opnsense firewall (thus providing internet access).

The first time I did it, the download ran for about 7-10 min, then all of a sudden the entire connection dropped - The internet went down through the PPPoE uplink/bridge.

The uplink was showing green on both the modem and opnsense (both different PPPoE lines). I was seeing around 400-425Mb/s (roughly the full 3Gbps in download) during the download process until the drop (I was able to access the modem through a separate ethernet port on my system).

Two minutes later, the internet returned by itself without any action. When I attempted the process again and every attempt after, the download went for around 4-5min consistently then same thing, connection drop. 2 Min down, then back online.

In my experience, this looks a lot like some kind of throttling or IPS/active defense blocking the connection after a certain sustained download threshold for size and time.

When I discussed this with 3 different reps, and after trying to explain the layout of the network, they kept insisting that I was having a hardware issue between the modem and the firewall. This was unlikely given I never lost connection to the upstream.

Eventually I managed to talk to someone from Level 2, which took another 30 min of explaining. After making me restart the modem 3-4 times, reboot the firewall and consistently encountering the problem 3-4 more times, he decided to contact his colleagues at SME Service, who were able to look at the packets and confirmed the connection was dropping.

The rep ultimately said that his SME Service colleague noticed the drops and mentioned the modem could be bad (noting that this is a replacement I received 3 days ago). They are shipping me a replacement.

After the call I attempted limiting the download bandwidth, to around 2.5Gbps on that download, but I encountered the same result.

I am not convinced another modem will make any difference; I feel like this is some kind of distribution center firewall/ips rule that blocks the kind of transfer that I need to perform at this point - This test transfer was around 500Gb, the full set is around 60Tb.

I'm not too happy with the fact that this is not working, especially since the rep kept repeating that there shouldn't be any limitations to bandwidth, duration or amount of data being transferred. This assumes I should be able to leverage the full pipe when available without restrictions.

I frequently transfer files around 10-15Gb without any problems at around 60-90Mbps. This issue is pretty strange.

I know this is a very particular use case, but I was wondering if anyone else was experiencing the same kind of issues that I am seeing with sustained full pipe transfers. Or if anyone could provide any insight on something config related that I could have missed that could be causing this problem.

Any thoughts, ideas appreciated.

Thanks!

[cd75]

---

@Vanadiel wrote:

Once you start adding pfsense, switches, direct PPPOE connections into the mix it becomes hard to troubleshoot.

---

Not true

[Vanadiel]

How so? The more equipment you add, the more possibilities. The more you eliminate, the less possibilities.

If you can eliminate it to the modem, a pc and the connection, that would greatly lessen possible causes.

So far nobody has been able to say exactly what the cause is, just a possibility of firewall or Throttling. 

[ZaneP]

Bell does not throttle. It's illegal in Canada for ISPs to do this.

From what's been reported on the DSLR Bell forum, it's also unlikely to be any other bandwidth management intervention. The connection is dropped, not managed. The cause is uncertain, and there seems to be no obvious solution since Bell will not engage on this.

Perhaps you could do your real-world download test - a very large file download - and contribute to the problem-solving?

[Vanadiel]

I don't have a HH4000, but a HH3000 on 1.5/1 plan.

I downloaded 120 GB yesterday, which was the installer for BG3. Zero issues, and I am behind a UDM Pro SE and a 48 port switch and another 8 port switch. 48 port is linked to the UDM with SFP+ at 10 gig. And that is doing double NAT and includes the firewall on the UDM.

Has anybody ever considered it could be Amazon dropping the connection, or a routing point other than Bell's dropping the connection?

Is ICMP still working when the connection is dropped, what is the TCP/IP response to an ping request of the gateway?

What is the ICMP data on the Amazon side when the connection drops? Not sure if that is accessible or not.

The ICMP data should contain the information to determine to some extend what happened. That is why the protocol exists...

[ninja0n3]

120GB is probably not enough to trigger the cutoff. This typically happens when the 1.01Gbps+ download is sustained for longer than 5-6 minutes. For example if you're trying to download a TB of data, you might experience that.

---

@Vanadiel wrote:

Has anybody ever considered it could be Amazon dropping the connection, or a routing point other than Bell's dropping the connection?

Is ICMP still working when the connection is dropped, what is the TCP/IP response to an ping request of the gateway?

What is the ICMP data on the Amazon side when the connection drops? Not sure if that is accessible or not.

The ICMP data should contain the information to determine to some extend what happened. That is why the protocol exists...

---

It is not AWS, if that were the case, the rest of the internet would be reachable, which it isn't. The IP of the edge gateway (beyond the modem) is still reachable (valid ping), nothing beyond that.
This is something well documented that Bell has consistently refused to acknowledge.
There have been some exceptions depending on the assigned IP which has prompted speculation as to what the cause could be.

I'm not going over all the details again, there's around 350 posts on the DSLReports thread that goes over all the logs, proof, information, and tests done by multiple people who have experienced this problem; it is a long read, but describes all the issue and diagnostics in great detail.

[bimmer]

It's not the size that's a problem, it's the speed. Mine usually gives out within 40-50gb downloaded at over 300MB/s. (example: downloading COD off BattleNet)

[Vanadiel]

I don't know any location with a 1 TB data file, would be more than happy to download it.

What is the ICMP packet when pinging the next routing point past the gateway? It should contain some clue as to why the ICMP fails.

[Vanadiel]

Can't reach those speeds on mine, so I guess I cannot test it.

[Junrhod]

From my experience the issue can happen with less than 100GB, I frequently have the issue downloading games on Steam at max speed (Fibe 1.5).

Same symptoms that you describe, and rebooting the modem solves it. 

[Vanadiel]

That is interesting, because the general consensus in that DSL thread is that it's some kind of network protection kicking in due to the amount of open/closed connections.

But based on what you are saying, a download from Steam less than 100GB on a 1.5/1 profile, shows the same symptoms.

I can do 120 GB from Steam on the same profile, without any issues.

[Junrhod]

Had the issue yesterday downloading a game 88.2GB at around 90%. I don't have half of the tech knowledge of people on that thread, but considering how steam downloads works I suspect it's the size of packets/data of the game installs that make the protection kicks in. 
88.2GB/165 files total, out of which 75 files for 87GB of 0.75 to 1.5GB, could this be linked to the size/number of files ?

[Vanadiel]

I did 120 GB last week, without an issue.

[Junrhod]

I have nearly a 100% repro rate with this specific game.

[Vanadiel]

What's the name of the game? If it's free or I have it, I will download it to see what it does on mine.

[Junrhod]

New world, not free unfortunately. Is there anything I could use to log the issue on my side while I repro it ?