Packet loss and increasingly high latency at fixed intervals

[MicrosoftSam]

Hello Bell community !

4-5 days ago I started experiencing weird latency issues. Here you can find a log of my ping output over a period of time to better understand the issue. Every so often the latency gets really high then slowly drops to normal (5ms to 8ms) then it jumps up again. Of course the modem integrated speed test tool is unable to detect such problem as it does not do a continous ping and I de believe that their speed test server is inside Bell's network which is not real world metrics IMO. Because of that Bell's technical support tells me that they are unable to help as the integrated speed test does not show any problems.

My current setup is like this at the moment : Windows 11 Machine -> Intel PCI-E Dual SFP+ (LC Fiber) -> TL-SG3428X SFP+ -> Cisco SFP+ DAC cable -> Sophos SG 230 (pfSense) SFP+ -> 10G SFP+ to RJ45 module -> CAT6a STP cable -> 10G port on Giga Hub

I'm usually doing PPPoE straight in pfSense like this : 

• No VLAN tagging
• MTU : default
• MSS : default

For testing I did the followings : 

• Tried doing PPPoE using only RJ45 from Sophos box to Giga Hub
  • Tried Giga Hub 1G and 10G ports
• Tried connecting my machine to the switch using the builtin RJ45 interface
• Tried connecting my machine to the Sophos box directly
• Changed the Sophos SG 230 for a Sophos SG 450 (same pfSense config)
  • This determined it is not a hardware issue on my side
    • Inter-VLAN traffic does not experience the same latency issue
• Tried changing MTU to 1492 and MSS to 1452 for the PPPoE connection
• Tried using Advanced DMZ instead
• Tried doing simple double NAT
• Tried using Sophos XG v19 as an OS instead (Yes I actually did put in a new drive an install and configure it -_-)
• Replaced cabling
• Tried connecting directly into the Giga Hub (read below)

The issue is also the same at a greatly lesser extent if plugged into the Giga Hub directly but I figure this is because there is actually no load on it since everything at home is currently in VLANs but I experience the same spikes (70ms to 100ms). For exemple, doing a speed test on speedtest.net will often cause packet loss and/or high latency spikes due to load. Giga Hub is on version 1.15.2 but the issue was present on v1.13.

I have never had this problem in 4 years with the exact same setup. (I do internal monitoring with Prometheus -> Grafana) I previously had the HH3K before upgrading to 1.5Gbps but also never had the issue with the Giga Hub until now. This is a show stopper for me, worse latency then hybrid fiber-coax.

I am wondering if someone here has a suggestion to further troubleshoot the problem or maybe someone working at Bell can help. @BellPatricia Maybe ?

Thank you for reading this and I am looking forward to read the replies.

[MicrosoftSam · Accepted solution]

New test : When establishing a routed (VTI) IPsec tunnel with my dedicated server in OVH Montreal I experience no latency issues. Constant 5ms-6ms.

[CrAwZ]

Hey!  So I just got off the phone with Bell for the same issues you describe above.  Past week or so, high latency...sites load slowly, links slow to respond etc.  It comes and goes, some times it's great, sometimes it's bad.  After a long wait, they said that my modem shows "issues with its sync to Bell. " and they are mailing me a new one to try.  PS, my integrated speed test and 3rd party speed test does give me 1GB returns, it's just the latency of getting things going that's the issue.

[Vanadiel]

Try a TTL of 250, see if that makes a difference.

Another thing to test is ping 1.1.1.1 and the home hub at the same time. That should provide a better understanding of what the home hub is doing. If it has processing issues it's ping should also increase.

After that I would use WinMTR and use 1.1.1.1 and let it run for a while. That should provide some more insight in what the routing points are doing and if anything wonky is happening there.

[CrAwZ]

This doesnt look great?

[CrAwZ]

gets worse, those are bell hops losing my packets?

[Vanadiel]

1 is Bell, 108.162.239.32 is Cloudflare.

Try increasing your ping interval from 1 second to 3 seconds, under settings.

It's not uncommon for routing points to either not respond to ICMP, hence you see 100% packet loss, or to respond to some but not all resulting in partial packet loss.

Your average latency is pretty ok and your maximum latency seems to be decent. I am leaning towards a stable connection.

[George6]

im getting high packet loss with fibe 1.5g and a friend of mine who lives a couple hours away who also has fibe 1.5g is getting the same packet loss. we both have run trace routes using the program Ping Plotter and we are both loosing 50% of our packets on hop 6 at ip 142.124.125.14 which looks like it is a bell ip that is the second last hop before it leaves bells network. is there any fix for this?

[XHZ379]

I'm in the same situation here, we have had random intermittent delays and packet loss on our bell Fibe 1.5G since April 15th, especially at night. I've done a speed test on the modem, which shows that I'm getting full speed, but the speed test on my device shows high peak packet loss.

I've tried restarting and resetting the modem, and resetting my pc network. I also contacted bell customer service online at noon the day before yesterday, and he ran some tests and said that there was nothing wrong with my modem and that my device would not be affected by network congestion, but I'm still suffering from packet loss.

This is really painful, and the packet loss makes it almost impossible for me to do any high-bandwidth activities such as live streaming at night, and I never encountered this situation before April 15th.

Also looking for a solution.

[BellPatricia]

Hi there @George6 & @XHZ379 

Thank you for reaching out to the Bell community 🙂

We would like gather a bit more information. To start, please share a screen shot of your trace route/speed tests.

• Which modem do you have?
• Is the packet loss specific to one device or all of the devices on your network?
• Are you connected via Wi-Fi or Ethernet and have you tested both connection types?
• Around what time in the evening does this occur?
• Are you using any third party equipment?

Hope to hear from you soon,

@BellPatricia 

[XHZ379]

his is my PC speed test during the week of severe network fluctuations, packet loss and latency, using Wifi.

• The modem is Home Hub 4000
• The packet loss occurs on all devices on the network
• I have only tested the wifi connection on different devices so far. Because my modem is hung in the elv box in the apartment, we have never used Ethernet and have no Ethernet cables. But our devices are no more than 5m away from the modem, and there is only a wall in between.
• The latency and packet loss are severe between 9 pm and 12:30 pm every day, but there are random packet losses at other times
• I have never used third-party devices such as routers.

My feeling is that my network has become unstable since April 15th. Before that, I could easily live broadcast online games on Twitch at a bit rate of 10000 kbps without any latency. But after April 15th, whenever I turned on the live broadcast at night, I would suffer from huge lag and packet loss from time to time. The game delay could reach 3000 ms in an instant (it was only about 20 ms before), and there would be random packet loss when turning on the live broadcast at other times, and the ping would jump from 20 ms to about 400 ms.

When the packet loss occurred, my network monitor also showed something strange. Normally when I was live streaming, it would go forward with an upload speed of about 10 Mbps. But when there was packet loss, the upload speed would suddenly come to a very low value (even less than 1 Mbps at the lowest), last for a short time, and then suddenly reach a very large value (40-50 Mbps), and then return to normal speed, and again.

[dks]

Thanks for your concern. One reason for packet loss during evening hours is spectrum crowding. This can happen in environments where there are a lot of devices competing for radio spectrum. This is especially true in a multi-unit building. This can also happen when there are a large number of devices close to each other. How many devices are connected wirelessly to the modem? Radio signals can also be affected by concrete walls and steel beams. 

[Vanadiel]

Try this test . Based on the screenshots you posted your pings are great on an idle connection, but as soon as you start transferring data it seems the line is choking itself.

Likely buffer bloat. Can have different reasons. I noticed you are on WiFi when doing those tests, you might want to try using a wired connection to rule WiFi issues out.

[XHZ379]

It seems to be getting better these past two days. I just ran this speed test on Wifi and got a B.

Anyway, I've already bought the relevant accessories and will switch to Ethernet next week to test the speed again.

[Vanadiel]

Sometimes things solve themselves over time.

It's possible to get a perfect score on Bell Fibre. 

https://www.waveform.com/tools/bufferbloat?test-id=54700fdc-45a2-47ad-a5bf-96b937d163b4