Packet loss and increasingly high latency at fixed intervals

[MicrosoftSam]

Hello Bell community !

4-5 days ago I started experiencing weird latency issues. Here you can find a log of my ping output over a period of time to better understand the issue. Every so often the latency gets really high then slowly drops to normal (5ms to 8ms) then it jumps up again. Of course the modem integrated speed test tool is unable to detect such problem as it does not do a continous ping and I de believe that their speed test server is inside Bell's network which is not real world metrics IMO. Because of that Bell's technical support tells me that they are unable to help as the integrated speed test does not show any problems.

My current setup is like this at the moment : Windows 11 Machine -> Intel PCI-E Dual SFP+ (LC Fiber) -> TL-SG3428X SFP+ -> Cisco SFP+ DAC cable -> Sophos SG 230 (pfSense) SFP+ -> 10G SFP+ to RJ45 module -> CAT6a STP cable -> 10G port on Giga Hub

I'm usually doing PPPoE straight in pfSense like this : 

• No VLAN tagging
• MTU : default
• MSS : default

For testing I did the followings : 

• Tried doing PPPoE using only RJ45 from Sophos box to Giga Hub
  • Tried Giga Hub 1G and 10G ports
• Tried connecting my machine to the switch using the builtin RJ45 interface
• Tried connecting my machine to the Sophos box directly
• Changed the Sophos SG 230 for a Sophos SG 450 (same pfSense config)
  • This determined it is not a hardware issue on my side
    • Inter-VLAN traffic does not experience the same latency issue
• Tried changing MTU to 1492 and MSS to 1452 for the PPPoE connection
• Tried using Advanced DMZ instead
• Tried doing simple double NAT
• Tried using Sophos XG v19 as an OS instead (Yes I actually did put in a new drive an install and configure it -_-)
• Replaced cabling
• Tried connecting directly into the Giga Hub (read below)

The issue is also the same at a greatly lesser extent if plugged into the Giga Hub directly but I figure this is because there is actually no load on it since everything at home is currently in VLANs but I experience the same spikes (70ms to 100ms). For exemple, doing a speed test on speedtest.net will often cause packet loss and/or high latency spikes due to load. Giga Hub is on version 1.15.2 but the issue was present on v1.13.

I have never had this problem in 4 years with the exact same setup. (I do internal monitoring with Prometheus -> Grafana) I previously had the HH3K before upgrading to 1.5Gbps but also never had the issue with the Giga Hub until now. This is a show stopper for me, worse latency then hybrid fiber-coax.

I am wondering if someone here has a suggestion to further troubleshoot the problem or maybe someone working at Bell can help. @BellPatricia Maybe ?

Thank you for reading this and I am looking forward to read the replies.

[KingMat]

+1 here, last week i woke up to an error from my router telling me that there was a loss of internet during the night... since that i'm experiencing some high latency period... Before that problem i always had 3-5ms steady... and now each day if not mutiples times a day i'm getting high latency sometimes higher than 60ms...

[BellPatricia]

Hi @KingMat 

Thanks for joining the conversation.

Did you get a chance to try any of the suggestions posted in the thread above? If you are still having trouble let us know, and be sure to provide additional information as requested here

Looking forward to your reply,

@BellPatricia 

[SIPTO]

Yeah so I googled this after another "pause" in the internet. This has been happening couple days now. This is packet lost to everything a hiccup then fine after a minute. Very annoying if you are WFH and on a teams call.

[dks]

Excellent test. Thanks for that link. 

[BellPatricia]

Hi there @SIPTO 

Welcome to the Bell community.

The community needs more information before we can help. Please review the questions and suggestions in the thread above and provide further details.

Hope to hear from you soon,

@BellPatricia 

[dks]

I agree. I have had it happen from time to time when recording broadcast TV. 

[dsmider]

I'm experiencing the exact same issue as everyone else. Massive packet jitter and latency spikes of 40-100ms at fixed intervals. I first start noticing this around April as well for some reason, seems to be more of a problem in evening hours 9pm-1am. No way to fix it, the quality of service of Bell fibe is broken IMO, no way to raise attention to this to a support engineer or a real tech "inside" the walls, just CS rep that doesn't know anything about the nature of the issue or acknowledge that one exists ,just reads trough her troubleshoot flowchart. I'm on GPON (1.5 Gigabit), Bell Gigahub 4000.

[dks]

Thank you for your concern. There are a number of reasons for packet jitter and latency spikes, both within and outside Bell's system and control. If you read the posts in this thread, you will see any number of potential issues and responses. 

It would be helpful for anyone experiencing this problem to run a traceroute when this happens, whether using an external program or from the modem dashboard, so we can see where these events are occurring. Copy and paste the traceroute in the forum. Once that is posted, Bell staff can see the issue and, if it is internal to Bell, pass the information forward.

It is rare for customers to speak to technical staff, no matter who the provider is. That has, at least, been my own experience. 

[nts-brien]

I'm also having this issue. I've tried 3 different firewalls (sophos, pfsense and unifi) all have the same issue with random latency. I've tried fiddling with MTU/MSS to no avail but I did notice when I used my business VPN (going to digitalocean) I didn't experience the same issues. I setup a full tunnel vpn to my digitalocean instance and low and behold no more latency spikes after 12 hours. 

When I investigated the routing from a server:
|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.50.1 - 8 | 939 | 865 | 0 | 0 | 69 | 17 |
| 10.11.4.89 - 0 | 1221 | 1221 | 0 | 5 | 65 | 14 |
| No response from host - 100 | 247 | 0 | 0 | 0 | 0 | 0 |
| 10.115.2.65 - 0 | 1221 | 1221 | 14 | 25 | 120 | 14 |
| No response from host - 100 | 247 | 0 | 0 | 0 | 0 | 0 |
| 64.230.100.56 - 0 | 1221 | 1221 | 15 | 25 | 88 | 15 |
| 142.124.125.14 - 21 | 679 | 540 | 19 | 25 | 80 | 21 |
| 142.124.127.39 - 0 | 1221 | 1221 | 15 | 24 | 82 | 15 |
| 108.162.239.32 - 0 | 1221 | 1221 | 15 | 29 | 103 | 15 |
| 108.162.239.28 - 0 | 1221 | 1221 | 15 | 28 | 122 | 15 |
| one.one.one.one - 0 | 1221 | 1221 | 15 | 25 | 83 | 15 |
|________________________________________________|______|______|______|______|______|______|

Bell's server 142.124.125.14 is dropping packets. I connected to 2 other clients of ours (we are an IT department so we have access to quite a few sites for testing these kinds of issues) - they are also having the same problems. They use traditional phones not VOIP like we do which is why they haven't really noticed but now with teams and zoom they do.

Shaw/Rogers does not seem to have this problem because they use different routes:
|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.1.1 - 0 | 2324 | 2324 | 0 | 0 | 56 | 0 |
| 70.76.0.1 - 6 | 1919 | 1814 | 7 | 12 | 33 | 14 |
| rd3he-be100-1.su.shawcable.net - 1 | 2283 | 2272 | 6 | 13 | 32 | 14 |
| rd4he-be3.su.shawcable.net - 0 | 2325 | 2325 | 6 | 13 | 32 | 14 |
| rc2fs-be2.mt.shawcable.net - 0 | 2325 | 2325 | 15 | 21 | 39 | 22 |
| rc3ec-be10.il.shawcable.net - 0 | 2325 | 2325 | 26 | 32 | 50 | 31 |
| rc4ec-be6-1.il.shawcable.net - 0 | 2324 | 2324 | 26 | 32 | 55 | 32 |
| 141.101.73.96 - 6 | 1909 | 1801 | 26 | 34 | 95 | 30 |
| 141.101.73.216 - 6 | 1889 | 1776 | 26 | 34 | 98 | 29 |
| one.one.one.one - 0 | 2325 | 2325 | 25 | 31 | 54 | 33 |
|________________________________________________|______|______|______|______|______|______|


In short.. when i route through bell in my area we get massive lag spikes - at least at myself and 5 other places of business. It may or may not be due to this server (142.124.125.14)  but I believe bell needs to take this seriously and check it out. No amount of changing the homehubs is going to fix that problem.

[Vanadiel]

You have 8 dropped packets at 192.168.50.1. Not sure how you have your network setup, but you should not have any packet loss internally.

142.124.125.14 could be dropping packets due to ICMP protection. Try increasing the ping times to intervals is 2 second, to see if that makes a difference.

[User4802]

Hi,

It Seems i got the same issue happening for the past week. I have the 1.5G service with a Giga Hub and around 8:00 pm (est) everyday i get high latency lasting 1 to 2 hour. In the past week the worst it got was yesterday with +- 150 ms. I did all the proposed test in this thread, an from what i can see, the latency jump up drastically, when my connection get routed to this point 

Any idea what it can be ? 

[BellPatricia]

Hey there @User4802 

Welcome to the Bell community 🙂 

What types of devices do you have and how are they connected (via Wi-Fi or Ethernet)? Are you using any third party equipment?

If you are testing over Wi-Fi, try switching to Ethernet to see if you get the same results.

Keep us posted,

@BellPatricia 

[User4802 · Accepted solution]

Hi @BellPatricia 

Thanks for the reply

I got hold on a tech on the phone yesterday. He asked me to perform a factory reset of my hub. From his end it looked like a wrong config. So far, it seem to be working. No latency event yesterday evening.

[User4802]

Hey ! 

Sadly im back with some bad news. 
The issues is back, latency is still really high between 7:30 and 10:00 pm.

I’ve been testing with the ping and traceroute utility within the bell hub and I get some really high latency on 2 or 3 hops in the chain, in the 120 ms range. Ping is also in the 100 ms

 Makes all online game unplayable. 

I'm still on a wired connection, testing directly from the hub, with no third party hardware in between the hub and the computer used for the test 

thanks 

[dks]

Thanks for your concern. Please post a traceroute from your modem tools dashboard when you are experiencing this issue. This will help us understand where the high pings are occurring.