I usually only watch in the evening so haven't noticed if there's a difference with daytime. I've noticed even the ads are glitchy. It seems to get a bit better as the night wears on, maybe cause it's caching it better, who knows.
Usually when I'm watching TV on Bell Fibe, I'm watching the hockey game on CBC, and it seems like almost every time the broadcast is glitchy! To confirm it's not just the game, I have switched to other channels which also experienced similar glitches...