It didn't matter if my work VPN was on or not. The work VPN did affect my speed but not a 20x decrease.
I figured out the problem last night though and thought I would share. We have a Disney Circle at home for parental controls on the network and have had it for a few years. Devices are assigned to family members, be Unmanaged, or assigned to "Home". I figured out that it appears as though devices on the "Home" profile are being throttled down to about 10Mbps. I'm looking through the app to see if I can find a setting for this but so far no luck. In any case, once I moved the computer from Home to my profile or Unmanaged, everything started working normally.