VERIZON... What did you change?

For two years now, Digital Operatives has hosted some systems on Verizon FIOS.   Largely it has been trouble free.  That is until about a week ago.

About a week ago,  two employees (eventually 3) described an issue that they were having connecting to the mail server and ssh server residing on this network.

When connected via their home FIOS connection, they would be able to establish a TCP connection,  and then the connection would timeout before any meaningful data could be communicated.

This happened to several of our employees,  but not all employees that had FIOS,  who connected from their home Verizon FIOS networks.

Those same employees,  could connect with the same system and establish strong connections, while connected via Comcast, or AT&T phone tether.

What makes this weirder,  is we have another chat server that those same employees could connect to and maintain connections despite the problems with SSH and mail.

What accounted for the difference?  Why would this happen?  

It took a few days for us to not just cough it up to something strange or intermittent.  Anybody that has ever dealt with an ISP's tech support, knows it is not going to be a terribly enjoyable experience.

Nevertheless,  after several days of troubleshooting (getting through the stages of Verizon help) they insisted on sending us a "new" Modem/Router.  This was tremendously frustrating as we are not "run of the mill" users,  and couldn't logically explain why it wasn't something inside Verizon's internal network as opposed to the router that we've had for two years.  

Before waiting for the router,  we decided to do some old school Internet troubleshooting.

Using the ping command with the -s option,  you can send packets with different sizes.

(From Server side)
ping <employee ip>      100% connectivity

ping -s 500 <employee ip>      100% connectivity

ping -s 1500 <employee ip>     0% connectivity    ???????   Strange

ping -s 1499 <employee ip>    100% connectivity

What in the hell?  Are you telling me that some router on Verizon's network can't handle 1500 byte packets?

Sure enough,  we changed the MTU size on the SSH server and mail server.   Connections were established and maintained just fine.  No more issues....

Somewhere inside Verizon's network is likely to be a misconfigured router or switch.   Not sure I know how to tell the right person.   Even if they are using a protocol or format that doesn't support 1500 byte MTU,  shouldn't they at least support fragmentation on those links?

The world may never know.   Good luck playing Call of Duty GHOSTS with your Verizon peers when the PS4 comes out.