We were made aware of an outage about an hour ago at the datacenter that services our ASP1 Hosting Platform. This outage may affect websites, email, FTP and DNS for affected customers. The datacenter is working on the issue and is aware of it. Updates will be posted here.
Update 1:20pm
We do know that there is an issue with the upstream provider at the datacenter where our ASP1 Platform is located. We have not had an update on an ETA but typically these types of issues are resolved rather quickly. As we get information it will be posted here.
Update 2:40pm
We've been notified by Level 3 that the service will be restored within an hour. We will update customers with a report of what caused this outage once services have been restored and we've gathered all information from Level 3.
Update 3:40pm
We were told that the issue would be resolved within the hour but have not received further information. As of this moment the outage still continues and we are awaiting an update on the ETA for resolution. The information we have been provided is that it is network related not system or server related.
Update: 4:20pm
We are still working with level 3 to get services back online. Everything is good on both ends, but we still have no circuit uplink. Engineers are still working to find the cause.
Conclusion: approximately 4:30pm the issue was resolved and all services came back online. We will be reviewing this with our providers to understand what the underlying issue was and how we can avoid having this type of outage in the future. We apologize for any inconvenience this may have caused and for those who were offline we really appreciate your patience and kindness as we let our upstream data provider sort through the issue.
Root Cause
Our ASP1 Windows Hosting Network/Datacenter uses 2 private AS numbers provided by Level 3 to allow for BGP routing between our routers and the Level 3 edge routers that service our circuits. This allows for failover between routers in the unfortunate event that one fails. Due to an outstanding order that was tied to our account, their system automatically closed the order and marked our AS numbers as inactive. Within 5 – 7 minutes their routers received the updated configuration and notified that these routes are now inactive and were removed resulting in all traffic to and from our network to be unroutable.
Because of the missing AS numbers under our account, the tier 2 support technician assumed we did not use BGP and rebuilt our connection without it. Our routers are configured for BGP and advertise BGP, as a result they were unable to establish a connection and advertise our IP routes to the edge router.
Tier 3 support was able to see that our configuration was invalid and we provided our BGP information and previous AS numbers. The technician was able to restore the BGP routes and within minutes our service was back online.