Over the weekend, the American Internet provider CenturyLink outage by serious technical failure causes 3.5% drop in global traffic. The BGP firewall and routing issue quickly spread outside the CenturyLink network and affected other service providers, causing many other fail of many companies.
Through CenturyLink’s fault, Amazon, Twitter, Microsoft (Xbox Live), EA, Blizzard, Steam, Discord, Reddit, Hulu, Duo Security, Imperva, NameCheap, OpenDNS and many, many others companies went offline. According to Cloudflare analysists, the CenturyLink incident triggered a 3.5% drop in global traffic.“The problem arose in a data center in Mississauga, a Canadian city near Ontario. At the root of what happened was an incorrect Flowspec announcement”, – says the official CenturyLink message.
Flowspec is an extension to BGP that allows companies to use BGP routes to propagate firewall rules throughout their network. Flowspec announcements are usually used during various information security incidents (for example, during BGP hacks and DDoS attacks), since they allow companies to make operational changes to their network and react to the situation in a matter of seconds.
The incorrect announcement of CenturyLink from the data center in Mississauga essentially prevented the company’s BGP routes from taking root.
“CenturyLink first announced a set of new BGP routes and then accidentally dropped all routes due to an incorrect Flowspec rule”, – told specialists insiders of the Cloudflare company.
Although because of this, some of the company’s routers went offline, others remained in service and began to announce incorrect BGP routes to the neighboring Tier-1. As a result, it was possible to observe the domino effect in action, as this was followed by failures and outages in the networks of other companies.
To get out of this situation, CenturyLink specialists took a step that is very rare for the provider: they asked all Tier 1 Internet providers to disconnect and ignore any traffic originating from the CenturyLink network. Companies rarely make such decisions as it means a complete loss of connectivity for all of their customers.
As a result, the CenturyLink staff had to reboot all equipment and return to clean BGP routing tables. This process took the company’s specialists almost seven hours.
Let us remind you that a year ago, hackers “put down” one of the largest African providers.
By the way, did you know that, according to research, up to a quarter of global traffic is generated by malware? Now, I think, you are aware)
User Review
( votes)
