Monthly Archive for February, 2008

How Pakistan caused Youtube to drop off the net

BGP (Border Gateway Protocol) is the core protocol that runs the internet infrastructure. It’s a de-centralized protocol, meaning that no one server or core controls the internet. It’s designed to establish “peering” type relationships as well as “upstream” type relationships between different providers. There are three primary tiers of IP network providers:
-Tier 1: Providers that are very large, usually own a large IP network with their own fiber, and only establish “peering” relationships with other providers (peering is to provide access to each other’s networks with no money changing hands).
-Tier 2: Providers of varying size networks that establish large amounts of peering relationships, but also rely on “upstream” relationships to Tier 1 providers (upstream is to provide the lower tier provider with routes to the entire internet, with the lower tier provider paying for access to this service)
-Tier 3: Smaller or private providers that only establish upstream relationships with higher tier providers.

All of this works around the concept of BGP ‘prefixes’. These are simply routes that are transmitted across a ‘point-to-point’ session between two routers. A BGP prefix is usually written in slash notation (10.0.0.0/8), and describes the AS (Autonomous System) numbers of the network path that prefix has to travel through, as well as a route to that network. Like any other form of subnet routing, routers can receive multiple routes through to a particular IP or network. This allows redundancy, as well as the router to choose the shortest path through to that destination. There are many things that factor into these routing decisions, but the two we’re going to focus on is specificity of the route and AS path length.

Now my earlier example of 10.0.0.0/8 is a very large route. It’s a route to all the addresses in-between 10.0.0.0 and 10.255.255.255. Lets say your router calculates a route to this network that has 3 AS networks in-between. Now another router starts advertising a BGP route to 10.0.0.0/16. This is a route to just 10.0.0.0-10.0.255.255; a much smaller group of addresses. Now I’m really over simplifying the routing decision, but for the sake of explanation, in many cases this new route would take preference as it is routing to a more specific group of addresses.

This is exactly what happened to YouTube. YouTube was advertising a prefix of 208.65.152.0/22. The Pakistani government decided to block YouTube (see news article) for what is says is “anti-islamic” content. The WAY they did this, was by getting the major Pakistani ISP, Pakistan Telecom, to advertise a more specific BGP route of 208.65.152.0/24. This caused all traffic in the ISP to pick up the new route, which told the router to drop the traffic.

Now how this spread is even more interesting. Due to the nature of BGP, prefixes are handed off in peering sessions between two routers. PCCW Global Crossing (3491) is a Tier 1 IP network provider that provides upstream services to Pakistan Telecom (17557). However, they were not conducting proper filtering of the routes that Pakistan Telecom was providing up to PCCW in their direct relationship. Once the routes are accepted by PCCW, they are then labeled as routes coming from PCCW and any other provider that trusts PCCW would get those routes. Very quickly this caused all YouTube traffic to be redirected into Pakistan Telecom, where their routers dropped the traffic. Not only did they succeed in cutting off YouTube from Pakistan, but they cut it off for the world.
YouTube Traffic routing to Pakistan

Very quickly, PCCW noticed the large redirection of traffic to Pakistan Telecom, and had to terminate their BGP session with Pakistan Telecom to cut off the route from taking effect. This effectively cut off Pakistan from the rest of the internet while PCCW worked to filter out the malicious route.
YouTube routing restored

This isn’t the first time something like this has happened, and there isn’t a lot in place to prevent it from happening again. The nature of BGP is insecure, and the routers mainly go on a trust relationship that every provider is going to filter on the outside borders of their network and that inter-network traffic routes can be trusted.

It’s really cool when you start to grasp some of these concepts and how they work. A little over 2 months ago I had no clue what BGP even stood for, let alone that it routes the entire internet.

Good Game, Savvis.

PEER1 runs a redundant network utilizing 4 Tier 1 upstream providers. One of these providers, Savvis, had a major network fault last night. I started getting reports from customers that they are either unable to access their server, or their server was unable to access the internet outside the PEER1 network. After a bunch of diagnosis, I was able to trace the issue to our Savvis transit upstreams in the east coast.

I then checked a very handy service called Internet Pulse, and discovered that Savvis was having network issues pretty much all over North America.
Savvis Network Issues

Fantastic. However, with the help of the Engineering department, we were quickly able to react, and redirect traffic out the other upstreams, avoiding Savvis where possible. Most providers, no matter how big or small, will have network issues, faults, and generally not nice stuff happen to them.. However what makes the difference between the decent networks, and the great networks (like PEER1) is the ability to react quickly to changes in the structure of the network and minimize latency, packet loss, and downtime.