Akamai War Stories Bruce Maggs Akamais First Network

Akamai “War” Stories Bruce Maggs

Akamai’s First Network Connection We moved into our offices at 201 Broadway at midnight, December 1, 1998, and built our first cluster that night. An important potential investor was coming to visit on December 2. But our Internet service provider didn’t show up on December 1! We had to engineer a solution!

Lost in Space • The most worrisome attack we ever faced: • One of our servers was receiving properly • • • authenticated messages from an unknown host Fortunately, the messages were not formatted properly and were discarded After two days of investigation, we discovered that the “attacker” was an old Akamai server that we had lost track of It had been sending these messages for months! 4

David is a Night Owl • Your servers aren’t responding! • Why don’t you support half-closed • • connections? Why don’t you support “transactional” TCP? (Why would transactional TCP be bad for Akamai? ) 5

The Dreaded Double Header • http: //images. xyz. com/logo. gif - customer has delegated images. xyz. com to Akamai, registered image server • http: //images. xyz. com/logo. gif - didn’t work for Dave, but worked for me! • Akamai server strips off first header, sends GET /images. xyz. com/logo. gif to customer image server • 5 of 8 customer image servers had been patched to ignore /images. xyz. com 6

Steve can’t see the new Powerbook • Steve’s assistant Eddie explains the • • • problem I spend all night poring through the logs Eddie sneaks into Steve’s office Mystery solved 7

Packet of Death • Akamai servers take care of each other • A router in Malaysia is taking down our • • whole system! The mysterious 570 -byte MTU The “final” Linux 2. 0 kernel isn’t so final • 2. 0. 36 (Nov. 1998) 2. 0. 37 (June 1999) 8

The “Magg Syndrome” • We “hijack” a customer’s site? • I become the most hated person on the • • Internet We isolate the problem (nine months of work) Nobody cares? 9

BIND Miseries • • Open-source DNS server code Messy, buggy implementations Our customers still run old versions! BIND 4. 8 TTL issue • Refresh attempt when 15 minutes left • Success if new list of IP’s overlaps with old • list of IP’s Otherwise, refuse to resolve for next 15 minutes! 10

We can’t see the pictures on our site! • Customer can view its web pages, but • • • none of the pictures served by Akamai are displayed. The rest of the world can’t access the site. (Because it was hosted in their office building, which lost Internet access. ) Lingering doubt about the day Akamai didn’t serve the images… 11

Don’t do this at home • • • Irate end user threatens to go to police Akamai is attacking my home system! It’s in the logs. It all began in a Yahoo! chat room Have your lawyers call our lawyers 12