Hawaii IPv 6 Deployment Experiences Alan Whinery U

Hawaii IPv 6 Deployment Experiences Alan Whinery U. Hawaii Chief Internet Engineer President, IPv 6 Forum Hawaii alan. whinery@ipv 6 hawaii. org Webcast for IPv 6 Hawaii. org (IEEE GLOBECOM 2009 IPv 6 FORUM, November 30, 2009)

Acknowledgements • Antonio Querubin – • Bill Becker – – • President, Global IPv 6 Forum David Lassner – • Network Administrator, Honolulu Community College Pacific Center for Advanced Technology Training Latif Ladid – • Systems Engineer, Lavanet U. Hawaii VP for Info Tech UH ITS Network Team 2

IPv 6 is not a “project” • We don't have an “IPv 6 person”, or an “IPv 6 team”, or an “IPv 6 initiative”. • It is our policy to deploy IPv 6 where we deploy IPv 4 • As upgrades or maintenance or changes are scheduled, IPv 6 is on the to-do list. 3

IPv 6 is not a “project” • Start now • Don't forklift • Consider IPv 6 in the course of your design and purchasing decisions • Work toward including IPv 6 in what you do. 4

Most things support IPv 6 Now • Clients – – Windows (XP, Vista, 7) Mac OS X • Router/Switch – Cisco, Juniper, Brocade (Foundry) • Server – Linux, Solaris, Win 2003/2008, Mac. OS – Apache, BIND, Postfix, Sendmail 5

UH System Network • 18 Campuses and learning centers on 6 islands • ~50 separate research/outreach facilities • ISP for – State Government – Bishop Museum • Internet 2 connector for – Dept. Of Education – NOAA NWS and NMFS 6

7

8

9

U. H. IPv 6 History • January 2004: “turned on” peerings with 3 RENs • IPv 6 Demos to Korea/Japan in 2004 • ITS network engineers and Honolulu Community College v 6 enabled ever since • Addressing moved from “provider” to self in December 2007 • Turned on State-wide OSPFv 3 and IPv 6 addressing in 2008 10

U. H. IPv 6 History • Initial campus deployment (2004) was through Cisco 7500 running an experimental IOS load – (Juniper supported in main release from 2003) • Distribution to user vlans was through a trunk into (inter-)campus VLANs • Although operational DNS servers running BIND were capable early, we waited for Infoblox appliance to do IPv 6 records – For user interface 11

U. H. IPv 6 History • Support for Cisco Catalyst 6500 was late in the game • Our production, mission-critical IPv 4 multicast infrastructure had us running nonstandard IOS in various places • Cisco 3550's are still not capable, but we anticipate IPv 6 and other factors will drive replacement of those devices 12

Watershed Moments • Acquired address block 12/2007 • (2008) Long-awaited Cisco IOS Catalyst 6500 and 3750, permitting us to “go native” on most core infrastructure • In June 2009, we added IPv 6 to our TWTC peering, which made “commodity” IPv 6 viable – Prior paths to commercial sites were circuitous – Netflix “Instant” over IPv 6 works well 13

Current UH IPv 6 Deployment Status • DNS Stores and answers forward and reverse IPv 6 records – Forward currently served in IPv 4 packets – Reverses currently served in IPv 4 and IPv 6 • State-wide, 99% of facility gateway routers have IPv 6 • 3 campuses have deployed IPv 6 in some form – Manoa, Honolulu CC and Maui CC • UH Manoa has native v 6 turned on to all user segments 14

v 6 Peerings • UH – – – TW Telecom Pacific Northwest Giga. Po. P DREN (Hawaii Intranet Consortium) Hawaii Internet e. Xchange AARNET (Seattle, Sydney, [LA]) • Lavanet – – – Hurricane Electric Sprint AT&T Verizon Business Toward. EX Hawaii Internet e. Xchange 15

Lava. net Status • • • Multiple v 6 peers network core (unicast and multicast) public servers DNS hosting Lavanet web site mail (SMTP/IMAP/POP) NTP most office workstations DSL or frame-relay (on request) 16

UH Short Term Goals • DNS – workflow • DNS • Address management – full services in IPv 4 packets – full services in IPv 6 packets • Dual stack all IP routing – Currently @ ~ 99% • Dual-stack all public-facing services – Currently @ ~ 5% 17

UH Short Term Goals Provide v 6/SLAAC to dual stack clients Provide Tunnel endpoints for 6 to 4, Teredo, ISATAP – – – Currently have in-state 6 to 4 with Lavanet Teredo is harder to affect ISATAP is a “good idea” 18

UH Medium Term Goals • Develop single stack v 6 capability • DHCP 6 • Translation • Proxying • Move email infrastructure to IPv 6

Mail 20

Mail • • • Careful planning will be necessary Training the email sysadmins will be necessary When email is not flowing, a crisis is occurring 21

The Big Island Router Memory Thing ($$$) • Problems on the south REN path from Mauna Kea Observatories on the Big Island of Hawaii – – – • Cisco 3750's with BGP tables of ~12, 000 prefixes Adding IPv 6 overwhelmed the fixed-config memory We were required to re-design and buy an extra Juniper M 7 i to support v 6 there This would have occurred regardless of IPv 6 – This is an expense to expedite, not simply to deploy 22

Cost • IPv 6 is not value-added software – – Cisco now has “feature parity” Juniper has stopped charging for it • Most of our costs, Lavanet's costs are in staff time and training. • Lavanet has participated in Opensource projects and contributed IPv 6 code • Cost can be controlled if you simply place IPv 6 on your requirements list, start requiring it, and don't panic • The Big Island router memory re-design is so far the highest-cost IPv 6 deployment measure (by far). 23

List of Problems: Native IPv 6 Deployment To User Networks • • Honest: not a single one. 24

UH Client OS Distribution Volume of HTTP GETs categorized by User-Agent 25

Out-Of-Box V 6 Readiness 26

Tunneled Forms of IPv 6 • Teredo (Significant incidental traffic) – Included with Windows XP, Vista, 7 – Used from behind NAT device (real/virtual) • 6 to 4 (some incidental traffic) – Included with Windows/Mac OS – tunnels via well-known address (i. e. 192. 88. 99. 1) • ISATAP (? ? traffic) – Included in Windows, some Cisco IOS – Tunnels through guessable domain name 27

Tunneled v 6 In The Wild • Sources of incidental 6 to 4, Teredo seem to be applications which require IPv 6, e. g. P 2 P clients – – Teredo can be used as an indicator of NAT There may be more insidious things present • Setting up local tunneling services can mitigate cost and issues for tunneled clients • Native IPv 6 deployment should stop 6 to 4, but Teredo will persist from behind NAT • Un-managed tunnels can represent increased attack surface and firewall by-pass. 28

UH Teredo Traffic • All clients use one of three Teredo servers: – 207. 46. 48. 150 (Microsoft Asia) – 213. 199. 162. 214 (Microsoft Europe) – 65. 55. 158. 80 (Microsoft USA) • • NAT causes Teredo traffic Virtual machine NATs cause Teredo traffic Exceedingly complicated Presumably initiated by an application install 29

SNMP • Cisco devised interim MIBs – • • Which persist in our current IOS Cisco's MIB support notes are fiction CISCO-IETF-IP-MIB 30

Graphing v 4/v 6 • • The old MRTG model of graphing interface Octet-counts doesn't do per protocol accounting Various non-optimal things can be done – • The following graphs were by using 8 “bpf” counters fed by individual filter expressions – – • ACLs feeding counters, etc No packet was examined Not a scalable approach Data represents 1 day on our TWTC v 6/v 4 31 peering

32

33

34

Performance • High-throughput transfers can exhibit throughput differences between v 4 and v 6 • Usually because v 6 is being processorswitched (configuration? ) • v 4 and v 6 paths to a resource often differ. • We have been happy if it worked at all – Now we need it to perform well • The time for optimization is at hand. 35

Comparing v 6/4 paths (UH) 36

Comparing v 6/4 paths (Lava. Net) 37

Every Firewall, ACL, etc • • Web server access controls acls firewall setups PHP code to return restrict content based on IP address • Max. Mind Geo. IP – is v 6 capable 38

IPv 6 Deployment Scenarios • Laissez-faire (ignoring your destiny) – Your resources will be unreachable via IPv 6 – external IPv 6 resources will either be unreachable or tunneled per client • You probably have significant tunneled traffic now – Your existing IPv 6 traffic will be high latency, poor performance • Possibly without the end-users' knowledge, they will simply blame you for bad performance • Or they will blame IPv 6 and turn it off – Requires IPv 4 addresses • “I don’t believe the v 6 transition is occuring” – Means “I choose denial instead of participation” 39

IPv 6 Deployment Scenarios • IPv 6 Only ☺ - Client support sketchy - Translation necessary to reach IPv 4 Internet + Some value in enabling v 6 -only servers • Dual-stack ☺☺☺ Client support good + No translation necessary + Serves potential v 6 -only groups - Requires IPv 4 addresses + 40

Stateless Auto-configuration (SLAAC) • Many operating systems have IPv 6 turned on by default • With SLAAC, if your router interface is using v 6, then you are too. You may use v 6 without realizing it • Your machine determines your IPv 6 address, and adds it to the prefix advertised by the router • Some OS build the RH 64 bits using the MAC address • Others will make up random (currently only Vista and W 7) – complicates address accounting/management 41

Getting a DNS Server address • Stateless auto-configuration gets you an address and gateway • But no DNS server • Of course, if you have DNS through IPv 4, you will learn v 6 addresses through that DNS server • Currently, the only way for a v 6 -only host to auto-learn the name server address is DHCPv 6 • Attachments to SLAAC are proposed – RFC 5006 (IPv 6 Router Advertisement Option for DNS) 42

IPv 6: Apple OSX 10. 4+ • On by default • Missing DHCP 6 • Can't specify v 6 address for networked printer, because the preferences pane for printer set-up considers a colon ‘: ’ as preceding a port number (? 10. 6) – Printer can, however, be specified by name 43

Apple OS X Applications • Firefox – once required v 6 “turn on” – This seems to have changed • • • Safari – does browse IPv 6 ping – works with separate “ping 6” traceroute – works with separate “traceroute 6” SSH client – works telnet – works to router: fe 80: : 209: 7 bff: fedc: 400%en 0 email – no server to test to yet 44

IPv 6: Windows XP (SP 2+) • You can add it to an interface with the interfaces “Properties” pane, just like IP(v 4) or IPX/SPX or Net. BIOS • Once added, there is no GUI config, although some things can be accomplished with the command line • Will not do DNS queries in IPv 6 packets • Will receive IPv 6 info from DNS in IPv 4 packets • Is Ultimately doomed. 45

Windows XP Applications • Firefox – will browse IPv 6 • IE 7 – will browse IPv 6 • ping – works – Tries first address as returned by DNS • tracert – works – Tries first address as returned by DNS • Telnet – doesn’t appear to work • Thunderbird – no server to test to yet 46

IPv 6: Windows Vista and 7 • On by default • Does DHCP 6 • There have been some problems – Passing of ICMP 6 messages to applications 47

Windows Vista Applications • Firefox – will browse IPv 6 • IE 7 – will browse IPv 6 • ping – works – Tries first address as returned by DNS • tracert – works – Tries first address as returned by DNS • Telnet – doesn’t appear to work • Thunderbird – no server to test to yet 48

IPv 6: Ubuntu 8 • On by default • Does DHCP 6, if you install it • Since Linux (and BSD OS) are typically used for reference implementations, support is pretty good 49

Ubuntu Linux Applications • • Firefox – will browse IPv 6 ping – works as “ping 6” traceroute – works as “traceroute 6” Telnet – doesn’t appear to work • Linux is a kernel. – Linux distributions are operating systems. They differ as to what apps they provide for various roles. – “Distributions” means, Red Hat, Ubuntu, Suse, Debian, Slackware, etc. 50

Steps To Dual-stack IPv 6/(4) Deployment • • Get addresses Configure routers Configure DNS Configure public-facing services (web/mail/etc) • Configure clients – Probably only necessary to the extent that you have Windows XP 51

Steps to single-stack IPv 6 Deployment • • Get addresses Configure routers Configure DNS (in v 6 only) Configure public-facing services (web/mail/etc) • Provide gateway to v 4 • Configure clients – Need DNS server entry – Manual or DHCP 52

IVI V 6 to V 4 gateway • • Implementation of Internet Draft From CERNet and 清華大學 (Beijing) License unclear Involves patches to out-dated kernel (2. 6. 18) – Which doesn’t compile under current libc/gcc • I have seen it work well, in February 2009, at Joint Techs, Texas A&M 53

Trying Out Your IPv 6 • It’s hard to know whether you are using it. – Show. IP add-on for Firefox helps – But it isn’t perfect • When the OS provide resolution and connectivity – The applications still may • Or may not 54

Perl Programming • Can't simply handle addresses as integers anymore, without a 128 -bit data type. – Math: : Big. Int – Net: : IPv 6 Addr (eats RFC 1884 addr format) 55

pcap filters expressions Used to specify packet captures: with tcpdump, wireshark, ngrep, etc • tcpdump udp and 'udp[16: 4] = 0 x 20010000' – capture packets from Teredo clients • tcpdump net 20 C 0: FFEE: : /32 – capture for the specified 6 net • tcpdump ip 6 – capture all ipv 6 traffic • tcpdump proto ipv 6 – capture ipv 6 encapsulated in IPv 4 OR IPv 6 56

Dirty Tricks: OK! • You can direct the DNS AAAA record for an existing IPv 4 services to a separate device – Use Apache as a transparent proxy to make it look like the content has a v 6 address – This is GREATLY simplified if the content has an alternate name – Some scenarios/services can simply use a different host 57

Dirty Tricks: OK! • Nothing says that the interface or device that offers services via IPv 6 is required to be the same as the one that offers those services over IPv 4 58

Multicast • For IPv 6 Multicast, the capabilities of Layer 2 switches become critical – Currently our v 6 Multicast is only experimental – We do not have plans to migrate our multicast applications 59

What can I reach with IPv 6? More and more. See http: //ipv 6 hawaii. org “Things You Can Reach With IPv 6” 12/8/2020 60

Returning To Work On Monday • Hawaii IPv 6 Forum – http: //ipv 6 hawaii. org 61