Net Flow and flowtools OARnet Mark Fullmer mafsplintered

Net. Flow and flow-tools @ OARnet Mark Fullmer maf@splintered. net

Agenda • OARnet - who we are, what we do. • Flow-tools summary - software we use to collect and process Net. Flow data. • Net. Flow deployment at OARnet. • How we use Net. Flow. Detailed examples with flow-tools. Hopefully some tips that can be used or passed on to customers.

OARnet • Internet access to about 100 of Ohio’s colleges and universities. • Some commercial clients / co-lo services. • Internet 2 / Abilene access to member school’s in Ohio. • Sink over 1 Gb/s of traffic from the Internet during peak hours.

OARnet • Multiple Transit providers, access to two of the NAPs, OC 48 to Abilene. • In the middle of a state wide network rebuild, moving from ATM circuits over leased lines to POS and Gig. E over DWDM using Cisco 15454’s.

Flow-tools • Started back in June of ‘ 96 working from Darren Kerr’s fdg. c (a very simple example of displaying Net. Flow v 1 data). No commercial or open source Net. Flow collectors available at the time. • Released as an open source toolkit for working with Net. Flow data from Cisco routers. At the time mostly used internally at Ohio State University, usually for network forensics.

Flow-tools • Still available as open source, active development on and off as resources (time) are available. Keep working it as long as people are interested. • Many other options available now for collectors and post processing both open source and commercial. • A lot of sites use it, over 700 subscribers to the support mailing list.

Net. Flow deployment • Collector running Free. BSD and flow-capture at each POP (distributed collection). • Sampled Net. Flow data feeds (where software permits). • Flow-capture stores the exports to disk with line compression at 5 minute intervals. • Some reports run on the collectors, some at back at a central server (combination of distributed and centralized report generation).

Net. Flow deployment OC 48’s to other POPs. Core router Flow collector. 80 GB RAID storage. Aggregation routers.

Net. Flow At Work

Traffic Engineering • We have multiple transit providers with various contractual obligations. • With provider T we must purchase a minimum of 300 Mb/s / month. • Provider Q recently reduced their prices to half of provider T. • So we want to minimize our transit costs by using provider T for no more than 300 Mb/s during peak hours until contract runs out (or they reduce their prices).

Traffic Engineering Clearly above 300 Mb/s on on outbound traffic, a little over on inbound.

Traffic Engineering • 170 prefixes that are announced to the internet. • Prefix length does not necessarily correlate to traffic load. I. e. some smaller school’s with /16’s. • So to influence provider T to send us less traffic to a prefix, pad the AS path.

Traffic Engineering • Use Net. Flow to determine which prefixes to pad. • Select traffic that has an input interface of the provider circuit. • Add local masks. • Scale the packets and octets by 100 (sampling rate is 1/100). • Summarize on destination IP & mask (zero out the host bits).

Traffic Engineering Collected flows(flow-capture). Select a days data (flow-cat) Filter on provider interface (flow-nfilter) Fix masks (flow-mask) Summarize by destination IP & mask (flow-report) Format tabular data (flow-rptfmt)

Traffic Engineering Flow-nfilter configuration: # if. Index 46 is the POS interface to the Internet filter-primitive CLMBO-R 4 -INTERNET type ifindex permit 46 # Match on traffic to the Internet by POS interface. filter-definition CLMBO-R 4 -INTERNET-OUT match output-interface CLMBO-R 4 -INTERNET # Match on traffic from the Internet by POS interface. filter-definition CLMBO-R 4 -INTERNET-IN match input-interface CLMBO-R 4 -INTERNET

Traffic Engineering Flow-mask configuration: mask-definition AS 600 -ADV-ALL prefix 128. 146. 0. 0/16 16 prefix 140. 254. 0. 0/16 16 prefix 164. 107. 0. 0/16 16 prefix 192. 12. 205. 0/24 24 prefix 192. 68. 143. 0/24 24 prefix 128. 156. 0. 0/16 16 prefix 139. 88. 0. 0/16 16 prefix 192. 55. 90. 0/23 23 prefix 129. 1. 0. 0/16 16 prefix 129. 137. 0. 0/16 16. . .

Traffic Engineering Flow-report configuration: stat-report CLMBO-R 4 -R 8 -FROM-INTERNET-BY-DESTINATION-PREFIX type ip-destination-address filter CLMBO-R 4 -INTERNET-IN ip-destination-address-format prefix-mask scale 100 output options +header, +xheader sort +octets fields -duration, +bps, +pps path @P/CLMBO-R 4 -FROM-INTERNET-BY-DESTINATION-PREFIX. txt output sort +octets fields -duration path |flow-rptfmt -Fip-destination-address, flows, octets, packets > @P/CLMBO-R 4 -FROM-INTERNET-BY-DESTINATION-PREFIX. fmt

Traffic Engineering Run the report: #!/bin/sh FLOW_DATA=. . /CLMBO-R 4/2004 -11 -08/* flow-cat $FLOW_DATA | flow-report -s report. cfg -Sdaily-summaries -v. P=daily-summaries/

Traffic Engineering Flow-report output: # ---- Report Information --- --# build-version: flow-tools 0. 68 p # name: CLMBO-R 4 -FROM-INTERNET-BY-DESTINATION-PREFIX # type: ip-destination-address # scale: 100 # options: +header, +xheader # ip-dst-addr-type: prefix-mask # sort_field: +octets # fields: +key, +flows, +octets, +packets, +other # filter: CLMBO-R 4 -INTERNET-IN # records: 139 # first-flow: 1099890083 Mon Nov 8 00: 01: 23 2004 # last-flow: 1099976485 Tue Nov 9 00: 01: 25 2004 # now: 1100221034 Thu Nov 11 19: 57: 14 2004 # # mode: streaming # compress: off # byte order: little # stream version: 3

Traffic Engineering Flow-report output: XXX # recn: ip-destination-address*, flows, octets, packets 164. 107/16, 1555282, 227567453500, 369333900 64. 247. 64/18, 2276007, 145782119200, 379959000 129. 137/16, 678257, 132932164100, 179344100 134. 53/16, 786497, 110197887600, 175669700 198. 30/16, 523565, 91210633100, 126912900 129. 1/16, 651195, 77953466900, 104152800 130. 101/16, 389654, 60952342600, 87133000 132. 235/16, 379746, 55747741000, 102050900 140. 254/16, 328851, 54698141100, 65434300 131. 183/16, 416366, 53890273300, 76920100 128. 146/16, 511276, 52719529000, 92782100 199. 218/16, 746850, 52400585900, 165697600 130. 108/16, 308359, 49463259900, 66772600 205. 133/16, 441714, 44835214400, 71746400 206. 244/16, 268486, 37445229500, 50199700 206. 21/16, 380376, 35449280700, 59676600 …

Traffic Engineering Flow-report-fmt output: # ['. /flow-rptfmt', '-Fip-destination-address, flows, octets, packets'] ip-destination-address flows octets packets 164. 107/16 1555282 227567453500 369333900 64. 247. 64/18 2276007 145782119200 379959000 129. 137/16 678257 132932164100 179344100 134. 53/16 786497 110197887600 175669700 198. 30/16 523565 91210633100 126912900 129. 1/16 651195 77953466900 104152800 130. 101/16 389654 60952342600 87133000 132. 235/16 379746 55747741000 102050900 140. 254/16 328851 54698141100 65434300 131. 183/16 416366 53890273300 76920100 128. 146/16 511276 52719529000 92782100 199. 218/16 746850 52400585900 165697600 130. 108/16 308359 49463259900 66772600 ……….

Traffic Engineering Flow-report-fmt output (top 10 percent totals): # ['. /flow-rptfmt', '-m 10', '-p', '-Fip-destination-address, flows, octets, packets'] ip-destination-address flows octets packets 164. 107/16 10. 906643 12. 768776 12. 746539 64. 247. 64/18 15. 960832 8. 179813 13. 113235 129. 137/16 4. 756376 7. 458804 6. 189566 134. 53/16 5. 515425 6. 183187 6. 062754 198. 30/16 3. 671576 5. 117815 4. 380048 129. 1/16 4. 566600 4. 373958 3. 594546 130. 101/16 2. 732506 3. 420027 3. 007155 132. 235/16 2. 663024 3. 127998 3. 522005 140. 254/16 2. 306116 3. 069105 2. 258284 131. 183/16 2. 919828 3. 023775 2. 654685

Traffic Engineering • A day long summary usually works for situations like this, but we may want to look at a time-series report over a day period (or longer). • Gnuplot, or rrdtool, or gdchart (among others) can be used for this. • Flow-rpt 2 rrd will convert the flow-report output to rrd format. For automated reports rrdtool works well since it bounds the storage requirements.

Traffic Engineering #!/bin/sh FLOW_DATA=. . /CLMBO-R 4/2004 -11 -08/* for name in $FLOW_DATA; do echo working. . . $name flow-report -s report. cfg -S 5 min-summaries -v. P=5 min-summaries < $name done for name in 5 min-summaries/*; do echo working. . . $name. /flow-rpt 2 rrd -K as 600. keys -p rrds < $name done

Traffic Engineering During peak hours moving traffic for 164. 107/16 elsewhere will drop inbound load by between 20 and 30 Mb/s.

Traffic Engineering • Minor changes to the report to get other results • Outbound traffic by source prefix (busy sources in OARnet). • Outbound traffic by destination prefix or AS (where are our sinks). • Inbound traffic by source prefix (where do our sinks draw traffic from). • Source / Destination pairs.

Traffic Engineering stat-report CLMBO-R 4 -TO-INTERNET-BY-SOURCE-PREFIX type ip-source-address filter CLMBO-R 4 -INTERNET-OUT ip-source-address-format prefix-mask scale 100 output options +header, +xheader sort +octets fields -duration path @P/CLMBO-R 4 -TO-INTERNET-BY-SOURCE-PREFIX. txt

Traffic Engineering stat-report CLMBO-R 4 -TO-INTERNET-BY-DESTINATION-AS type destination-as filter CLMBO-R 4 -INTERNET-OUT scale 100 output options +header, +xheader sort +octets fields -duration path @P/CLMBO-R 4 -TO-INTERNET-BY-DESTINATION-AS. txt

Traffic Engineering # ['. /flow-rptfmt', '-p', '-n', '-Fdestination-as, flows, octets, packets'] destination-as flows octets packets SCRR-10796 3. 769746 7. 881135 4. 664673 ATT-INTERNET 3 4. 636261 5. 987038 4. 844542 DNEO-OSP 1 3. 066557 3. 309393 3. 005959 CCINET-2 2. 258435 2. 896718 2. 529854 ROGERS-AS 1. 537620 2. 664075 1. 913102 CHARTER-NET-HKY-NC 2. 369852 2. 100978 2. 175202 RR-CINCINNATI-ASN-01 0. 732147 1. 989616 1. 190482 SCRR-11427 1. 588737 1. 646306 1. 647938 VIDEOTRON-LTEE 0. 735799 1. 323654 1. 078804 SCRR-11426 1. 183881 1. 289154 1. 234044

Traffic Engineering During peak hours moving traffic destined to AS 10796 elsewhere will drop outbound load by between 20 and 30 Mb/s.

RPF checks • Provider Q wants to turn on strict RPF checks on our peerings. • For a variety of reasons this won’t work for us (load distribution for one), but we do agree to implementing a traffic filter. • There are places OARnet has not enforced RPF checks, mostly for historical reasons.

RPF checks • So want to understand the impact of applying the traffic filter before applying it to prevent dropping valid customer traffic. • Expect problems mostly to be multi-homed clients who may not be advertising all their address space to us. • …Plus unexpected surprises.

RPF checks Collected flows(flow-capture). Select a days data (flow-cat) Filter on provider interface (flow-nfilter) Filter on announced address space Summarize results with flow-report Format tabular data (flow-rptfmt)

RPF checks Flow-nfilter configuration: #; OSU filter-primitive AS 600 -ADV-ALL type ip-address-prefix default permit deny 128. 146. 0. 0/16 deny 140. 254. 0. 0/16 deny 164. 107. 0. 0/16 deny 192. 12. 205. 0/24 deny 192. 68. 143. 0/24 deny 128. 156. 0. 0/16 deny 139. 88. 0. 0/16 deny 192. 55. 90. 0/23 deny 129. 1. 0. 0/16 deny 129. 137. 0. 0/16. . .

RPF checks Flow-report configuration: stat-report CLMBO-R 4 -RPF-FAIL type ip-source-address filter CLMBO-R 4 -RPF-FAIL ip-source-address-format prefix-mask scale 100 output options +header, +xheader sort +octets fields -duration path |. /flow-rptfmt stat-definition CLMBO-R 4 -RPF-FAIL report CLMBO-R 4 -RPF-FAIL

RPF checks Results % FLOW_DATA=. . /clmbo-r 4/2004 -11 -08/* % % flow-cat $FLOW_DATA | flow-report -s report. cfg -SCLMBO-R 4 -RPF-FAIL ip-source-address flows octets packets 198. 234/16 24689 3478979000 4894300 66. 144/15 22661 1422430800 3345200 204. 130. 176/24 4328 398581200 737000 156. 63/16 2387 279678500 387900 208. 44/14 992 6527900 105700 65. 112/12 845 4986800 88900 66. 194. 129. 106/32 381 2150400 38400 53/8 91 1886400 39300 18/8 90 1737600 36200 45/8 89 1660800 34600 38/8 91 1622400 33800

Access Lists • Same procedure can be used to test a firewall or access list before applying it to live traffic. • Determine what the impact of a change will be before impacting customers.

Traffic Classification • Abilene / Internet 2 is a high speed research network that OARnet provides connectivity to for Ohio school’s. • Only some school’s participate. • Of those school’s that participate they can sponsor smaller school’s or other institutions for example a medical college.

Traffic Classification • The participants all share the cost for an OC 48 to Abilene. They want to know who’s using it both by school and group. • School’s may have multiple networks. • Not all school’s have AS numbers. • School traffic is combination of prefixes. • Group traffic is combination of schools.

Traffic Classification • Flow-tag adds a new 32 bit source tag and destination tag to the flows. • Tags can be set based on criteria in the flow, such as source or destination prefix. • Reports can then be run on the tagged traffic.

Traffic Classification Collected flows(flow-capture) Select a months data (flow-cat) Add tags based on interface and prefix (flow-tag) Filter on provider interface (flow-nfilter) Summarize results with flow-report Graph tabular data (gnuplot)

Traffic Classification # # tag format # #0 7 15 23 31 # 0000 0000 (32 bits) # RRRRRRR TTTT NNNNNNNNNN # | | | Site name # | | Site type # | Reserved # ID Name #----------------# 0 x 0001 OSU # 0 x 0002 CWRU # 0 x 0003 BGSU # 0 x 0004 UC # 0 x 0005 UAKRON # 0 x 0006 WRIGHT # 0 x 0007 KENT … # ID Type #------------# 0 x 01 Participant # 0 x 02 SEGP # 0 x 03 Sponsored-Participant # 0 x 04 Gigapop # 0 x 05 MULTICAST

Traffic Classification tag-action OHIO-GIGAPOP_DST type dst-prefix # OSU match 128. 146/16 set-dst 0 x 010001 match 164. 107/16 set-dst 0 x 010001 match 140. 254/16 set-dst 0 x 010001 match 192. 153. 26/24 set-dst 0 x 010001 match 199. 18. 22. 24/30 set-dst 0 x 010001 match 199. 18. 4. 24/30 set-dst 0 x 010001 # CHMCC match 204. 152. 48/24 set-dst 0 x 030014 match 204. 152. 49/24 set-dst 0 x 030014 match 205. 142. 196/24 set-dst 0 x 030014 match 205. 142. 197/24 set-dst 0 x 030014 match 205. 142. 198/24 set-dst 0 x 030014 match 205. 142. 199/24 set-dst 0 x 030014 match 199. 18. 107. 220/30 set-dst 0 x 030014

Traffic Classification stat-report TS-CLMBQ-R 2 -FROM-ABILENE-BY-TAG-GROUP type destination-tag tag-mask 0 x 00 FF 0000 filter CLMBQ-R 2 -ABILENE-IN scale 100 output options +header, +xheader sort +octets fields -duration path @P/CLMBQ-R 2 -FROM-ABILENE-BY-TAG-GROUP. %j. txt stat-definition clmbq-r 2 -daily-summaries tag OHIO-GIGAPOP report TS-CLMBQ-R 2 -FROM-ABILENE-BY-TAG-GROUP report TS-CLMBQ-R 2 -FROM-ABILENE-BY-TAG-CUSTOMER report TS-CLMBQ-R 2 -TO-ABILENE-BY-TAG-GROUP report TS-CLMBQ-R 2 -TO-ABILENE-BY-TAG-CUSTOMER

Traffic Classification

Traffic Classification

Traffic Classification • Groups of IP addresses - campus department traffic summaries. • Individual IP addresses - Find busy hosts on a campus. • Ports and protocols - find all the SMTP clients or hosts using file sharing services.

Traffic Classification • We have the ability to provision dedicated l’s between many of the larger school’s. • Cost money…Some clients want to know if they should be purchasing dedicated bandwidth between each other.

Traffic Classification

Traffic Classification

Traffic Classification

Traffic Classification

Enterprise/Campus proactive Security • A lot of low hanging fruit. • Hosts with high packet/octet rates to/from Internet. • Hosts that connect to a large number of external sites. • Hosts that use many ports. • Usually not useful in our environment -problems easy to find, hard to fix.

Enterprise/Campus security • Usually use Net. Flow to diagnose problems on the fly or historical events. • Customer calls and wants to know why their T 1 is full. Usually network scans or Do. S events. • A few weeks of historical data is very useful. If there is one compromised host there are usually others…

Enterprise/Campus security Internet Net. Flow v 5 exports Campus Network Hosts with known security issue(s) Net. Flow collector & Archive.

Enterprise/Campus security Scan & compromise Internet Net. Flow v 5 exports Campus Network Compromised hosts. Net. Flow collector & Archive.

Enterprise/Campus security Remote trigger attack Internet Net. Flow v 5 exports Campus Network Compromised hosts. 1/5 used to attack. Net. Flow collector & Archive.

Enterprise/Campus security • For some reason alerted to attack - proactive measures, local network slowdowns, victim network complaints, etc. • Find and disable compromised host. • Use Net. Flow archive to find suspicious traffic from attacker. • Look back in history for other traffic from that IP. Find other compromised hosts before they are used in future attacks. • At least disable the attacker IP to campus.

Enterprise/Campus security • Good success rate at retroactively identifying worms and viruses. • Not too smart, usually have same packet signature. Many 1 packet per flow from an infected host, or constant packet size, constant source or destination port. • Can at least identify infected machines that are causing disruption to a campus. • One would guess as more tools like Net. Flow are deployed miscreants will make efforts to hide signatures.

Flow-tools • A lot more is possible. • Flow-nfilter can filter on any fields in Net. Flow v 1 - v 8 flows. • Flow-report has about 80 built in reports. • Usually reports and graphs are generated automatically. • On old PC with Free. BSD/Linux is enough to get started.

Flow-tools • Contributed software such as Dave Plonka’s Flow. Scan and Perl module http: //net. doit. wisc. edu/~plonka • Other information at http: //www. splintered. net/sw/flow-tools

Thanks…