Wise Traf View Flowbased Measurement and Analysis System

Wise* Traf. View Flow-based Measurement and Analysis System Pipefilters BOF, TIP 2004 Jan. 27, 2004 Hyungseok Chung ETRI ITU-T SG 3/WP 1

Topics Backgrounds Content-aware Application Recognition Wise*Traf. View Functionality GUI Snapshots & Demonstration 2

Common Measurement & Analysis Tools RTT, Packet loss Tool : ping, echoping, fping, gnuplotping, sting, … System : Ping-ER, AMP, IWR - use ping internally One-way delay, Packet delay Designated measurement BOX Surveyor, RIPE-NCC project, Smartbits Route discovery traceroute, skitter, mtr, ping plot, visualroute, neotrace, … Misc : remote traceroute execution server Throughput netperf, iperf, treno, tcpblast, tcpspray, ttcp, pchar Packet Capture & Analysis tcpdump, Coralreef, cflowd, snoop, ethereal, … Per Interface Traffic Volume and Errors MRTG 3

Why they are not enough? Capturing Packets in Current Networks High-speed networks (Mbps Gbps Tbps) High-volume traffic Streaming media (Windows Media, Real Media, Quicktime) P 2 P traffic Network Games Network Security Attacks Typical Flow-based Measurement Non-flow based measurement is not enough for the above requirements Typical Flow-based Measurement Typically a flow is defined as a set of packets passing an observation point in the network during a certain time interval and having a set of common properties 5 -tuple packet header fields are used for this New applications such as P 2 P, streaming and network games have characteristics of dynamic port allocation More Detailed Analysis is needed Typical Flow-based Measurement is not enough Need more detailed analysis depending on applications It may require content filtering 4

How does Wise* AS 100 Router Traf. View work? AS 200 Splitter Switch Network Operators Traffic Capture Agent raw streams of packets analysis result flow records Analysis Server 5

Application Recognition Limitations of port-based recognition The port database maintained by IANA doesn’t reflect real-world situation well Most newer applications simply do not register their ports Sometimes they even take advantage of well-known ports to pass thorough firewalls Most bandwidth hogs, nowadays, dynamically allocate ports They are not linked up with any fixed ports! 6

Real-world Situation Pos. Tech Traffic Breakdown Port/Application 80/HTTP Port-based Accounting 67 GB Contents-aware Accounting 59. 1 GB (11. 8% reduced) 21/FTP_CTRL 0. 29 GB 0. 28 GB 20/FTP_DATA 43 GB 42 GB 6 GB ? /FTP_DATA_PASSIVE n/a (14. 3% of FTP_DATA, 2% of the total volume) HTTP: 13. 2 MB 5003/? 692 MB BUGS_MUSIC: 420. 8 MB EDONKEY: 172. 3 MB etc. : 85. 7 MB - Pos. Tech Campus Network (24 h sum in May, 304 GB total volume) 7

Enhanced Application Recognition Wise*Traf. View utilizes some enhanced proprietary recognition mechanisms in a comprehensive way Internet Application Classification signature matching flow correlation dynamic port recognition and utilization some heuristics Not only capable of discriminating applications, but also their sub-flows e. g. , HTTP_REQ, HTTP_REP, HTTP_REQACK, etc. 8

Internet Application Classification Type S: Simple Application Type for an application which uses a well-known port number or which uses a registered port number but are popularly used Type P: Payload Application Type for an application which uses a registered or ephemeral port number but requires payload inspections for precise classification Type R: Reverse Application Type for an application which uses a registered or ephemeral port number but requires comparison with a correlated reverse flow for the precise classification Type C: Co-related Application Type for an application which uses a dynamic port number assignment Type U: Unknown Application Type for applications which do not use registered port numbers and do not belong to any of the four types mentioned above 9

Application Recognition Configuration Language (ARCL) application EDONKEY { application WWW { port_rep_name EDONKEY_DOWN port 4662 protocol TCP{ port_rep_name HTTP port 80 protocol TCP{ dst_disc_pattern=="0 xe 33 d 000000" in pkt 2 -3 at byte 0 - 4 decision_group HTTP_REQ_REP_ACK { decision_group EDONKEY_DOWN_REQ_REP_ACK { src_port >= 1024 dst_port == 80 dst_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 } } decision_group HTTP_REQ_ACK { decision_group EDONKEY_DOWN_REP_REQ_ACK { src_port == 80 src_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 dst_port >= 1024 }} }} port_rep_name HTTP_ALT port 8080 protocol TCP{ application FTP { src_disc_pattern=="HTTP" in pkt 0 -2 at byte 0 4 port_rep_name FTP port 21 protocol TCP{ ( dst_disc_pattern=="GET" in pkt 0 -3 at byte 0 src_ref_pattern=="r/227 Entering Passive Mode 10 || (d{1, 3}, (d{1, 4}), (d{1, 4}))/$src_port = dst_disc_pattern=="POST" in pkt 0 -3 at byte 0 atoi($1)*1024 + atoi($2)" in pkt any at byte 0 -35 induce - 10 ) FTP_DOWN_P decision_group HTTP_ALT_REQ_REP_ACK { decision_group FTP_REQ_REP_ACK { src_port >= 1024 dst_port == 8080 dst_port == 21 } } decision_group HTTP_ALT_REP_REQ_ACK { decision_group FTP_REQ_ACK { src_port == 8080 src_port == 21 dst_port >= 1024 }} }} } } 10

Application Recognition Example % ftp server % ls % passive % get wmggw. mp 3 % quit server. 21 (FTP_CTRL_REQ) client. 1302 server. 21 (FTP_CTRL_REP) 49152 server. 20 (FTP_DATA_DOWN) client. 1303 server. 20 (FTP_DATA_UP) client. 1306 server. 49152 (FTP_DATA_PSV_UP) server. 49152 (FTP_DATA_PSV_DOWN) client. 1306 0 2 4 6 8 10 12 Time (sec) 11

System Architecture Overview GUI Database ARCL Config-File Recognition and analysis Results (ODBC) Analysis Server Flow and packet Records (NFS) Capture Agent NIC . . . IPCAP Card Capture Agent . . . NIC . . . IPCAP Card 12

Capturing Internet Traffic Passive traffic capture No side-effect imposed on any network devices and links An optical or electric splitter, a. k. a. tap, is utilized Wise*Traf. View’s approach Splitters + Packet Capture Card + High Performance Capture Engine Adaptability maintained by supporting software-based capture as well PCAP (Packet Capture) library Doesn’t necessarily require a dedicated capture card; common Unix boxes can substitute the cards But yet, software-based capture is not equivalent to the card-based capture in terms of performance and functionality 13

Specialized Packet Capture Devices Link Signal Splitters Electrical Ethernet tap, DS-3 tap, etc. Optical ordinary optical splitter independent of physical and data-link layer protocols High Performance Packet Capture Cards Model A: for lower speed links Ethernet, Fast. Ethernet, DS-3/(E 3) Model B: for middle speed links ATM at OC-3, POS at OC-3, OC-12 (622 Mbps), and Giga. Ethernet 14

Flow Concept A “flow” is a sequence of packets whose <src and dst IP addresses, src and dst port numbers, and protocol> are all identical Why flow? The size of entire raw packet streams for a given unit time are prohibitively enormous to be analyzed in time Each individual packets in a flow contain duplicate information Packets in the same flow are correlated; we can identify more packets which were previously categorized as unknown application a flow a packet a distinctive signature of application “X” Now, these pkts can also be identified as “X” 15

Agent Side: Generating Flow Records Agents carry on simple filtering and signature matching functions generate flow records This procedure aggregates and organizes the traffic information and reduces the amount of traffic volume transferred to the server 16

Agent Structure 17

Server Side: AS and Country Mapping Identifying flow sources and destinations Both source and destination IP address of a flow are mapped to ASes and finally to countries This helps to locate the source and the sink of a flow Discrimination among transit, inbound, and outbound traffic flows 18

Analysis Server Structure 19

Configurability and Adaptability Why adaptability so important? The highly frequenting nature of Internet applications’ appearance and disappearance Swift mutation of applications Localization of the use patterns of applications Wise*Traf. View copes with the problem by introducing ARCL By taking advantage of ARCL, Wise*Traf. View doesn’t need to be re-built or re-installed by any module for extension can be easily reconfigured to handle a new application; modifying the configuration in ARCL and re-enforcing suffices 20

The Major Functionality of Wise*Traf. View Transparent Packet Capture complete independence of the existing networking equipment Flow-based Measurement and Analysis reduced load higher degree of recognition Understanding Application Specific Contexts by means of enhanced application recognition algorithms Scalable can scale up from tens of Mbps to Gbps supports various physical and data-link layer technologies Highly Extensible and Adaptable easy configuration with ARCL 21

User Interface Web-based Interface simple easy to use intuitive portable A web site for each measurement site can be easily established Authentication and authorization supported 22

Visualization: Traffic Breakdown Report 23

Visualization: Traffic Matrices 24

Platforms Hardware For lower speed links (<= 622 Mbps) capture agent n high performance PC: 2 * P-III 1 GHz+ CPU, 2 GB+ RAM, 30 GB+ HDD analysis server n high performance PC: 2 * P-IV 1 GHz+ CPU, 1 GB+ RAM, 100 GB+ HDD For Higher speed links ( > 1 Gbps) Dedicated Standalone Capture System Hardwised logics for supporting wire-speed processing Software capture agent Linux analysis server Linux, My. SQL 25

A Possible Deployment Scenario GPS Satellite Traffic Generator ISP 1 CDMA Basestation Traffic Center Traffic Generator ISP 2 Server ISP 3 Server Traffic Generator Server IX Router Traffic Center Analysis Server Measurement Agent 26

Thank you ! Q&A Contact: Hyungseok Chung, Taesang Choi, Taesoo Jeong {chunghs, choits, tsjeong}@etri. re. kr ITU-T SG 3/WP 1