New Directions in Traffic Measurement and Accounting Cristian

New Directions in Traffic Measurement and Accounting Cristian Estan (joint work with George Varghese)

The Problem • Measurement and monitoring of network traffic required for Internet backbones. • Useful for short-term monitoring (e. g. , DOS attacks), traffic engineering (e. g. , rerouting), and accounting (e. g. , usage based pricing) • How can we do so without “tracking millions of ants to track a few elephants”?

State of the art – Cisco Net. Flow • Sample packets at high speeds; • Per flow information based on samples; • Aggregate (based on ASes, prefixes, ports) at the router; • Problems: inaccurate (due to sampling and loss), memory-intensive, slow (needs DRAM).

Towards a Net. Flow Alternative • Small Percentage of flows (elephants) account for large percentage of traffic. • Top 9% of flows account for 90% of AS pair traffic in backbones (Fang-Peterson). • Can we directly track flows that send say over 1% of link bandwidth without keeping track of all flows?

How to identify large flows? We introduce two new methods for this purpose: • Sample-Counting: uses sampling only to decide which flows to watch exhaustively. • Multistage filter: uses multiple hash tables allowing large flows to be identified while only allowing a small number of small flows (false positives) to pass through filter.

Identify large flows by sampling

Multistage filters

Operation of Sampled Net. Flow How accurate is Sampled Net. Flow? 1 gigabyte/100 megabytes of data Sampling one in 100 packets Error 1 GB 100 MB 1% 39. 24% 79. 03% 3% 1. 07% 41. 48%

Operation of our algorithms Error 1 GB 100 MB 1% 5. 6 E-32 0. 09% 3% 1. 8 E-94 6. 96 E-10 Sampling 1/1000 Error 1 GB 100 MB 1% 0. 08% 49. 69% 3% 4. 66 E-10 12. 25% Filter error: 0%

Comparison Accuracy Sampled Net. Flow Medium False neg. Few (high var. ) Few (low var. ) Never False pos. Few Few Memory Big Small Very small Low Medium Complexity Low Identify by sampling Good Multistage filters Good

High Speed Implementation? • John Huber of MMC Networks did a design of a chip doing filter counting. • 450, 000 transistors, under 1 watt of power, runs at OC-192 rates, 32 nsec per packet • Seems easily feasible to implement sample counting with similar complexity.

Potential Application: scalable threshold accounting • Measure flows sending over x% of link bandwidth using sample/filter counting. • Bill using flat fee + per byte charge for flows over x% • Track aggregates directly to avoid evasion using several flows, each < x% • Generalizes usage based (x = 0) and duration based (x = 100) pricing.

Conclusions • Paradigm shift for measurement by concentrating only on “heavy flows” • Two new techniques (sample and filter counting) with small memory footprints and provable performance. • Techniques make “threshold accounting” feasible, generalizing usage and duration based pricing.