Data Streaming in Computer Networking Cristian Estan George

  • Slides: 30
Download presentation
Data Streaming in Computer Networking Cristian Estan, George Varghese University of California, San Diego

Data Streaming in Computer Networking Cristian Estan, George Varghese University of California, San Diego

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm:

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm: packet scheduling example u l New streaming problems Detecting malicious traffic u Understanding networkloads u June 8, 2003 Data streaming in computer networking - MPDS 2003 2

Internet service model Source port Destination port Source IP address Destination IP address Data

Internet service model Source port Destination port Source IP address Destination IP address Data Header Flow Internet June 8, 2003 Data streaming in computer networking - MPDS 2003 3

Traditional router functions ? IP Lookup Incoming 1 Outgoing 1 Incoming 2 Outgoing 2

Traditional router functions ? IP Lookup Incoming 1 Outgoing 1 Incoming 2 Outgoing 2 Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003 4

Traditional router functions Out 2 IP Lookup Incoming 1 Outgoing 1 Incoming 2 Outgoing

Traditional router functions Out 2 IP Lookup Incoming 1 Outgoing 1 Incoming 2 Outgoing 2 Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003 5

Traditional router functions Incoming 1 Incoming 2 Incoming 3 June 8, 2003 Out 2

Traditional router functions Incoming 1 Incoming 2 Incoming 3 June 8, 2003 Out 2 Out 3 Switching Out 3 Out 1 Out 2 Data streaming in computer networking - MPDS 2003 Outgoing 1 Outgoing 2 Outgoing 3 6

Traditional router functions Scheduling Incoming 1 Outgoing 1 Flow 2 Incoming 2 Flow 3

Traditional router functions Scheduling Incoming 1 Outgoing 1 Flow 2 Incoming 2 Flow 3 Incoming 3 June 8, 2003 Outgoing 2 Outgoing 3 Data streaming in computer networking - MPDS 2003 7

Traditional router functions Scheduling Incoming 1 Outgoing 1 Flow 2 Flow 3 Incoming 2

Traditional router functions Scheduling Incoming 1 Outgoing 1 Flow 2 Flow 3 Incoming 2 Outgoing 2 Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003 8

Rules of the game l Wire speed processing u u l What does this

Rules of the game l Wire speed processing u u l What does this mean for algorithms? u u l At 40 gigabits/s 8 nanoseconds per packet - need fast SRAM Limited SRAM (say 32 megabits) but millions of flows Low worst case complexity bounds Low bounds on the amount of memory used Differences from databases u u u June 8, 2003 One pass vs. multiple passes Worst case vs. average case Small constants vs. asymptotic complexity Data streaming in computer networking - MPDS 2003 9

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm:

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm: packet scheduling example u l New streaming problems Detecting malicious traffic u Understanding networkloads u June 8, 2003 Data streaming in computer networking - MPDS 2003 10

Iteration paradigm l l Many networking algorithms use iteration in time Way to allow

Iteration paradigm l l Many networking algorithms use iteration in time Way to allow multi-pass algorithms without storing input by assuming inputs do not change quickly Many examples (MULTOPS for Do. S detection [Gil 01], CSFQ for scheduling [Stoica 98]) Would be nice to formalize tradeoff between quality of results and drift rate of input June 8, 2003 Data streaming in computer networking - MPDS 2003 11

Example: Core Stateless FQ R R Mark rate R June 8, 2003 R If

Example: Core Stateless FQ R R Mark rate R June 8, 2003 R If R>F drop with probability 1 -F/R Iteratively compute fair share F Data streaming in computer networking - MPDS 2003 12

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm:

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm: packet scheduling example u l New streaming problems Detecting malicious traffic u Understanding networkloads u June 8, 2003 Data streaming in computer networking - MPDS 2003 13

New streaming problems l Detecting malicious activity u u u l Flooding (denial of

New streaming problems l Detecting malicious activity u u u l Flooding (denial of service attacks) Worms Scans looking for vulnerable servers Understanding workloads u u u June 8, 2003 Billing Planning network growth Application mix Data streaming in computer networking - MPDS 2003 14

Detecting malicious traffic l Well defined building blocks u Detecting large aggregates » u

Detecting malicious traffic l Well defined building blocks u Detecting large aggregates » u Counting active flows in an aggregate » l Similar to iceberg queries Similar to counting distinct values Many open problems: e. g. detect worms and Do. S attacks (not clear what is right formal problem statement) June 8, 2003 Data streaming in computer networking - MPDS 2003 15

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm:

Talk structure l Traditional streaming in networking Rules of the game u Iteration paradigm: packet scheduling example u l New streaming problems Detecting malicious traffic u Understanding networkloads u June 8, 2003 Data streaming in computer networking - MPDS 2003 16

Informal problem definition Analysis Traffic reports Applications: 50% of traffic is Kazaa Terabytes of

Informal problem definition Analysis Traffic reports Applications: 50% of traffic is Kazaa Terabytes of measurement data Sources: 20% of traffic comes from Steve’s PC June 8, 2003 Data streaming in computer networking - MPDS 2003 17

Informal problem definition Analysis Traffic reports 20% is Kazaa from Steve’s PC Terabytes of

Informal problem definition Analysis Traffic reports 20% is Kazaa from Steve’s PC Terabytes of measurement data June 8, 2003 50% is Kazaa from the dorms Data streaming in computer networking - MPDS 2003 18

Formal problem definition l Define clusters: u u u l Threshold clusters: u l

Formal problem definition l Define clusters: u u u l Threshold clusters: u l Atoms: fields 1 to n with hierarchies in each field including * Cluster: intersection of one set from each field hierarchy Example: Source=*, Destination=CS Net, App= Email Report traffic clusters above threshold T (e. g. 1% of traffic) Omit redundant clusters: u June 8, 2003 Compression rule: remove general clusters from report when its traffic can be inferred (up to error T) from on nonoverlapping more specific clusters Data streaming in computer networking - MPDS 2003 19

Solution status l The good: u u u l The bad: u u l

Solution status l The good: u u u l The bad: u u l Offline tool Auto. Focus; SIGCOMM 2003 paper Detected worm, busy servers, squid cache, etc. Network managers like it Takes long: 3 hours at T=0. 5% for one day trace Needs much memory 300 Mbytes The wanted: u June 8, 2003 Streaming algorithm - we invite improvements Data streaming in computer networking - MPDS 2003 20

Conclusions l l New rules: strict constraints on algorithms running in routers Iteration in

Conclusions l l New rules: strict constraints on algorithms running in routers Iteration in time: can give simple algorithms, but needs more formalization as to quality of results General open problems: many challenges in detecting malicious traffic such as worms and Do. S attacks Specific open problem: computing traffic cluster reports in streaming fashion June 8, 2003 Data streaming in computer networking - MPDS 2003 21

Thank you! Alg ori thm s ? Databases g Ne June 8, 2003 in

Thank you! Alg ori thm s ? Databases g Ne June 8, 2003 in ork tw Data streaming in computer networking - MPDS 2003 22

Unidimensional clusters 15 10. 8. 0. 2 35 30 40 10. 8. 0. 3

Unidimensional clusters 15 10. 8. 0. 2 35 30 40 10. 8. 0. 3 10. 8. 0. 4 10. 8. 0. 5 June 8, 2003 160 35 110 10. 8. 0. 9 75 10. 8. 0. 10 10. 8. 0. 14 Data streaming in computer networking - MPDS 2003 23

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29 380 10. 8. 0. 0/30 50 10. 8. 0. 4/30 70 10. 8/30 305 75 10. 8. 0. 12/30 10. 8. 0. 2/31 50 10. 8. 0. 4/31 70 10. 8/31 270 10. 8. 0. 1 35 0/31 75 10. 8. 0. 14/31 15 10. 8. 0. 2 35 30 40 10. 8. 0. 3 10. 8. 0. 4 10. 8. 0. 5 June 8, 2003 160 35 110 10. 8. 0. 9 75 10. 8. 0. 10 10. 8. 0. 14 Data streaming in computer networking - MPDS 2003 24

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29 380 10. 8. 0. 0/30 50 10. 8. 0. 4/30 70 10. 8/30 305 75 10. 8. 0. 12/30 10. 8. 0. 2/31 50 10. 8. 0. 4/31 70 10. 8/31 270 10. 8. 0. 1 35 0/31 75 10. 8. 0. 14/31 15 10. 8. 0. 2 35 30 40 10. 8. 0. 3 10. 8. 0. 4 10. 8. 0. 5 June 8, 2003 160 35 110 10. 8. 0. 9 75 10. 8. 0. 10 10. 8. 0. 14 Data streaming in computer networking - MPDS 2003 25

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29 380 10. 8/30 305 10. 8/31 270 160 110 10. 8. 0. 9 June 8, 2003 Data streaming in computer networking - MPDS 2003 26

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29

Unidimensional clusters 10. 8. 0. 0/28 500 10. 8. 0. 0/29 120 10. 8/29 380 10. 8/30 305 10. 8/31 270 160 110 10. 8. 0. 9 June 8, 2003 Data streaming in computer networking - MPDS 2003 27

Multidimensional clusters l Two dimensions u u Source network Protocol (traffic type) l Trees

Multidimensional clusters l Two dimensions u u Source network Protocol (traffic type) l Trees turn into lattice l Multiple parents l Nodes overlap June 8, 2003 Data streaming in computer networking - MPDS 2003 28

Offline solution June 8, 2003 Data streaming in computer networking - MPDS 2003 29

Offline solution June 8, 2003 Data streaming in computer networking - MPDS 2003 29

Sample report June 8, 2003 Data streaming in computer networking - MPDS 2003 30

Sample report June 8, 2003 Data streaming in computer networking - MPDS 2003 30