Tranalyzer Feel the packets be the packets Stefan

Tranalyzer Feel the packets, be the packets Stefan Burschka

What we do: Network Troubleshooting, Security: l l TRANALYZER(T 2/3): High Speed and Volume Traffic Analyzer TRAVIZ: Graphical Toolset for Tranalyzer Complete Tool Sets for Traffic Mining (TM), Forensics Artificial Intelligence Research: TM & Visualisation Brain support 4 multi-dim datasets Encrypted Traffic Mining Operational Picture Malware and covert channel detection Nifty stuff 2

“It's the network – go fix it!” l 3

The Network is slow, The Network is insecure; NO, it's not Microsoft, shut up, It wasn't me. . . Manager (MBA) Always right, Do. R License to Powerpoint Production (poor Techie) Knows, Always warned, Always his fault: FUBAR License to get fired Finance (MBA) Knows basic calculus License to Excel We didn't find the problem in 4 months, can you do the job in 2 weeks? (We supply 20 TB data)

Troubleshooting, Security Traffic Mining: Change your perspective 5

What is wrong here? 6

See the disaster now? Now you have context! 7

Traffic Mining(TM): Hidden Knowledge: Listen | See, Understand, Invariants, Model Application in – – 8 Troubleshooting, Security (Classification, Encrypted TM ) Netzwerk usage (Voi. P, P 2 P traffic shaping, application/user profiling) Profiling & Marketing (usage performance- & market- index) Law enforcement and Legal Interception (Indication/Evidence)

Basic Need: Versatile Flow Compression A B Definition: (6 -Tuple) Vlan(s), src. IP, srcp. Port, dst. IP, dst. Port, L 4 Protocol Or why not a bit more context and meaning ? src. Who, dst. Who src. Network, dst. Network Bad, Good Internal / External 9

Closed source loud Tools Netflow (Sometimes not so loud, comes with routers) Pro: Good hands-on tool, flow statistics, header parameters, standard Cons: Not all statistics we need, no developer support Giga. Stor (Horrible loud and exceptional expensive HW) Pro: heuristic expert system, Graphics, reports, whatever is in the DB Cons: What we needed is not in the DB, no developer support DPI (Elacoya, Sandvine, . . ) (Terrible loud and expensive HW) Pro: good protocol resolution, nice reports Cons: Its a DPI not a verstile flow engine with developer support 10

Open source silent SW Wireshark, T-Shark (packet, flow statistics) Pro: Hands-on tool, protocol db, GUI, command line, filtering Cons: Limited flow statistics and file size, post processing difficult Silk (flow based) Cons: Not even close to Netflow, 5 tuple, esoteric config Netmate Pro: Flow, packet based, nice features, Cons: Config , handling, 5 tuple, that is, . . . University NTOP(ng) Pro: Monitoring, flow statistics, config, GUI, Graphics Cons: not really flow based as we need it, protocol encapsulation? IDS (SNORT, BRO) Pro: Alarming, regex, flexible 11 Cons: Alarming, no Flows, BRO: memory leaks, university stuff

Need an Allrounder, script friendly between Wireshark, Netflow and 2006: Somebody has to develop me !!

Tranalyzer 2(T 2), C 99, (Geek/Dev/Prof) High Volume Traffic Preprocessing and Troubleshooting Open Source Speed and Memory optimized by *. h“, config and. /autogen. sh -n Command line based, full pcap, eth and dag cards Post processing : HEX, ‘text t’; Bash, AWK, Perl, … friendly C Plugin based, Linux, Mac, (Windoof) Subnet labeling (Who, Where, What) BPF Hands-on: Anomaly and security related flags Researchers: Full Statistical and Packet Signal Analysis support Interfaces: Matlab, Gnu. Plot, SPSS, Excel, oocacl, soon Netflow tools The “-s” option: The command line AWK, Perl friendly packet mode GUI: Traviz (http: //sourceforge. net/projects/traviz) Easy to use but, • You have to know your shit 13 •

T 3, C 99, (Geek/Normalo Non. Dev/Prof) High Speed and Volume Troubleshooting, Security, Monitoring Complete new Concept and Design Full IPv 4/6, more protocols as T 2 Basic Features from T 2 + new nifty Plugins Full Subnet labeling and flexible flow aggregation Multi Threading and Interface: High performance GUI Support via professional Tool Set: Unlimited flows and files ip. SOM: AI Tool Set to answer ANY question Core functions into DSP and FPGA in future for the 40 Gig+ More non geek/dev user friendly but, You still have to know your shit 14 •

Report T 2 • /tranalyzer -r ~/wurst/data/weichwurst. dmp -w ~/wurst/results/hartwurst ======================================== Tranalyzer 0. 5. 8 (Anteater), beta. PID: 6123 • ======================================== Active plugins: 00: protocol. Statistics, version 0. 5. 8 01: basic. Flow. Output, version 0. 5. 8 02: mac. Recorder, version 0. 5. 0 03: port. Based. Classifier, version 0. 5. 8 04: basic. Layer 4 Calc. Statistics, version 0. 5. 6 05: tcp. Flags, version 0. 5. 8 06: tcp. States, version 0. 5. 6 07: icmp. Decode, version 0. 5. 8 08: connection. Counter, version 0. 5. 5 09: descriptive. Statistics, version 0. 5. 6 10: n. First. Packets. Stats, version 0. 5. 8 11: packet. Size. Inter. Arrival. Time. Histo, version 0. 5. 8 12: standard. File. Sink, version 0. 5. 0 13: text. File. Sink, version 0. 5. 8 --> --> --> --> _protocols. txt, _flow. txt / bin ports. txt subnet. txt _flow. txt / bin, portmap. txt _flow. txt / bin, _icmp. Stats. txt _flow. txt / bin creates text output _flow. txt creates binary output _flow. bin Start processing file: /home/wurst//data/weichwurst. dmp BPF: (null) Dump start: 1351794649. 186547 sec : Wed 01 Nov 2012 18: 30: 49. 186547 Shutting down Tranalyzer 0. 5. 8. . . Dump stop: 1351837376. 118852 sec : Thu 02 Nov 2012 06: 22: 42. 118852 Total dump duration: 42712. 932305 sec Number of processed packets: 6497970 Number of processed traffic bytes: 1749617780 Number of ARP packets: 1603 Number of RARP packets: 5 Number of IPv 4 fragmented packets: 299 Number of IPv 6 packets: 0 Number of IPv 4 flows: 3395325 Average snapped Bandwidth: 327. 634 KBit/s Average full IP Bandwidth: 326. 386 Kbit/s Warning: IPv 4 Fragmentation header packet missing 15 •

T 2 Protocol File Total packets captured: 42278 L 4 Protocol # Packets Relative Frequency[%] Protocol description 1 21 0. 049671 Internet Control Message Protocol 2 6 0. 014192 Internet Group Management Protocol 6 41698 98. 628128 Transmission Control Protocol 17 250 0. 591324 User Datagram Protocol 103 28 0. 066228 Protocol Independent Multicast Total TCP packets: 41698 Port # Packets Relative Frequency[%] 80 41519 99. 570723 World Wide Web HTTP 445 8 0. 019186 Win 2 k+ Server Message Block 5557 147 0. 352535 Total UDP packets: 250 Port # Packets Relative Frequency[%] 53 2 137 50 138 21 1900 18 1908 2 1985 156 0. 800000 Domain Name Server 20. 000000 NETBIOS, [trojan] Msinit 8. 400000 NETBIOS Datagram Service 7. 200000 SSDP 0. 800000 Dawn 62. 400000 Hot Standby Router Protocol 16 •

T 2 ICMP Stats File Total # of ICMP messages: 22258 ICMP / Total traffic percentage[%]: 0. 343 Echo reply / request ratio: 0. 892 Type Code # of Messages Relative Frequency [%] ICMP_ECHOREQUEST 111 0. 499 ICMP_ECHOREPLY 99 0. 445 ICMP_SOURCE_QUENCH 15 0. 067 ICMP_TRACEROUTE 0 0. 000 ICMP_DEST_UNREACH ICMP_NET_UNREACH 60 0. 270 ICMP_DEST_UNREACH ICMP_HOST_UNREACH 15674 70. 420 ICMP_DEST_UNREACH ICMP_PROT_UNREACH 0 0. 000 ICMP_DEST_UNREACH ICMP_PORT_UNREACH 3100 13. 928 ICMP_DEST_UNREACH ICMP_FRAG_NEEDED 0 0. 000 ICMP_DEST_UNREACH ICMP_SR_FAILED 0 0. 000 ICMP_DEST_UNREACH ICMP_NET_UNKNOWN 0 0. 000 ICMP_DEST_UNREACH ICMP_HOST_ISOLATED 0 0. 000 ICMP_DEST_UNREACH ICMP_NET_ANO 8 0. 036 ICMP_DEST_UNREACH ICMP_HOST_ANO 600 2. 696 ICMP_DEST_UNREACH ICMP_NET_UNR_TOS 0 0. 000 ICMP_DEST_UNREACH ICMP_HOST_UNR_TOS 0 0. 000 ICMP_DEST_UNREACH ICMP_PKT_FILTERED 776 3. 486 ICMP_DEST_UNREACH ICMP_PREC_VIOLATION 0 0. 000 ICMP_DEST_UNREACH ICMP_PREC_CUTOFF 0 0. 000 ICMP_REDIRECT ICMP_REDIR_NET 1125 5. 054 ICMP_REDIRECT ICMP_REDIR_HOST 589 2. 646 ICMP_REDIRECT ICMP_REDIR_NETTOS 0 0. 000 ICMP_REDIRECT ICMP_REDIR_HOSTTOS 0 0. 000 ICMP_TIME_EXCEEDED ICMP_EXC_TTL 95 0. 427 ICMP_TIME_EXCEEDED ICMP_EXC_FRAGTIME 0 0. 000 ICMP_TRACEROUTE 0 0. 000 17 •

T 2 Flow Header File: Hands-On 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 . . . 8: NR Minimum layer 3 packet size 8: NR Maximum layer 3 packet size 19: NR Average packet load ratio 19: NR Send packets per second 19: NR Send bytes per second 19: NR Packet stream asymmetry 19: NR Byte stream asymmetry 8: NR IP Minimum delta IP ID 8: NR IP Maximum delta IP ID 7: NR IP Minimum TTL 7: NR IP Maximum TTL 7: NR IP TTL Change count 13: NR IP Type of Service 14: NR IP aggregated flags 8: NR IP options count 13, 15: NR IP aggregated options 18 •

T 2 Flow Header View: Hands-On 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 8: NR TCP packet seq count 10: NR TCP sent seq diff bytes 8: NR TCP sequence number fault count 8: NR TCP packet ack count 10: NR TCP flawless ack received bytes 8: NR TCP ack number fault count 8: NR TCP initial window size 19: NR TCP average window size 8: NR TCP minimum window size 8: NR TCP maximum window size 8: NR TCP window size change down count 8: NR TCP window size change up count 8: NR TCP window size direction change count 13: NR TCP aggregated protocol flags (cwr, ecn, urgent, ack, push, reset, syn, fin) 14: NR TCP aggregated header anomaly flags 8: NR TCP options Packet count 8: NR TCP options count Yes I know, I should do something 15: NR TCP aggregated options special for the Time. Stamp option 8: NR TCP Maximum Segment Length 7: NR TCP Window Scale 19: NR TCP Trip Time Syn, Syn-Ack | Syn-Ack, Ack 19: NR TCP Round Trip Time Syn, Syn-Ack, Ack | TCP Ack-Ack RTT 19: NR TCP Ack Trip Min 19: NR TCP Ack Trip Max 19: NR TCP Ack Trip Average 13: NR TCP aggregated protocol state flags 15, 14: NR ICMP Aggregated type & code bit field 19: NR ICMP Echo reply/request success ratio 9: NR Number of connections from source IP to different hosts 9: NR Number of connections from destination IP to different hosts 19 •

T 2 Flow Header View: TM geeks 68 19: NR Minimum packet length 69 19: NR Maximum packet length 70 19: NR Mean packet length 71 19: NR Lower quartile of packet lengths 72 19: NR Median of packet lengths 73 19: NR Upper quartile of packet lengths 74 19: NR Inter quartile distance of packet lengths 75 19: NR Mode of packet lengths 76 19: NR Range of packet lengths 77 19: NR Standard deviation of packet lengths 78 19: NR Robust standard deviation of packet lengths All you never wanted to 79 19: NR Skewness of packet lengths 80 19: NR Excess of packet lengths know about statistics in a 81 19: NR Minimum inter arrival time flow 82 19: NR Maximum inter arrival time 83 19: NR Mean inter arrival time 84 19: NR Lower quartile of inter arrival times 85 19: NR Median inter arrival times 86 19: NR Upper quartile of inter arrival times 87 19: NR Inter quartile distance of inter arrival times 88 19: NR Mode of inter arrival times L 2/3/4/7 89 19: NR Range of inter arrival times configurable 90 19: NR Standard deviation of inter arrival times 91 19: NR Robust standard deviation of inter arrival times Packet 92 19: NR Skewness of inter arrival times Statistics 93 19: NR Excess of inter arrival times 94 8, 25: R L 2 L 3/L 4/Payload( s. PACKETLENGTH in packet. Capture. h) length and inter-arrival times for the N first packets • 20 95 8, 9, 9: R Packetsize Inter Arrival Time histogram bins

HOW TO find the needle in the flow stack? Have a break have a HEX & ¦ scripting!

T 2 Text Flow File: Basic plugins A 1196278772. 439355 1196279184. 642073 412. 202718 0 x 9 B 42 192. 168. 1. 10 0 x 00000001 2119 68. 3. 4. 5 0 x 800806034 80 00: 0 f: 1 f: cf: 7 c: 45_00: 0 c: 07: ac: 0 a_6387 http 6387 8272 5437587 0 4 15. 494803 1. 125660 -0. 128590 0. 999829 1 87 128 0 x 00 0 x 42 0 x 0000 116 6231 4116 5437724 2253 63754 64831. 988281 65535 3342 2904 5713 0 x 18 0 x. F 900 0 x 0000 0 x 03 0 x 0000 -1. 0 1 1 1. . . 22 6 464 62501 B 22 6 1196278772. 409312 1196279184. 642073 412. 232761 0 x 9 B 43 192. 168. 1. 10 0 x 00000001 80 68. 3. 4. 5 0 x 80080634 2119 00: d 0: 00: 64: d 0: 00_00: 0 f: 1 f: cf: 7 c: 45_8272 http 8272 6387 5437587 464 0 1380 20. 066333 13190. 574633 0. 128590 0. 999829 1 3 63 63 0 x 00 0 x 42 0 x 0000 8146 5440245 109 116 464 8104 5840. 000000 65535 0 0 0 x 18 0 x 1 B 00 0 x 0000 0 x 03 0 x 0000 -1. 0 1 1 1 22 • 5840. . .

T 2 Binary Coding Status: 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^7 2^8 2^9 2^10 2^11 2^12 2^13 2^14 2^15 0 x 0001 0 x 0002 0 x 0004 0 x 0008 0 x 0010 0 x 0020 0 x 0040 0 x 0080 0 x 0100 0 x 0200 0 x 0400 0 x 0800 0 x 1000 0 x 2000 0 x 4000 0 x 8000 Flow Warning Flag: If A flow: Invert Flow, NOT client flow Dump/flow: L 3 Snaplength too short Dump/flow: L 2 header length too short Dump/flow: L 3 header length too short Dump: Warning: IP Fragmentation Detected Flow: ERROR: Severe Fragmentation Error Flow: ERROR: Fragmentation Header Sequence Error Flow ERROR: Fragmentation Pending at end of flow Flow/Dump: Warning: VLAN(s) detected Flow/Dump: Warning: MPLS unicast detected Flow/Dump: Warning: MPLS multicast detected Flow/Dump: Warning: L 2 TP detected Flow/Dump: Warning: PPP detected Flow/Dump: 0/1: IPv 4/IPv 6 detected Flow/Dump: Warning: Land Attack detected Flow/Dump: Warning: Time Jump So what is: 0 x 9 B 43 23 •

T 2 Flow Binary Coding: ip. Flags 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^7 2^8 2^9 2^10 2^11 2^12 2^13 2^14 2^15 0 x 0001 0 x 0002 0 x 0004 0 x 0008 0 x 0010 0 x 0020 0 x 0040 0 x 0080 0 x 0100 0 x 0200 0 x 0400 0 x 0800 0 x 1000 0 x 2000 0 x 4000 0 x 8000 IP Options present, s. IP Options Type Bit field IPID out of order IPID rollover Fragmentation: Below expected RFC minimum fragment size: 576 Fragmentation: Fragments out of range (Possible tear drop attack) Fragmentation: MF Flag Fragmentation: DF Flag Fragmentation: x Reserved flag bit from IP Header Fragmentation: Unexpected position of fragment (distance) Fragmentation: Unexpected sequence of fragment L 3 Checksum Error L 4 Checksum Error Snap. Length Warning: IP Packet truncated, L 4 Checksums invalid Packet Interdistance == 0 Packet Interdistance < 0 Internal State Bit for Interdistance assessment So what is: 0 x 1 C 21 24 •

T 2 Flow Binary Coding: tcp. Flags 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^7 2^0 2^1 2^2 2^3 2^4 2^5 2^6 2^8 2^9 2^10 2^11 2^12 2^13 2^14 2^15 0 x 01 0 x 02 0 x 04 0 x 08 0 x 10 0 x 20 0 x 40 0 x 80 FIN SYN RST PSH ACK URG ECE CWR 0 x 0001 0 x 0002 0 x 0004 0 x 0008 0 x 0010 0 x 0020 0 x 0040 0 x 0080 0 x 0100 0 x 0200 0 x 0400 0 x 0800 0 x 1000 0 x 2000 0 x 4000 0 x 8000 No more data, finish connection Synchronize sequence numbers Reset connection Push data Acknowledgement field value valid Urgent pointer valid ECN-Echo Congestion Window Reduced flag is set Fin-Ack Flag Syn-Ack Flag Rst-Ack Flag Syn-Fin Flag, Scan or malicious packet Syn-Fin-Rst Flag, potential malicious scan packet or malicious channel Fin-Rst Flag, abnormal flow termination Null Flag, potential NULL scan packet, or malicious channel XMas Flag, potential Xmas scan packet, or malicious channel Due to packet loss, Sequence Number Retry, retransmit Sequence Number out of order Sequence mess in flow order due to pcap pkt loss Warning: L 4 Option field corrupt or not acquired Syn retransmission Ack number out of order Ack Packet loss, probably on the sniffing interface Internal state: TCP Window Size Machine So what is: 0 x 1 B 0 x. C 403 25 •

T 2 Flow Binary Coding: icmp. Flags Aggregated ICMP Type & Code bit Field So what is: 0 x 00000100_0 x 0001 26 •

T 2 Packet Signal: Encrypted Vo. IP Mining Packet. Length_Packet-Interdistance; … 1023_0. 000000; 758_0. 030043; 1380_0. 110201; 80_0. 00000; 369_0. 000010; 230_0. 02002 9; 1380_0. 070101; 80_0. 000000; 50_0. 060086; 1380_0. 070101; 80_0. 090130; … Packet Length time 27 Post processing scripts: /tranalyzer/trunk/scripts

T 2 Statistical Application / User profiling Packet length-Interdistance Statistics: Fingerprint Pkt. Len_Packet-IAT_cnt. Pkt. Len_cnt. IAT; … 0_0_2322_6271_2396; 0_2_82_6271_90; 0_4_114_6271_114; 0_6_138_6271_140; 0_8_162_6271_164; 0_10_157_6271_160; 0_12_220 _6271_224; 0_14_217_6271_222; 0_16_325_6271_325; 0_18_373_6271_376; 0_20_493_6271_498; 0_22_340_6271_343; 0_24_238_62 71_238; 0_26_283_6271_284; 0_28_143_6271_143; 0_30_114_6271_114; 0_32_139_6271_140; 0_34_175_6271_176; 0_36_72_6271_7 3; 0_38_25_6271_25; 0_40_20_6271_20; 0_41_12_6271_13; 0_42_8_6271_8; 0_43_6_6271_6; 0_44_6_6271_6; 0_45_4_6271_4; 0_46_5 _6271_5; 0_47_9_6271_10; 0_48_9_6271_9; 0_49_6_6271_6; 0_50_4_6271_4; 0_51_4_6271_4; 0_52_5_6271_5; 0_53_3_6271_3; 0_54_ 9_6271_9; 0_55_7_6271_8; 0_56_1_6271_1; 0_57_4_6271_4; 0_58_1_6271_1; 0_59_3_6271_3; 0_60_4_6271_4; 0_61_4_6271_4; 0_62_ 2_6271_2; 0_63_1_6271_1; 0_64_1_6271_1; 0_65_1_6271_1; 4_0_74_116_2396; 4_2_8_116_90; 4_6_2_116_140; 4_8_2_116_164; 4_10 _3_116_160; 4_12_4_116_224; 4_14_5_116_222; 4_18_3_116_376; 4_20_5_116_498; 4_22_3_116_343; 4_26_1_116_284; 4_32_1_116_ 140; 4_34_1_116_176; 4_36_1_116_73; 4_41_1_116_13; 4_47_1_116_10; 4_55_1_116_8 …. . • Post processing scripts: /tranalyzer/trunk/scripts Skype: Vulnerable against TM Attack 28 •

Some T 3 Plugins L 7 Protocols: Mail, HTTP, etc Routing: OSPF DNS / DHCP Full PCRE Regex Signal Processing Artificial Intelligence (RNN, Bayes, ESOM), nifty entropy shit Connection Matrix, Centrality IP Statistics: Host Database 29 •

So what? Some Examples

The one way TCP Flow problem Symptom: on and off access problems TCP flows established, unidirectional T 2 proofed: Reverse connection exists, not through firewall Not communicated online mis-configuration of firewall Trampel OSPF

FFT of some Packet Signals Packet Length • time • 32 •

Traffic Mining: Encrypted Content Guessing SSH Command Guessing IP Tunnel Content Profiling Pitch based Classification Encrypted Voip Guessing: CCC 2011 33

TM Your OWN: Packet Length Signal See the features? Codec training Burschka (Fischkopp) Linux Dominic (Student) Windows SN Ping min l =3 34

Connection plugin: Social Behaviour 35 •

What is the Unknown? 36 •

HOW TO find Bad Guys? Day: 0. 7% of all users 42% bandwidth, WTF? P 2 P Traffic Average Users ? ? ? l. U nti rce Pe Percentil User r se ffic a. Traffic r T l a Normal Norm 37

HOW TO find Bad Guys? Night: Same guys @ night 3 am, . . . P 2 P Traffic Average Users Machines of WAREZ guys Perce ntil U ser No 38 ffic a r T l rma

Layer 3/4/whatever Visualization Graphviz --> Operational Picture in Bootcamp _flow. txt Your AWK script Graphviz: dotty 39 •

Layer 3/4 Visualization Graphviz --> simple forensic Picture

Network Classification Centrality Connection Matrix PCA Largest Eigenvector Plot / t 41 •

Network / Host Classification Centrality

ip. SOM Operational Picture: 13 Dim statistical T 2 Flow parameters Now conceivable by human brain Bot Scanner DNS Zone Transfer 43

Questions / Comments RFM and try me Join the development force Who wants Bootcamp? http: //sourceforge. net/projects/tranalyzer/ http: //tranalyzer. com http: //sourceforge. net/projects/traviz 44 Google: Dataming for Hackers stefan. burschka@ruag. com