Big Data Intelligence Harnessing Petabytes of WAF statistics

Big Data Intelligence Harnessing Petabytes of WAF statistics to Analyze & Improve Web Protection in the Cloud Ory Segal, Tsvika Klein Akamai Technologies

About Us • Ory Segal – Principal Product Architect, Cloud Security • Tsvika Klein – Product Manager, Cloud Security Hosted by OWASP & the NYC Chapter

Topics to Cover Akamai & OWASP Mod. Security CRS Relationship Security Big Data @ Akamai Measuring WAF Accuracy @ Akamai CRS through the Big Data Prism (Lessons Learned) Hosted by OWASP & the NYC Chapter

About Us But we only have 45 minutes… And too much data to cover…

Akamai & OWASP CRS This is not an Akamai marketing presentation Akamai has been offering its cloud-based WAF since 2009. Kona Site Defender: – – – OWASP CRS (Akamai Kona Rules) DDo. S Protection DNS Protection Bot Detection Site Shield / Site Cloaking OWASP CRS was ported to Akamai MD, and does not run directly on Mod. Security

SECURITY BIG DATA @ AKAMAI

Akamai Intelligent Platform Akamai’s cloud platform enables secure, high-performing user experiences on any device, anywhere 120, 000+ Servers 2, 000+ Locations 750+ Cities Highlights: § 100 million page views per second and 500 billion hits per day § 734 Million IP addresses seen quarterly § 260+ Terabytes of compressed daily logs § 30% of all internet traffic 82 Countries 1, 100+ Networks

CSI Platform Statistics 10 Terabytes of daily attack data 2 Petabytes of security data stored 45 days retention 140 K concurrent connections (incoming data) 600 K log lines / sec. indexed by 30 dimensions 8000 queries daily scanning terabytes of data

CSI High Level Architecture FE Applications AKAMAI EDGE SERVERS BE Applications YODA LOG AGENT HBASE HADOOP YODA ADAPTER

Yoda (Distributed Query Engine) Interactive Multiple data streams Intuitive query language High cardinality aggregation

Security Big Data Challenge #1

Security Big Data Challenge #2

Sample Data App - SARA Interactive Tool to Analyze Security Events

BACK TO WAF & OWASP CRS…

WAF Accuracy Lingo • Imagine a WAF that protects against 100% of all possible attack vectors …by blocking 100% of all HTTP requests • Accurate WAF testing requires you to measure: • • How many real attacks got blocked (TP) How much valid requests were allowed through (TN) How much valid traffic was inappropriately blocked (FP) How many attacks were allowed through ((FN) Lets talk about measuring Precision, Recall, Accuracy, MCC…

Things You Need to Know % of blocked requests that were actual attacks % of attacks that were actually blocked % of decisions that were good decisions Correlation between WAF decisions and actual nature of requests * MCC: http: //en. wikipedia. org/wiki/Matthews_correlation_coefficient

Lets Look at Some Examples WAF Type Requests Valid Attacks Blocked TP TN FP FN P R A MCC Real 1000 990 10 11 8 987 3 2 0. 73 0. 8 0. 995 0. 76 Off 1000 990 10 0 0 990 0 10 N/A 0 0. 99 0 Always Block 1000 990 10 1000 10 0 990 0 0. 01 1 0. 01 0 Noisy 1000 990 10 31 8 967 23 2 0. 26 0. 8 0. 975 0. 45 Conservative 1000 990 10 2 2 990 0 8 1. 00 0. 2 0. 992 0. 45 A WAF’s accuracy needs to be measured both in its ability to block attacks, as well as it’s ability to allow good traffic through…

Introducing: Akamai WAF Testing Framework

Akamai WAF Testing (AWT) Framework • Ability to send both valid & attack traffic • Easily create or add new test cases: • 3 methods: Text files, Burp Extender, Wireshark. pcaps • Easily import test cases from Akamai’s Big Data platform • Configurable and can work with any WAF • Easily define success / fail criteria • Intuitive XML & HTML reports • Easy debugging of FP/FN w/ Anomaly Scoring (rule comb. )

AWT Built-In Test Cases In order to accurately assess WAF, we collected test cases from the following sources: Web interaction recordings of Alexa Top 100 internet sites – Commerce, Health, Consumer Electronics, Reference, Finance, … Recorded commercial web application scanner traffic Ported common False Positive cases from Akamai customers (Big Data) Attacks from Akamai CSI big data platform Ported “valid” test cases from other tools* Exploits from the internet (fuzzers, exploit-db, …* Havij & SQLMap attacks Tens of Thousands of HTTP Requests, divided 95% - 5%

AWT Reports – High Level Statistics

AWT Reports – Protection Statistics

AWT Reports – False Positives Analysis

OWASP CRS – LESSONS LEARNED

CRS Issue #1 – Risk Groups • CRS 2. 2. x uses a single anomaly score – Visibility (granularity) issues – What really happened? • Separate anomaly score “accounting” to smaller risk groups (attack types) – Clear understanding of which attack took place XSS = 35, SQLi = 10, RFI = 0, LFI = 0, … • Challenge: – requires rule mapping to risk groups – Some rules contribute to more than 1 risk group – Requires to put some more thought into anomaly scoring – it’s not just one pile of rules/scores

CRS Issue #2 –Multiple Thresholds Threshold <xss> 25 <xml> 5 XSS Attack: <script>alert('xss')</script> => Score 30 CMDi Attack: ; /bin/sh Valid XML: <book> cat /etc/passwd => Score 5 Hello World </book> => Score 10 Different risks require different anomaly thresholds

CRS Issue #2 30 25 20 15 10 TH 5 0 XSS SQLi CMDi HTTP RFI

CRS Issue #3 – HTTP Violations “BLOCK HTTP PROTOCOL VIOLATIONS ? !? ? ? THAT’S LIKE 1. 21 LOGS PER DAY!!!!!” PETABYTES OF

CRS Issue #3 – HTTP Violations • HTTP RFC Enforcement? ! Good Luck! – APIs, REST services, RSS feeds, Good Bots – most don’t adhere to HTTP RFC – Prior to system tuning: • Missing Accept Header (960015): 14% • Missing User-Agent Header (960009 ): 3% • Can’t trust HTTP violation rules on their own – “Invalid HTTP” risk group with its own threshold • Blocks only seriously-damaged HTTP requests – Build more focused tool fingerprints • See next slide for an explanation on 960015

960015 – Research into 3 hours of triggers Which URLs trigger this rule? 85% Static Media Files Perhaps a Unique User-Agent? “Android” String found in 50% Anything in Common? 95. 1 K “Unique” UAs Common: Android (50%), Apple. Web. Kit (19%), News (21%), App (20%) Can You Give Me Something Else?

CRS Issue #4: Cookies YEAR: 2003 SESSID = 12 f 0 a 0193 b 4 d 93 e 9 s 92 a 39 af; Quite easy to spot a SQLi or XSS payload in a cookies

CRS Issue #4: Cookies YEAR: 2013 C 1 state = 24~1~-1~1~E~6~6~6~10~10~0~0~|~37 A 1 B 34 A~2 EBA 820 B~0 AEBA 380~130959 B 9~032 7 C 30 B~7617 CC 73~21 B 797 A 5~C 6392 AF 5~5 FE 036 DB~|~8 A 173 E 13~7 F 5 D 33 BF ~30 DFEF 65~|~~|~0~1~2~3~4~5|3~4~6~7~8||0~1~2|4~4~6||~|~0~0~0 ~0~|~0~0~0~|~~|~~|; C 2 state = PC#1382573257902104085. 19_06#1384742638|cat#true#1383533098|session#1383533019933 -203317#1383534898; C 3 data = {"v": 1, "rid": "1371546489873_699561", "to": 5, "c": "http: //www. some. site/pa ge. aspx? a=5", "pv": 2, "lc": {"d 0": {"v": 2, "s": true}}, "cd": 0, "sd": 0, "f": 1371546904 751} ; Cinfo = 1403 D 3394_232#scroll on "//<![CDATA[(function() { var f 5_cspm = { pass_params: '1102912_0394939_19210_24253. . . "

CRS Issue #5: Score Spreading Across Selectors In many FP scenarios, score spreads across “selectors” c 1 = 1384044727071|ABCD: 2: : |AC: 1: : |PSD: 0: AKFJ~MOBILE^CLAK_KOL: 1385149290276 [950901 - 5] c 2 = bn: Samsung|mn: GT-I 9300 Galaxy S III|tb: false|mb: true|dos: Android|dosv: 4. 1|bos: KJSKKL|bosv: 9 [981172 - 3] c 3 = PC#1383939352901916004. 20_14#1386636727|check#true#1384044787|session#1384044726390399957#1384046587 [981231 - 3] c 4: = ”” [981318, 981242 – 2, 5] (Total Score: 18) Consider a FP reduction heuristics that reduces the total score when spread across selectors? There are security implications, …

CRS Issue #6: Rule Inefficiency During our big data analysis & AWT usage, we noticed a few troubling rule issues: – Many rules have redundancies in expressions • This tends to push the anomaly score up in many scenarios (“reinforcing a FP”) • Forces pushing the threshold much higher than really needed – Some rules combine weak & strong signatures • FP-prone rules generate high score – reducing their “weight” hurts the accurate signatures in them – Some rules seemed almost useless – e. g. 981172

Summary • Big Data: – OWASP / Mod. Security should consider collecting anonymized trigger information – CRS would greatly benefit from a much larger sample set • CRS Future: – Akamai has already contributed to the CRS project, and would continue to contribute back to the community – We highly recommend adopting some of the major changes done @ Akamai – mainly the “risk groups” model & multiple thresholds • WAF Testing: – Now that the WAF industry has matured, it is time that WAF deployments will be measured for accuracy using tools & methods mentioned here – Precision, Recall and MCC

THANK YOU