Applications of Machine Learning in Cisco Web Security

Cisco Web Security • Cisco, Ironport and Scan. Safe • Request time filtering • Categorization and classification • Reputation • Response time filtering • Malware types and attack vectors • Malware detection • Dynamic classification • Other challenges © 2011 Cisco Systems, Inc. All rights reerved. 2

The Ubiquitous Speaker Slide • Richard Wheeldon • UCL Graduate in 1999 • Ph. D from Birkbeck in 2003 • Joined Cisco December 2009 • http: //www. rswheeldon. com/ • Acknowledgements • Steve Poulson - spoulson@cisco. com • Bryan Feeney - b. feeney@cs. ucl. ac. uk © 2011 Cisco Systems, Inc. All rights reerved. 3

Cisco, Ironport and Scan. Safe • Cisco • World’s leading network company • Ironport • Leader in Anti-spam • Provide Web Security Appliances • Scan. Safe • World leader in “Security as a Service” • Scans 1. 8 billion web requests a day • Blocks 32 million of them © 2011 Cisco Systems, Inc. All rights reerved. 4

We’re local © 2011 Cisco Systems, Inc. All rights reerved. 5

Previous MSc projects • Tree Kernels for CFG similarity • Guangyan Song, 2010 • Fast computation of the Kernel of a Tree and applications to Semi-Supervised Learning • Malcolm Reynolds, 2009 • Comparing N-gram features for web page classification • Noureen Tejani, 2007 © 2011 Cisco Systems, Inc. All rights reerved. 6

We’re hiring • Positions • Software Developers • QA, Operations, Research • Locations • Scan. Safe • UK - Bedfont Lakes, Reading, Staines, Edinburgh • Galway, EMEA, US, Worldwide • Graduate recruitment • http: //www. cisco. com/go/universityjobs • http: //www. cisco. com/careers/ • rwheeldo@cisco. com © 2011 Cisco Systems, Inc. All rights reerved. 7

Scansafe’s Saa. S 1. Availability 2. Latency Time our service is available to scan traffic 99. 999% guaranteed availability Additional load time attributable to services Evaluated by 3 rd party analysis 3. False Positives 4. False Negatives Pages that were blocked but should not have Pages that were not blocked, but should have © 2011 Cisco Systems, Inc. All rights reerved. 8

Risks of Unfiltered Content • Software threats • Malware • Phishing • Botnets • Business threats • Productivity Loss • Bandwidth congestion • Legal liability • Data Leaks © 2011 Cisco Systems, Inc. All rights reerved. 9

The Web vs. Email Web Email Most web traffic is good Most e-mail is bad Easy to find safe sites Easy to get Spam Harder to get dangerous URLs Harder to get examples of good mail Blocking web sites is visible Blocking email is invisible Performance gain from white-listing Performance gain from blocking Very Real-Time (<2 s) Not Real-Time (<Nhrs) © 2011 Cisco Systems, Inc. All rights reerved. 10

Request time filtering • Motivation • Quicker blocks save bandwidth and processing time • If the request is made, the damage may be done • Techniques • Databases • Reputation • Rules • Trained systems © 2011 Cisco Systems, Inc. All rights reerved. 11

Category-based filtering • Responsible for most blocks • High-risk and high-traffic • Manual categorizers • 10 million URLs • 97% of traffic • 2 million porn sites © 2011 Cisco Systems, Inc. All rights reerved. 12

Web Reputation • Feeds • Phishing sites • Malware sites • Heuristics • In spam but not in ham • Age of domain registration • High traffic – e. g. Alexa 1000 • Scanned but never blocked © 2011 Cisco Systems, Inc. All rights reerved. 13

Web Reputation in the WSA © 2011 Cisco Systems, Inc. All rights reerved. 14

© 2011 Cisco Systems, Inc. All rights reerved. 15

Keyword-based URL filtering • Keyword rules • Strange URLs • Fitness -> Health • whorepresents. com • Basketball -> Sport • therapistfinder. com • Pizzeria -> Food • speedofart. com • Restaurant -> Food • expertsexchange. com • Whore -> Porn • penisland. com • powergenitalia. it © 2011 Cisco Systems, Inc. All rights reerved. 16

Recognizing Porn URLs • http: //www. penisland. com • Example of segmentation problem P('peni') X P('sland') P('penis') X P('land') P('pen') X P('island') • Extends to classification P('penis') X P('land') X P(porn|'penis') X P(porn|'land') P('pen') X P('island') X P(not_porn|'pen') X P(not_porn|'island') © 2011 Cisco Systems, Inc. All rights reerved. 17

Phishing and Malware Examples • Phishing examples • http: //pavpals-com-usaprewiwerluithaniirse. 345. pl • http: //82. 195. 143. 18/onlinepaypal. com/ • http: //www. jetboatflush. com/~nfioemro/www. paypal. fr/webscrcmd=. . . • Malicious examples: • www 1. scan-projectrf. cz. cc • www 1. scan-projectsi. cz. cc • www 1. scan-projectst. cz. cc • www 1. scan-projectte. cz. cc • www 1. scan-projectti. cz. cc © 2011 Cisco Systems, Inc. All rights reerved. 18

Searchahead • If we can identify bad URLs we can warn before the user clicks. • Over 90% of new sites are visited as the result of an Internet search Acceptable Uncategorized Prohibited Malicious © 2011 Cisco Systems, Inc. All rights reerved. 19

Response Time Scanning Keyloggers Webmail Links Rootkits Graphics Banner Ads • Trusted sites are targets Backdoors • Strength-in-depth combination of commercial scanners and in-house technology. Worms Comments Blogs Trojan Horses Ad Links New Web Pages © 2011 Cisco Systems, Inc. All rights reerved. 20

Exploited sites in recent years • Facebook • Times India • Miami Dolphins •

Nothing is safe – not even Twitter! http: //www. youtube. com/fslabs © 2011 Cisco

Signature Databases • From 2006 to 2008, the F-Secure signature database grew from 250000 entries to 1. 5 million • The rate at which variants of viruses come out is growing rapidly • No vendor can rely exclusively on signatures © 2011 Cisco Systems, Inc. All rights reerved. 23

Zero-hour protection • Vendors take time to release signature updates • Win 32. Ist. Bar. jl trojan • Outbreak Intelligence (OI) provides proactive threat detection • A huge data set of traffic to be leveraged © 2011 Cisco Systems, Inc. All rights reerved. 24

How does OI use Machine Learning? • Approaches • Malware detection • Anomaly detection • Dynamic categorization • Techniques Employed • Supervised Learning • Unsupervised Learning • Sandboxing © 2011 Cisco Systems, Inc. All rights reerved. 25

Dynamic Classification • Document classification across 80 categories • Increases coverage • Language identification • Identifies inappropriate content • Porn is relatively easy • Phishing is harder – but not impossible? • Hate speech is harder still © 2011 Cisco Systems, Inc. All rights reerved. 26

DC for identifying malicious sites • Automated tools generate malicious sites • Fake escrow • Fake pharmacy • Mule recruitment • Examples from Richard Clayton’s 2010 FOSDEM talk • http: //www. google. com/search? q=%22 before+that+was+a+com mercial+manager+of+a+large+corporation+engaged+in+electro nics+production%22 • http: //www. google. com/search? q=%22 as+the+most+trusted+es crow+service+on+the+internet%22 © 2011 Cisco Systems, Inc. All rights reerved. 27

Malicious Executable Files • The final stage of an attack is frequently downloading an executable • Traditionally blocked using signatures • We use a combination of signature-based scanners and machine-learning © 2011 Cisco Systems, Inc. All rights reerved. 28

Drive-by attacks • Almost no-one opens executables from odd sources any more, so instead people use driveby attacks. • A normal file (e. g. Flash, PDF, Javascript, Image file) is crafted to exploit a vulnerability in a viewer or library and execute code embedded within the file. © 2011 Cisco Systems, Inc. All rights reerved. 29

Flash “Symantec recently highlighted Flash for having one of the worst security records in 2009. We also know first hand that Flash is the number one reason Macs crash. We have been working with Adobe to fix these problems, but they have persisted for several years now. We don’t want to reduce the reliability and security of our i. Phones, i. Pods and i. Pads by adding Flash” Steve Jobs, April 2010 http: //www. apple. com/hotnews/thoughts-on-flash/ © 2011 Cisco Systems, Inc. All rights reerved. 30

The growing threat of Java • Almost as common as Flash • 90% of PCs have Java • 700 000 JDK downloads per month • 3. 48 Million JRE downloads per month • Growth in known vulnerabilities • 29 patched in a single update (Oct 2010) • Growth in exploits reported by Sophos, Symantec, Microsoft and Cisco • Signatures + Trained Scanlet © 2011 Cisco Systems, Inc. All rights reerved. 31

Detecting Malicious Java. Script • Sandboxing • Behavioural checking • Good way to beat obfuscation techniques • Difficult to constrain • Trained classification • Analyse features © 2011 Cisco Systems, Inc. All rights reerved. 32

Javascript Features v 46 f 658 f 5 e 2260(v 46 f 658 f 5 e 3226){ function v 46 f 658 f 5 e 4207 () {return 16; } return(parse. Int(v 46 f 658 f 5 e 3226, v 46 f 658 f 5 e 4207())); }function v 46 f 658 f 5 e 61 f 4(v 46 f 658 f 5 e 7174){ function v 46 f 658 f 5 ea 0 cd () {return 2; } var v 46 f 658 f 5 e 813 e=''; for(v 46 f 658 f 5 e 9105=0; v 46 f 658 f 5 e 9105<v 46 f 658 f 5 e 7174. length; v 46 f 658 f 5 e 9105+=v 46 f 658 f 5 ea 0 cd()){ v 46 f 658 f 5 e 813 e+=(String. from. Char. Code(v 46 f 658 f 5 e 2260(v 46 f 658 f 5 e 7174. substr(v 46 f 658 f 5 e 9105, v 46 f 658 f 5 ea 0 cd())))); }return v 46 f 658 f 5 e 813 e; } document. write(v 46 f 658 f 5 e 61 f 4('3 C 5343524950543 E 77696 E 646 F 772 E 7374617475733 D 2 ')); The above is Java. Script, but where are the features? An exercise for the reader! © 2011 Cisco Systems, Inc. All rights reerved. 33

Obfuscation • Attackers use obfuscation • But so do legitimate vendors (e. g. Google) • And large Web 2. 0 libraries • Techniques include • Name changes • String concatenation (eval) • Dynamically loaded/generated/decrypted code (eval) • Splitting functionality across files © 2011 Cisco Systems, Inc. All rights reerved. 34

Malicious Non-Executable Files • There a lot of file formats out there – documents, pictures, videos. • For zero-day attacks, we have no data to compare against. • Basically this is anomaly detection. © 2011 Cisco Systems, Inc. All rights reerved. 35

Development Constraints • Low False Positive Rate • Robust • Tolerant against malformed data • Language-agnostic • Scalable • 1. 8 Billion requests per day on 1000 servers • Low latency © 2011 Cisco Systems, Inc. All rights reerved. 36

Back-end processing • If a technique is too slow for real-time scanning, that doesn’t make it useless. • Back end processing can generate lists of good and bad files and help evaluate new techniques. © 2011 Cisco Systems, Inc. All rights reerved. 37

Want to know more? • Cisco 2 Q 10 Global Threat Report http: //www. cisco. com/web/about/security/intelligence/ci sco_threat_072610_959. pdf • Richard Clayton : Evil on the Internet http: //www. securitytube. net/Phishing-(Evil-on-the. Internet)-FOSDEM-Talk-video. aspx • Kaspersky Lab Security News Service http: //threatpost. com/ • A plan for Spam http: //www. paulgraham. com/spam. html © 2011 Cisco Systems, Inc. All rights reerved. 38

Still want to know more? • Identifying Suspicious URLs : An Application of Large. Scale Online Learning http: //videolectures. net/icml 09_ma_isu/ • Peter Norvig Google : Statistical Learning as the Ultimate Agile Development Tool http: //videolectures. net/cikm 08_norvig_slatuad/ • Writing Clam. AV Signatures Alain Zidouemba http: //www. clamav. net/doc/webinars/Webinar-Alain 2009 -03 -04. ppt © 2011 Cisco Systems, Inc. All rights reerved. 39

Take Home Messages • Web Security • Challenging and interesting domain • Many applications for Machine Learning • Scan. Safe and Cisco • Many opportunities for collaboration • Several opportunities for student projects © 2011 Cisco Systems, Inc. All rights reerved. 40