Software Security Quality Testing Taxonomy Testing Tools Classification

: : Software Security Quality: : Testing Taxonomy & Testing Tools Classification OWASP App. Sec DC October 2005 Arian J. Evans, OWASP Tools Project Random Security Title, Fish. Net Security arian. evans@fishnetsecurity. com 913. 710. 7085 [mobile] Copyright © 2005 - The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License. The OWASP Foundation http: //www. owasp. org/

OWASP App. Sec DC NIST 2005 : : Software Security Quality: : Testing Taxonomy & Testing Tools Classification Subtitle: What is Software Security, how do we evaluate it, and what do we use to evaluate it? OWASP App. Sec DC 2005 2

--OR-- : : Software Security Quality: : Why does every web-based Document Management Solution Suck so Badly? OWASP App. Sec DC 2005 3

Testing Taxonomy & Tools Project This project, like many other OWASP projects (e. g. -Web. Scarab, . NET Tools), is: Performed mostly nights and weekends A personal labor of love (or dislike of FUD) Performed by a team of one As objective and vendor-agnostic as possible Owes thanks to Fish. Net for letting me use their lab and encouraging vendors to participate < Results are free and worth every penny < So don’t complain too much about the results. < Emphasis has been on “web app scanners” since that’s where the predominant market interest seems to lie right now. < < < OWASP App. Sec DC 2005 4

Who Arian J. Evans Tester, Teacher, Researcher, Decoder Application Security Services Fish. Net Security Consulting Company focused only on Infosec Work in App. Sec Group They pay me to test and help people fix things (blah, blah) OWASP App. Sec DC 2005 5

Outline: I. II. Introduction (done) Chapter One: In the Beginning was FUD. 1) 2) Problems About the OWASP Testing Taxonomy & Tools Project III. Chapter Two: What is Application Security? 1) 2) IV. V. VII. Distinctions Definition Chapter Three: Clarification and Goals Chapter Four: Taxonomy of Testing Chapter Five: Automation Tools Chapter Seven: Q&A (Shoot the Messenger) OWASP App. Sec DC 2005 6

Chapter One: In the Beginning was the FUD Problems with understanding application security, vendor FUD, appsec testing tools, and testing techniques. OWASP App. Sec DC 2005 7

Webappsec Vendor Marketing Hype I. The Problem of Particulars − − − II. XSS, XST, XDS, XDF, XBS: why particulars can hurt more than they help (reference: Issues slide) Selling managers/auditors bullet-point friendly products Classification of Threat/Risk/Issue: Lack of distinction between Category/Class/Particular—Does anyone take formal logic classes anymore? Immature Metrics − − − What is *really* being exploited? (This is not what the OWASP Top-10 or WASC taxonomy addresses. ) *What* exploits result in real loss? (Not what the threads on webappsec@sf are talking about. ) No reward in securing without demonstrable risk. OWASP App. Sec DC 2005 8

The Problems: < Different ways to “test” applications for security quality. < Different tools to “test” applications for security quality. < No categorization of differences or metrics to measure pluses and minuses of testing techniques/tools. < No single source of information on what tools are available for testing applications < No independent source of information on testing tools that are available. < Some tool vendors do a terrible job of educating the consumer on what they do and don’t do, and how they relate to their competition. OWASP App. Sec DC 2005 9

The Problems: <Unclear and immature terminology <Few Taxonomies, Categorizations, Metrics and Models on App. Sec <No Business Rule/Risk Taxonomy <No mature Threat Models/Taxonomy <No accurate Attack Models/Taxonomy <Other Taxonomies needed <We need a Taxonomy of Taxonomies OWASP App. Sec DC 2005 10

This Presentation is About: < The OWASP Tools Project Version 1. 2 which aims to: < Categorize the different ways to “test” an application for security. < Categorize the variety of tools that may be used “test” applications for security. < Provide basic insight into features and functionality of “software security tools”. < Offset the active dissemination of misinformation about what tools can and cannot provide. < I bit off more than I can chew here (why are there so many tools out there for this? ? ? ) OWASP App. Sec DC 2005 11

This Project is Headed: < The OWASP Tools Project Version 2. 0 which aims to: < Categorize & Classify tools that may be used “test” applications for security. < Work with groups like OWASP, WASC, NIST, and Mitre/CVE < Evaluate the value of & possibly provide checklist charts of features and functionality in specific versions of appsec tools. < I have concerns about above because for example not all XSS checks are equal, and the quality can and does change overnight in current tools. < In non-COTS software metrics tend to lack pragmatic value as software continues to increase size, complexity, extensibility, and develop emergent behaviors as disparate technologies are bolted together and new software evolves. OWASP App. Sec DC 2005 12

Limited Information; Full-on Equivocation I. Things Commercial Tool Vendors like to say: − Sanctum: “Oh, those are *network* scanners” − Compuware: “the only ones in the world who do this” − Cenzic: “We’ve automated webapp pen testing” − Several: “Oh, no, that’s a CVE scanner. It doesn’t find anything on our sample application which has “real” vulnerabilities. See here, we’ll just install our sample application we wrote right here and show you…” − Tautology: The Scanner and Firewall Vendors II. Goings-on in the App. Sec Industry: − No Standards or Standards-bodies − No Metrics − No Watchdogs − Vendors provide the vast bulk of “Factual Literature” OWASP App. Sec DC 2005 13

Chapter II: The Definition What is Application Security? OWASP App. Sec DC 2005 14

Why and What is Application Security? : Why do you care about App. Sec Testing Tools? When will the tools be mature enough that I just click “Scan” and get my report? OWASP App. Sec DC 2005 15

Why do you care about App. Sec Tools? <Because you lack expertise <Because you lack subject information. <Because you lack resources. <Because you lack time. <Because of the desire for automation. <Because of the need for reliable, repeatable tests that produce accurate quantifiable information about software defects with security implications. OWASP App. Sec DC 2005 16

When can I click ‘Scan’? Never. There are things that we can automate and things that we cannot automate. There are issues that are easy to automate reliable detection and issues that are not: - business logic flaws - workflow bypasses - human eyeball parsable errors. Terminology: Functional versus Technical Issues OWASP App. Sec DC 2005 17

Functional versus Technical Issues There are two main types of application security issues: I. Technical Vulnerabilities 1) Lack of Input Validation, resulting in: 2) XSS 3) SQL Injection 4) Parameter Tampering, etc. II. Functional or Logical Vulnerabilities 1) Click two or more buttons at the same time results in unexpected behavior. 2) Password change form disclosing detailed errors. 3) Session-idle deconstruction not consistent with Policies 4) Spend deposit before deposit funds are validated (B/L) OWASP App. Sec DC 2005 18

Functional versus Technical Issues These two types of security issues must be tested differently: I. Technical Vulnerabilities − Usually about Data Handling − Dealing with Strong Typing, Encoding, and Validation of Data that the application handles − Can be tested fairly effectively by automated tools (at least, in theory, as tools mature) II. Functional or Logical Vulnerabilities − Deals with issues allowed by design, but not foreseen by the designers (or understood to be a risk) − Deals with eyeball issues—does the error message text provide a brain-decipherable “secret message”? OWASP App. Sec DC 2005 19

Two Items to Reiterate Restated: I. Automation Tools − Testing: useful; use daily − Protection: maybe useful, maybe not (is your issue XSS? Do you know what your issues are? ) − Anecdote of client who solved 90% of their issues even though they didn’t know what they were. − Anecdote of WAF vendor who “successfully deployed in under an hour” for the java applet communicating on TCP/5000 with an encrypted binary protocol. II. Examples of Vendor Challenges − Sanctum Support Issues: XSS vs. Phishing debate − Other vendors “Phishing” “XSS” “blahfoo” feature FUD OWASP App. Sec DC 2005 20

What is “Application Security”? I. Things that Application Security is not: − It is not a product − It is not a firewall with “AI” (app intelligence) − It is not a NIDS/NIPS with app signatures − It is not any “perimeter access control”. − It is not a “web app firewall” widget. − For today and tomorrow’s web applications the concept of a perimeter is dead. OWASP App. Sec DC 2005 21

What is “Application Security”? II. Things that Application Security is not: − It is not a process (it is possible, however unlikely, to have secure applications with or without a process) − It is not “application hacking school” or “secure coding school” for your developers − It is not the BPPT (Big Production Penetration Test) you have at the end of your BUFD app lifecycle. OWASP App. Sec DC 2005 22

What is “Application Security”? III. Things that Application Security IS: − It is one possible emergent Quality of a well designed application. − It is about predictability, dependability, reliability. (Business speak !=“security”) − Rob’s Report must Run Right (bugs are not difference of kind but difference of degree and the issue is always the same) OWASP App. Sec DC 2005 23

How do we evaluate applications for Security? First and most important application security principle: <The Application Must Defend Itself OWASP App. Sec DC 2005 24

Presentation Outline Updated I. Introduction (done) II. Chapter One: The Problem (done) III. Chapter Two: The Definition (done) IV. Chapter Three: The Clarification and Goals V. Chapter Four: Taxonomy of Testing VI. Chapter Five: Tools of Automation VII. Chapter Six: Q&A (Shoot the Messenger) OWASP App. Sec DC 2005 25

Chapter III Clarification and Goals OWASP App. Sec DC 2005 26

The Categorization Basics: < < Software flaws with Security Implications Category Class Particular -Example- 1. Category (Coding Error) 2. Class (Output Encoding) 3. Particular (XSS (Cross-Site Scripting)) OWASP App. Sec DC 2005 27

Are we trying to Categorize and Classify? Clarification of Terms Categorization of Issues Abstract fixation from particulars Needs: Category, Class, Particular Needs Risk, Threat, Attack, Weakness, Vulnerability < Needs account for Severity, Attackability (Ease), Prevalence, Priority < < < OWASP App. Sec DC 2005 28

The Classification Basics: <Risk ((Loss|Impact) : Financial, Legal, Market) <Threat (Likelihood, Severity, Ease, Implication) <Attack (Exploit Vector) <Weakness (Fundamental Issue) <Vulnerability (Particular Issue/Flaw Instance) OWASP App. Sec DC 2005 29

The Classification Breakdown: -Example One- < Risk 4 Financial Loss, Repudiation, Market Goodwill < Threat 4 Elevation of Privilege; Forged Transaction < Attack (A: Spoofing of User Identity) 4 A 1 Session Fixation § A 1 -2 Session TokenCookie set a priori user authentication < Weakness 4 Insecure Authorization Mechanism; Insecure Session Handling < Vulnerability 4 Harvest Session Cookie on Login Form OWASP App. Sec DC 2005 30

The Classification Breakdown: -Example Two- < Risk 4 Financial Loss(? ? ? ), Market Goodwill < Threat 4 Elevation of Privilege; Sensitive Information Disclosure < Attack (A: Arbitrary Script Injection) 4 A 1 XSS/Cross-Site Scripting § A 1 -2 Embedded/Persisted Malicious Javascript < Weakness 4 Insecure Output Encoding < Vulnerability 4 Support forum page allows embedded linking to external javascript/flash/actionscript/etc. OWASP App. Sec DC 2005 31

Example Problems with Existing Threat Models: Microsoft: STRIDE and DREAD <Spoofing of User Identity (Threat or Attack? ) <Tampering with Data (Attack? ) <Repudiation (Risk or Threat? ) <Denial of Service (Threat or Attack? ) <Elevation of Privilege (Threat) STRIDE, DREAD and others mix of Threats, Attacks, Vulnerabilities, disparate elements. New TRIKE methodology shows promise. OWASP App. Sec DC 2005 32

More on Threat Modeling: Threat Modeling is very important; I use it extensively particularly in communicating to business owners or stakeholders of an application <For more see the OWASP Testing presentation <The OWASP Testing GUIDE <STRIDE & DREAD in MS Threat Modeling Book <TRIKE is a new Threat Modeling Whitepaper <Visit the Threats and Countermeasures Website by Foundstone OWASP App. Sec DC 2005 33

How do we evaluate applications for Security? First and most important application security principle: <The Application Must Defend Itself OWASP App. Sec DC 2005 34

Conclusion to Chapter III The previous material is presented to provide a common set of terms and framework for understanding what we can and can’t do with automated tools that discover software flaws with security implications. < Security is not a “feature” you implement in code < Security is an emergent Quality of well-designed and well-built software, much like speed and reliability < Categorical difference between technical and functional issues and how you discover them < Tools tend to flag everything as “vulnerabilities” < Tools really find a mix of threats, attacks, and vulnerabilities (XSS is an attack, not a vulnerability) < Tools tend not to indicate systemic weaknesses, like improper output encoding techniques, which is what we really need to be concerned with if our focus is *fixing* the application issues. OWASP App. Sec DC 2005 35

Chapter IV Taxonomy of Testing OWASP App. Sec DC 2005 36

Taxonomy of Testing Categories 1. 2. 3. 4. 5. 6. 7. 8. Vulnerability Scanning Fault-Injection Testing (Blackboxing) Fault-Injection Testing with Sandboxing Binary Analysis Source Code Analysis Threat Modeling/Architectural Analysis Rootkit and Trojan Analysis (What about runtime analysis and runtime patching? ) OWASP App. Sec DC 2005 37

Taxonomy of Testing Categories 1. Vulnerability Scanning } regex matches on version numbers. 2. Fault-Injection Testing (Blackboxing) } Inject all possible faults and parse responses. 3. Fault-Injection Testing with Sandboxing } Inject all possible faults and analyze data storage, manipulation, and local entry and exit points, and fuzz/fault-inject those points. 4. Binary Analysis } Analyze static binary or perform runtime analysis and look for breakpoints. OWASP App. Sec DC 2005 38

Taxonomy of Testing Categories 5. Source Code Analysis } Mostly static signature matching, but some tools claim to walk codepath, follow nested loops and other regression through objects 6. Threat Modeling/Architectural Analysis } Detailed, logical modeling. Time consuming, thorough; avoid wasted man hours of broken feature-implementation up front. Or find obvious design flaws quickly. 7. Rootkit and Trojan Analysis } Why not? The NCUA/OCCU mandates it. 8. What about runtime analysis and runtime patching? Another idea… OWASP App. Sec DC 2005 39

What Follows The next series of slides is a short summary of common and well-known applications that fall into these categories. It is not intended to be comprehensive (later slides will be more comprehensive). It is not intended to indicate quality or imply any form of advocacy for the listed software security tools. If you are looking for product advocacy there are plenty of sales people and “sales engineers” that can help you there. Most importantly: These tools are constantly evolving. Consider that the data presented may already be outdated. Use as a guide only and evaluate any selection you make Yourself, on Your Applications. OWASP App. Sec DC 2005 40

Vulnerability Scanning 1. 2. 3. 4. 5. 6. 7. ISS Internet Security Scanner (the old basic VA tool) e. Eye Retina (CGI Scanning? ) Qualysguard (XSS, limited SQLi) NGS Typhon (claims full JS Parsing) Mc. Afee Foundscan (DNE for webapps) n. Circle IP 360 (similar to Qualysguard) Known-File Scanners e. g. -Whisker, Nikto, Nessus+Nikto, Nstalker Nstealth 8. These tools do a regex version match (e. g. 12. 8. 1. 0 diff vulndatabase) or simply match a filename (e. g. “login. asp”…Oh No!, not a Login!) 41 OWASP App. Sec DC 2005

Fault Injection Testing 1. 2. 3. 4. 5. 6. 7. 8. 9. SPI Dynamics Web. Inspect 5. 5 Watchfire App. Scan 5. 5 Acunetix WVS 2. x Cenzic Hailstorm 2. 6 NTO Spider x? Compuware Devpartner x? NGS Typhon x? Parasoft Webking X? SPIKE (while using binaries with attached debuggers to monitor breakpoints like any traditional fuzzing) 10. Peach Fuzzer 11. These tools inject a fault into the application and in some cases evaluate the response. OWASP App. Sec DC 2005 42

Fault Injection with Sandboxing Tools: • Security Innovation’s Holodeck • Sysinternals filemon, regmon, procmon • Any sandboxing malware-analysis tools OWASP App. Sec DC 2005 43

Binary Analysis 1. 2. 3. 4. IDA Pro @stake Smart. Risk Analyzers Fortify Application Risk Analyzer Halvar Flake’s Sabre-Security projects – Bin. Navi – Bin. Diff – Bin. Audit 5. Wi. SA OWASP App. Sec DC 2005 44

Source Code Analysis 1. 2. 3. 4. 5. 6. 7. Compuware Coverity Fortify Software Ounce Labs Parasoft Secure Software Lots O’ Free & Open Source Stuff OWASP App. Sec DC 2005 45

Threat Modeling / Architectural Analysis 1. 2. 3. 4. Microsoft Threat Modeling Tool Securi. Tree PTA Technologies TRIKE TM tool (to be released) OWASP App. Sec DC 2005 46

Rootkit/Trojan/Backdoor Analysis Checking for Rootkits, Trojans, and Backdoors is another interesting issue that is recommended in the “Secure Coding” pamphlet sold as a book by O’Reilly, and recommended by the NCUA 2004 -03 Guidance letter “must review source code for backdoors, trojans, etc. ” 1. Rootkit and Binary analysis tools for Backdoors & Trojans OWASP App. Sec DC 2005 47

Runtime Analysis and Patching Tools that are designed to analyze behavior and code at runtime and perform runtime limitation/security enforcement and/or “virtual” patching of userspace coded and kernel code. 1. Immunix kernel & stack protection. 2. Some new virtual patching product for Linux that I accidentally deleted the info on and can’t find the emails of the kind folks who contacted me originally. OWASP App. Sec DC 2005 48

Chapter V Automation Tools (or manual human eyeball supplicants) YMMV Key: MA (Mature) HP (Has Promise) DNE (Did Not Evaluate) RE (Refused Evaluation or ignored repeated requests) TOY (Amusing little widget) OWASP App. Sec DC 2005 49

Vulnerability Scanning 1. 2. 3. 4. 5. 6. 7. 8. 9. ISS Internet Security Scanner (? ? ? ) e. Eye Retina (CGI Scanning? ) Qualysguard (XSS, limited SQLI) Mc. Afee Foundstone Enterprise n. Circle IP 360 Rapid 7 Nexus NGS Typhon (full JS Parsing; XSS & SQLI) Saint Known-File Scanners e. g. -Whisker, Nikto, Nessus+Nikto, Nstalker Nstealth OWASP App. Sec DC 2005 50

Commercial Fault Injection Test Tools 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. SPI Dynamics Web. Inspect (5. 5, MA) Sanctum now Watchfire App. Scan (5. 5 MA but may become a ‘dashboard’) Cenzic Hailstorm (HP…ways to go, version 2. 6 DNE) NT Objectives NTOSpider (HP, DNE new GUI version) Acunetix Web Vulnerability Scanner 2 (TOY, no scheduling, state, form-fill, etc. ) Compuware Dev. Partner Fault Simulator (RE) Whitehats Security (hosted application, ‘dashboard’ approach, think ‘Qualys’) Fortify Pen Testing Team Tool (RE, unknown; secretive) Burp Intruder (HP, nice fuzzer) Max. Patrol 7 (strange phone-home to Russia traffic observed, stopped eval) Syhunt Sandcat Scanner & Miner (evolving) Trust. Security. Consulting HTTPExplorer (TOY, limited) Ecyware Blue. Green Inspector (TOY, limited) NGS Typhon (DNE, claims full parsing/testing capability) Parasoft Web. King (DNE, positive references from credible sources) Kavado Scando (dead, IP purchased, may show up again, CVE-type scanner) App. Sec. Inc App. Detective for Web Apps (dead, believed permanently) @stake Web Proxy 2. 0 (dead or suffocating at Symantec) Sandsprite Web Sleuth (HP, but development End-of-Life? ) OWASP App. Sec DC 2005 51

Commercial Fault Injection Test Tools Data on FI Tools: Data includes version number as many of these tools have new features/improvements since time of testing: < Tools tested from March to August of 2005 < Tools tested on sample applications like OWASP Web. Goat < Tools tested on multiple real-world applications < Tools tested on same versions of applications < Tools tested on “default” settings and also attempted to customize tools as best possible to provide optimal use-case scenario for final result data < Many tools were configured with help from or onsite support from the vendors, all installed on clean, new boxes, so should represent the best the vendor could provide at that time. < I have been guilty of RTFM errors before, but for this testing this is why vendor configuration support was solicited. OWASP App. Sec DC 2005 52

Commercial Fault Injection Test Tools SPI Dynamics Web. Inspect 5. 5 Pros: < Best manual testing tools of the FI market (Proxy, Cookie Tester) < Excellent javascript parsing < Most powerful tools for writing custom checks < Best API integration with other QA tools < Have removed a lot of the marketing crap that other tools still report like detecting “ 31 Flavors of XSS” and Cons: < No ability to look under the hood on adaptive agents < No ability to customize default checks (to protect “IP”) < Come on guys, I can run a sniffer and see what you are doing anyway < Not the fastest OWASP App. Sec DC 2005 53

Commercial Fault Injection Test Tools Watchfire App. Scan 5. 5 Pros: < Fast on smaller applications < Best XSS testing < Most powerful right out of the box with no tweaking < Excellent reporting < New free manual testing toolkit; ‘Power. Tools’ Cons: < No ability to look under the hood or customize effectively < Unnecessary XSS variants, and made-up vuln terminology < Watchfire is a dashboard company and this is not a effective way to evaluate/measure application security < Limited ability to tweak to handle dynamic session affinity and complex authentication variants OWASP App. Sec DC 2005 54

Commercial Fault Injection Test Tools Cenzic Hailstorm 2. 3 Pros: < Allow you to get under the hood and customize everything. < Most potential for automating complex custom checks. < Nice drag-and-drop configuration in GUI. < Definitely a contender to keep an eye on. Cons: < Custom checks simply didn’t work right < Slowest tool tested < No ability to configure operational/engine parameters like threading, bandwidth, scope, etc. < At time of testing was long on hype, highest priced, low on delivery of results OWASP App. Sec DC 2005 55

Commercial Fault Injection Test Tools NT Objectives NTOSpider 1. something CLI-only beta version Pros: < Fast. Wow, this thing is fast. < Most effective XSS checks behind WI and AS. < Only tool with effective automated session token testing. < Auto-state detection and auto-login abilities. < Small, lightweight, simple configuration. Cons: < No CVE-style checks. < No ability to look under the hood or customize tests easily. < Roughly half the accuracy of WI or AS on XSS and SQLi. < No GUI at time of testing. OWASP App. Sec DC 2005 56

Commercial Fault Injection Test Tools Acunetix WVS Pros: < Nice, pretty GUI < Appears fast, but it doesn’t do much so I guess it’s easy to be fast at that. Cons: < Limited authentication support < No scripting or scheduling capabilities < Utter lack of session/state handling < No manual analysis tools < False positive centric < Marketing appears targeted at sub-100 IQ demographic < Included due to massive advertising blitz recently OWASP App. Sec DC 2005 57

Commercial Fault Injection Test Tools Compuware Dev. Partner Fault Simulator Pros: < Built with the Developer in mind < IDE integration < Integrates with Source-Code analysis < May be one to keep an eye on but looks like not intended to be “best of breed” solution. Cons: < Vendor has no clue about competitive landscape. < Vendor appears to have limited security knowledge. < Did Webex eval only. < Got tool but not license key; vendor promised follow up repeatedly but never did. < Vendor to provide data for the OWASP Tools project but never followed through in providing this. OWASP App. Sec DC 2005 58

Commercial Fault Injection Test Tools White. Hat Security (the Seattle Company) Hosted FI Tool; think Qualys of Web. App. Sec Pros: < DNE (Did Not Evaluate) < Some smart people building this < Hosting approach nice for repeated testing/change management < Cost effective Cons: < No replacement for manual testing < DNE OWASP App. Sec DC 2005 59

Commercial Fault Injection Test Tools NGS Typhon Pros: < DNE for Web Application Checks < Experience with NGS tools is that they are very fast, fairly accurate, lack ability to customize/configure much through the GUI and have terrible reporting. < Will include in next release of this project. Cons: < DNE. OWASP App. Sec DC 2005 60

Commercial Fault Injection Test Tools Parasoft Web. King Pros: < DNE < Have heard has is less mature but some nice features < Will include in next version of project Cons: < DNE OWASP App. Sec DC 2005 61

Commercial Fault Injection Test Tools Sand. Sprite Web Sleuth Pros: < Inexpensive < Simple easy-to-use Windows GUI < Nice code markup and views < Custom scripts for crawling, cookie testing, etc. Cons: < Geared towards all manual testing < No default pre-built checks < Doesn’t appear under active development OWASP App. Sec DC 2005 62

Commercial Fault Injection Test Tools OWASP Web. Scarab v. 20051012 Pros: < Many advanced testing features geared towards experienced application security professionals. < Free & Open Source. < Session Token entropy analysis. < Static string fuzzer. < User-extensible with custom agents. < Web Services testing interface for complex types Cons: < Non-intuitive, complex interface. < No “scan” or pre-built checks like CVE issues. < Tends to hang and need restarted. < No vulnerability reporting mechanism or management style reporting a’la Watchfire. OWASP App. Sec DC 2005 63

Fault Injection Test Tools Anecdotes M Similar to diet fads M My Book “Body of Anecdotal Evidence: Portionsized Marketing” out soon M That was a “Body for Life” joke for you not from the US M Again, there is no book for those of you that asked me after the presentation. M YMMV M Anecdotes are subjective M Anecdotes are not logically valid proofs M Repeat Disclaimer: YMMV OWASP App. Sec DC 2005 64

Fault Injection Tool Experiences The following metrics were gathered from realworld assessments performed by myself along with up to four other testers. These were commercial assessments on a mix of COTS software and custom applications developed by clients ranging from Fortune 100 to smaller ISVs that authorized benchmarking of these tools during the course of software security assessment. OWASP App. Sec DC 2005 65

Fault Injection Tool Experiences Application #1: < 150+ pages, Complex workflow < Dynamically built javascript menu/functions < Application contained 1 trivial XSS, 1 Workflow Bypass (WFB), 1 Brute Force (BF) < The tools in this test were run with default configurations; essentially clicked ‘scan’ as many people unfortunately use these tools in Real. Life™. Web. Inspect 5. 0: 147 pages, no XSS, WFB, BF App. Scan 5. 5: 38 pages, 1 XSS, no WFB, BF Acunetix 2: 6 pages, no valid positive results NTO Spider: 80+ pages, no XSS, WFB, BF Paros 2. something: 0 (zero) (loop) Cenzic Hailstorm: Couldn’t get working (The security defects detected in the applications were discovered during the course of manual analysis) OWASP App. Sec DC 2005 66

Fault Injection Tool Experiences Application #2: < 30+ templates, Dynamically generated content < 15 k+ actual pages, Long forms, simple workflow < 32 XSS from URL parameters to multi-form <--!--> < Weak Session Handling; Auto-scripting attacks (“Session Riding” or “Web Trojans”) < Tests run as ‘scan’ with minimal tweaking Web. Inspect 5. 0: 31 XSS, no SH or SR issues App. Scan 5. 5: 18 k+ pages, slowed to 1 rec/6 seconds Acunetix 2: No Results (couldn’t automate effectively) NTO Spider: 18 XSS, weak SH, no SR, fast! (CLI version) Hailstorm 2. 2: 3 days, 4 people, Cenzic onsite, nothing Paros 3. 0. 2: 18 pages, no results, would hang on audit OWASP App. Sec DC 2005 67

Fault Injection Tool Experiences Application #3: < 100’s of pages/forms < Strong UI, Input Validation, Output Encoding < Web Server packages XML POST string < Passes to routing servlet for authorization < Function Servlet name/route in Hidden FF < Create XML String and POST directly to DST servlet < XML Abstraction Layer has no authentication/authorization < ASMQ does not have crypto enabled; auth is function group code < Point of this example is that there are holes you could drive a Mac Truck through and every FI Scanner failed to detect them. Some things require humans. Web. Inspect 5. 5: Nothing App. Scan 5. 5: Nothing; WF Power. Tools Proxy: failed Acunetix 2: PHP issue, some other issues, false positives (200 codes) NTO Spider: Nothing (CLI version); no proxy Paros 3. 0. 2: Nothing; Proxy-mode: failed Proxies: SPI Proxy (only proxy to handle various cert-auth) OWASP App. Sec DC 2005 68

Fault Injection Tool Experiences Application #4: < AAWBDMS (Another Awful Web-Based DMS) < XSS, Parameter Tampering, etc. (everything you can imagine controlled in client-side hidden form-field parameters) < URL Parameters passed directly to compiled C code (DLLs, EXEs) running as Domain Admin with traditional buffer overflows. < Dual-Factor Authentication < Authentication Two-step Workflow < First page User. ID & PRNG Token number in HTML Form < Second Page identical User. ID and Password in HTML Form < All responses 200; everything returns one of two login forms Every darn tool: NADT: Not a Darn Thing Nobody could handle dual-auth with custom error pages identical to the login pages. OWASP App. Sec DC 2005 69

Fault Injection Tool Experiences XSS attacks should be fully automated, but as of yet are not. We find XSS in almost every application we test that is not found by any automated tool; some are trivially devastating. Examples: < < < Form value expects email address (arian. evans@fishnetsecurity. com) Form value validated by sloppy regex; Output not encoded Form error returns no data unless you bypass regex filter Tools submit the following types of checks: 4 4 4 <script>alert(‘somethingclever’); <script>silly@domain. com<script>alert(‘somethingclever’); <script> some replace ‘silly’ and ‘com’ with <script>alert(); <script> check some add escape characters like ‘>, ‘>>, ‘), and so on 4 4 4 encoding attack strings (reference OWASP Guide 2. 0. 1 or RSnake’s XSS page) partial encoding of attack string elements using specific HTML tags like body elements and background Actual XSS by silly@<script>alert()<script>. com Other attack vectors tools fail include OWASP App. Sec DC 2005 70

Open Source or Freeware Fault Injection Test Tools 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. Web. Scarab (DNE current version, re: HTTPush, Exodus) Paros Proxy (HP) Watchfire Power. Tools (HP, free, proxy nice GUI, rest so-so) Burp Spider (HP, free) Burp Proxy (HP, free) Snark HTTP Proxy & Tabby Tunnel (DNE) Peach Fuzzer (HP, free, Python fuzzer) SPIKE Proxy (Horrible interface, free, limited fuzzing) SPIKE Fuzzer (dev EOL? Dave pointing people at Peach) Achilles Proxy (TOY, first Windows proxy, free, old) Odysseus Proxy (some cool features, GUI needs work) Webstretch Proxy (DNE) Absinthe 1. 1 (formerly SQLSqueal) (HP, free) Sensepost E-Or, Wikto (Google hacking utility), other webappsec and pen testing tools (everything Sensepost is working on is worth looking at) OWASP App. Sec DC 2005 71

Open Source or Freeware Fault Injection Test Tools And your Basic Web Browsers…don’t forget them: 15. 16. 17. Internet Explorer + HTMLBar DLL extension (buggy) Firefox + Live. HTTP Headers, Tamper Data, Developer Tools, and many, more useful extensions…. however, keep in mind Firefox encoding is often different, more restrictive, sometimes more secure than IE so if the application client population includes/is predominantly IE MAKE SURE you test encoding/format in IE as well. Foundstone Free Tools: § Site. Digger (Google Hacking Utility) § SSLDigger § Probably others I haven’t looked at yet § Site. Generator (coming soon, may be useful for testing Fault-Injection tools for javascript parsing, etc. ) OWASP App. Sec DC 2005 72

Fault Injection with Sandboxing Tools (Commercial & Free): 1. Security Innovation’s Holodeck (HP) 2. Sysinternals filemon, regmon, procmon can be combined with VMWare 3. Any sandboxing malware-analysis tools available from various forensics sites and possibly from the AV vendors. OWASP App. Sec DC 2005 73

Binary Analysis 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. IDA Pro + scripts from Halvar & i. Defense Grammatech Code. Sonar Paradyn Project (claims superiority to IDA Pro when C++ variables present in object) FX Dumbug @Stake Smart. Risk Analyzers (gone/dead) Fortify Application Risk Analyzer (marketing freebie) Wi. SA (Wisconsin Safety Analyzer) SABRE Bin. Diff (IDA Pro extension) SABRE Bin. Navi (Graphical flowgraphs, playback of codepaths, Win/Linux debuggers) SABRE Bin. Audit (Full Binary Analysis; beta project; Halvar restarted this again) Secure Software Code. Assure Auditor (Binary Analysis) Brute Force Binary Checker (http: //bfbtester. sourceforge. org) OWASP App. Sec DC 2005 74

Commercial Source Code Analysis 1. 2. 3. 4. Compuware Dev. Partner Security. Checker. NET Languages Coverity SWAT C/C++ Escher Technologies Eschertech Fortify Software Suite (analysis, workbench, metrics & trending console, customization module, won’t show me anything) C/C++, Java, JSP, HTML, Java bytecode, (PL/SQL), C#, VB. NET, MSIL, … 5. Gimple PC and Flexe-Lint C/C++ (we use, simple, cheap, functional) 6. Grammatech Code. Surfer C/C++ 7. Ounce Labs Prexis C/C++, JSP, Java (nice, rough UI, recursion abilities? ) 8. Parasoft JTest Java (? NE) 9. Reasoning Security. Inspect “Service” 10. Secure Software Code. Assure Workbench C/C++, Java 11. Security-Assessments Code. Scan ? (April 01 05) won’t give specifics 12. 13. Note if you are in sales: My name is not Ariel, Arial, Adrian, Aruin, etc Wi. SA C/C++ OWASP App. Sec DC 2005 75

Open Source, Free, or Bundled Source Code Analysis 1. 2. 3. OWASP. NET (Reference Dinis Cruz presentations on OWASP website about OWASP. NET tools) Foundstone. NET Tools (same story as above; DNE but heard these were coming out soon) The Next Slide has All The Other Stuff OWASP App. Sec DC 2005 76

Open Source, Free, or Bundled Source Code Analysis 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. Bunch http: //serg. cs. drexel. edu/bunch Cigital ITS 4 (no longer supported) C/C++ CQual http: //www. cs. berkeley. edu/~jfoster/cqual David Wagner’s BOON C http: //www. cs. berkeley. edu/~daw/boon/ David Wheeler’s Flawfinder C/C++ www. dwheeler. com Reference/Credit David Wheeler’s Page Microsoft PREFix (full codepath execution, recursion), C families Microsoft PREFast (stripped down PREfix) only static signatures for performance reasons, shipping with Visual Team Studio 2005, C families Microsoft FXCop (even more limited signature tool) C/C++ only MOPS http: //www. cs. berkeley. edu/~daw/mops PScan http: //www. striker. ottawa. on. ca/~aland/pscan RAT C/C++, Java, Perl, Python, whatever other random signatures written Share. Fuzz SPLINT http: //splint. org/ OWASP App. Sec DC 2005 77

Threat Modeling / Architectural Analysis 1. Microsoft Threat Modeling Tool 1. (non-intuitive for beginners) 2. Securi. Tree 1. (clunky Java tool, not appsec focused) 3. PTA Technologies 1. (DNE, have had several references) 4. TRIKE Threat-Modeling Tool 1. (DNE, in development stage, unsure release 2. date, but idea has promise) 5. Need more tools here! OWASP App. Sec DC 2005 78

Rootkit/Trojan/Backdoor Analysis Checking for Rootkits, Trojans, and Backdoors is another interesting issue that is recommended in the “Secure Coding” pamphlet sold as a book by O’Reilly, and recommended by the NCUA 2004 -03 Guidance letter “must review source code for backdoors, trojans, etc. ” 1. Rootkit and Binary analysis tools for Backdoors & Trojans 2. rootkits. org and rootkit. nl + many Forensic Distributions have these tools OWASP App. Sec DC 2005 79

Recommended Motto for Vendors Ne Supra Crepidum Suter Judicaret OWASP App. Sec DC 2005 80

Chapter 6 Shoot the Messenger (he may duck and run) OWASP App. Sec DC 2005 81

Arian J. Evans Does Application Security Stuff Fish. Net Security 913. 710. 7085 [mobile] arian. evans@fishnetsecurity. com OWASP Tools Project OWASP Chapter Leader, Kansas City Feel free to contact me with questions Vendor updates, corrections, confusions, or complaints welcome Email me repeatedly if I miss your first one OWASP App. Sec DC 2005 82