Security Measures and Metrics Pete Lindstrom Research Director

Security Measures and Metrics Pete Lindstrom Research Director Spire Security

Agenda l Elements of metrics l Interlude: Four disciplines l Back to metrics l ROI/ROSI

Status of security l Difficult to define “good security” l Minimal difference between security and “lucky” l We don’t know how to measure success. l One incident doesn’t necessarily mean “failure”

Key elements of security metrics

Key elements of security metrics Activities: Four Disciplines Time: Hr/Day Month/Yr Resources: User accts, systems, apps People: Admins by Department Costs: Salaries, Consulting HW, SW, Maint. Building Blocks Let’s put them together…

Agenda l Elements of metrics l Interlude: Four disciplines l Back to metrics l ROI/ROSI

Process Effectiveness Metrics a. k. a. “doing things right” Elements: • Activities • errors For example: • Accts person • Vulns person • Patches person error rates

Security reference model 2. Control sources (users/others) Identity MANAGEMENT 4. Monitor/detect inappropriate and/or malicious activity Trust MANAGEMENT Vulnerability MANAGEMENT 1. Harden the 2. Infrastructure 3. Harden the Process/data Threat MANAGEMENT

Four disciplines of security management Identity Threat MANAGEMENT Identity Validation Account Management Password Management Threat Identification Security Monitoring Incident Management Authentication User Access Control Intrusion Prevention Encryption Integrity System Access Control INLINE Policy Management Security Arch. Design Ticket Management Vulnerability Assessments Patch Management Software Security Trust Vulnerability MANAGEMENT

Identity management Functions l Identify users l Assign accounts/rights l Maintain identity (passwords) l Validate sessions l Authorize access

Vulnerability management Functions l Scan for exposures l Eliminate vulnerabilities l Remediate vulnerabilities l Mitigate vulnerabilities l Manage compliance

Trust management Functions l Write policies l Design security l Ensure confidentiality l Ensure integrity

Threat management Functions l Analyze traffic l Analyze logs l Manage incidents l Conduct forensics

Agenda l Elements of metrics l Interlude: Four disciplines l Back to metrics l ROI/ROSI

Process Effectiveness Metrics a. k. a. “doing things right” Elements: • Activities • errors For example: • Accts person • Vulns person • Patches person error rates

Process effectiveness l Error rates l Identity management • Request errors l Vulnerability management • Vulnerabilities remaining l Threat management • Incident response l Trust management • Policy violations

Staff productivity Staff Productivity Metrics a. k. a. “people doing things” better Elements: • People • Activities For example: • Accts person • Vulns person • Patches person

Staff productivity l Productivity and workload for all manual activities (activities/people) l Identity management • Requests per administrator • Account disablements per admin • Password resets per admin l Vulnerability management • Vulnerabilities resolved per administrator l Threat management • Incidents person l Trust management • Policy changes person

Cycle Time Metrics a. k. a. avg “time to Cycle Time perform activity x” Elements: • Time • Activities For example: • Accts per month • Vulns fixed per month • Patches per month

Process efficiency (cycle time) l Time/activities l Identity management • Request time l Vulnerability management • Remediation time l Threat management • Incident response time l Trust management • Policy creation time

Admins by Department Efficiency Metrics Efficiency a. k. a. “people doing things” quicker 2000 Hours per FTE Elements: • People • Activities • Time For example: • Accts/person/hr • Vulns/person/hr • Patches/person/hr

Admins by Department Cost Effectiveness Metrics Elements: • People • Activities • Costs For example: • Cost per acct • Cost per vuln fixed • Cost per patch Cost effectiveness a. k. a. “people doing things” cheaper Salaries, Consulting Fees

Cost effectiveness l Dollars/activities; dollars/resources; dollars/demographics l Identity management • Cost per request • Cost per password reset l Vulnerability management • Cost per vulnerability • Cost per system setting l Threat management • Cost per incident l Trust management • Cost per policy • Cost per project

When to use metrics l Process effectiveness • Six Sigma l Staff productivity • ROI / promotions l Cycle time • Balanced scorecard l Efficiency • ROI l Cost effectiveness • Activity-based costing • ROI/TCO

Business uses of security l Benchmarking (Balanced scorecard) l Baselining (Six Sigma) l Activity-based costing/Mgt l ROI l Risk management (ROSI)

Elements: • Activities • Resources Missing Element: RISK!Risk Management Four Disciplines: • Identity Mgt • Vuln Mgt • Trust Mgt • Threat Mgt a. k. a. “people doing things” more securely!

Risk metrics l Resources/resources; resources/demographics l Identity management • User accounts per application l Vulnerability management • Vulnerabilities per resource l Threat management • Incidents per resource l Trust management • Policies per resource

Risk effectiveness l Activities/activities (automated) l Identity management • Failed logins/total logins l Vulnerability management • Access denied/total access l Threat management • Incidents/events l Trust management

Agenda l Elements of metrics l Interlude: Four disciplines l Back to metrics l ROI/ROSI

Examples: Return on Investment (ROI) & Return on Security Investment (ROSI)

The elements of value (Loss) l ROI • • IT productivity (time) User productivity (time) …these also have ROSI value l ROSI • • • Legal/regulatory costs (fees/fines) Direct revenue Stored asset value (intellectual property, financial assets)

Let’s talk ROI l Keyword is efficiency l Reduced Capital Expenditures (Cap. Ex) • Lower h/w, s/w costs § Scalability, manageability, performance l Reduced Operating Expenditures (Op. Ex) • Lower IT, end-user costs § (higher productivity)

Productivity l Where users and IT spend their time. l Time-is-money philosophy. l Often the only aspect of loss we quantify. l Basic source of ROI. l Hourly rate x hours of effort. l In order to determine the value of activities, you first have to determine what activities are performed.

Identity management ROI l Provisioning • New employee productivity • Automated account management l Password management • Reduced help desk time • Employee productivity l Web access control • Developer efficiency (build vs. buy)

Trust management ROI l Public Key Infrastructure • Managing certificates l Virtual Private Networks • Leased lines l SSL Acceleration • Hardware efficiency

Vulnerability management ROI l Firewalls • Reduce ACL management l Vulnerability assess/remediate • Reduce manual efforts l Patch management • Automate patching l Software quality • Reduce bug fixes

Threat management ROI l Antivirus • Recovery of systems l Network IDS • Reduce manual detection/forensics l Host IDS • Manual log efforts l Security Event Management • Aggregation/prioritization of work

Getting to ROI l Identify amount of labor allocated to individual security activities. l Identify solution and its corresponding activities. l Identify labor difference with and without solution.

The roots of ROSI l Our overall objective is to reduce risk. l We are relatively “new” to spending on solutions. l We often didn’t really do anything that was considered a recurring expense (I am guessing a bit here). l But, the Internet has changed all that (or at least made it apparent).

Return on Security Investment l Keyword: Effectiveness l Effectiveness = Reduced risk l Protecting Value and Loss • • • Legal/regulatory costs (fees/fines) Direct revenue Stored asset value (intellectual property, financial assets)

Legal/regulatory costs l Lawsuits: • • • Privacy suits Downstream liability Legal fees l Regulatory issues: • • Regulatory fines Remediation costs

Direct revenue l E-Commerce systems l Level of materiality l Seasons, cycles, forecasts drive expected losses l Some benchmarks: shrinkage; materiality (internal controls)

Stored asset value l Stored Value (financial assets) l Stored Knowledge (intellectual property) l Market Cap (or equivalent) – Book Value = Goodwill (intangible assets) l Some % of this Goodwill is attributable to information assets. • Professional services – higher percentage • Contract manufacturing or retail - lower

Determining loss l No physical goods l Ubiquitous supply l Full asset value is not necessarily lost l Look at loss in other ways: • • Type of loss For each application/system

Types of losses l How much value would be lost under the following conditions (for each app/dataset)? l Information-centric loss • Modified data (Integrity) • Copied data (Confidentiality) • Deleted data (Availability) l System/App-centric loss • Resource availability (Productivity) • Resource misuse (Liability)

Loss potential Read Modify Delete Avail Misuse Asset Value H M M L L Revenue M H H H L Fines M/H H L L ? IT Prod. L H M L M EU Prod. L L M H M

Calculating potential loss Annual Loss Expectancy = Probability x Value ALE = P x A (Insurance Industry) l Level One: Calculate overall loss potential in 5 categories. • ALE = P x L(Assets, Revenue, Fines, IT Prod, EU Prod) l Level Two: Take above and factor in types of losses. • ALE = P x (C(A, R, F, I, E); I(A, R, F, I, E); A(A, R, F, I, E)) l Level Three: Perform above for all applications/data. • ALE = P x App 1(C(A, R, F, I, E); I(A, R, F, I, E); A(A, R, F, I, E))… Appn(C(A, R, F, I, E); I(A, R, F, I, E); A(A, R, F, I, E))

Getting to ROSI l Determines cost effectiveness of proposed solution. l Calculate losses with and without solution. l Compare the difference.

Agree? Disagree? Pete Lindstrom petelind@spiresecurity. com www. spiresecurity. com