Presentation to ISACA New England Chapter March 20

Presentation to ISACA New England Chapter March 20, 2008 Examination of IT Audit

Examination of IT Audit About Today’s Session: Speaker: Ken Fortier, CISA IT Examination Specialist Federal Reserve Bank of Boston Supervision, Regulation & Credit Department Large Bank Supervision Disclaimer: The opinions expressed are those of the speaker and do not represent official policy or guidance of the Federal Reserve Bank of Boston or the Federal Reserve System. 2

Examination of IT Audit About Today’s Session: Reach in for a handful of what you want. 3

Examination of IT Auditor or Examiner: What’s the difference? 4

Examination of IT Auditor or Examiner: What’s the difference? Who we work for. Auditor: Accountable to the Board of Directors • Audit Plan approved by the Audit Committee • Staffing & Budget governed by Executive Management Examiner: Accountable to their Federal or State supervisory agency. • Supervisory Plan set by supervising agency / interagency process 5

Examination of IT Auditor or Examiner: What’s the difference? Our perspective. Auditor: “Internal” perspective: • “Risk to the Institution” View • Strives to maintain independence from control processes and management influence • Focus on financial, operational & technical controls • Provides “Opinion” on Compliance with Regulations & Guidances • Reluctant to assess management capacity & effectiveness • Does not assess Board / Audit Committee 6

Examination of IT Auditor or Examiner: What’s the difference? Our perspective. Examiner: “External” perspective • “Risk to the Banking Industry” View • Independent of institutional processes & management structure • Focus on Risk Management Programs, including management’s ability to: – Identify, measure and monitor risk – Apply appropriate controls to mitigate risk – Adjust programs to changing Risk Profiles • Emphasis on oversight control and governance • Determines Regulatory Compliance • Evaluates Management (capacity, effectiveness, commitment) • Assesses the Audit Function • Determines whether Board / Audit Committee responsibilities are being met 7

Examination of IT Auditor or Examiner: What’s the difference? Our Approach. Auditor: Performs Continuous Audit Processes for timely measurement of key risks/exposures • Threshold Alerts (Transaction/Aggregation Limits; Volume/Capacity) • Review of MIS (e. g. Business Line Metrics, Activity/Volume Reports, Exception Reports, Change Reports, Event Logs/Alerts, etc. ) • Maintains ongoing dialog with management (e. g. , Performance; Strategic & Organizational Changes; etc. ) • Targeted Key Controls Validation • Monitoring of New Initiatives & Systems Development Projects Performs Audit Review of High-risk Projects 8

Examination of IT Auditor or Examiner: What’s the difference? Our Approach. Auditor: Performs “Traditional” Audits of Business & IT Functions • Reviews Processes for Effectiveness & Efficiency • Assesses adequacy of defined policies, procedures & standards • Conducts Validation Testing to determine adherence to policies, procedures and standards • May deploy Integrated Audit Software for independent control validation or data extraction on production systems May perform Horizontal, Vertical and Integrated (Financial/Operational/IT) Audits for comprehensive view Performs Technical Audits (e. g. , Application, Operating & Database Systems; Utility Software; Network & Telecom Components; etc. ) 9

Examination of IT Auditor or Examiner: What’s the difference? Our Approach. Examiner: Performs Continuous Supervision Processes for timely measurement of key risks/exposures • Monitors changes in institution risk profile: – Review MIS (e. g. , Performance/Capacity Reports; Volume/Trend Reports; etc. ) – Review Management Summary Reports (e. g. , Board/AC Packages; Steering Committee Reports; Project Status Reports; Incident Reports; etc. ) – Review Assessment Summaries (e. g. , Audit Reports; ERM Reports; Third-party Test Summaries; DR Test Summaries; Regulatory Analysis; etc. ) – Maintains ongoing dialog with management (e. g. , Performance; Strategic & Organizational Changes; etc. ) • Conducts “Target” Exams (Large Bank Supervision) – Risk Management Programs (e. g. , Information Security Program, Business Continuity Planning, etc. ) – Business Lines/Functions (e. g. , Consumer Finance; Wire Transfer; etc. ) Conducts Exams to Support Ratings Assignment • Leverage Continuous Monitoring and “Target” Exams • Uniform Rating System for IT (URSIT) Components: Management, Audit, Development & Acquisition, Support & Delivery 10

Examination of IT Auditor or Examiner: What’s the difference? Our Approach. Examiner: Looks to leverage various monitoring, validation and assessment activities designed to ensure the reliability of controls • • • Audit Reports Penetration Tests Network & Web Application Vulnerability Scans Systrust/Webtrust Reviews Business Continuity / Disaster Recovery Test Summaries System Patch Level & Anti-virus Maintenance Scans May conduct validation testing on a risk-basis • Most agencies do not deploy independent validation/data extraction software on bank systems 11

Examination of IT Auditor or Examiner: What’s the difference? How we enact change. Auditor: Submits Audit Report to Management • Provides Conclusions on Audit Scope Areas • Communicates Findings & Recommendations to Management – “Severity” ratings often applied to issues • Seeks Management “Buy-in” to Recommended Action • Requests Management Response & Evaluates Action Plan • Assigns an Audit Control Rating (Auditor “Opinion” on Effectiveness/Reliability of Controls) Tracks & Reports Status of Open Issues • Provides Summary to Audit Committee • Escalation Channel: Executive Management, Audit Committee 12

Examination of IT Auditor or Examiner: What’s the difference? How we enact change. Examiner: Issues Report of Examination to Executive Management / Board • • • Provides Conclusions on Exam Scope Areas Communicates “Required Action” and “Recommendations” Requires Response & Evaluates Management Action Plan Assigns an Examination Rating (URSIT Rating for IT) Risk-based approach to Issue / Action Plan Tracking Escalation Channel: Additional Enforcement Actions – Board Resolution, Civil Money Penalties, Cease & Desist, Removal – Impact on Applications Process (Mergers & Acquisitions, New Ventures) 13

Examination of IT Audit Examination Review Points: IT Audit 14

Examination of IT Audit Examination Review Points: IT Audit Examination objectives & procedures are identified in the FFIEC IT Examination Handbook: Audit Available at the FFIEC Website (www. ffiec. gov) • http: //www. ffiec. gov/ffiecinfobase/html_pages/it_01. html Other FFIEC IT Examination Handbooks Available: • • • Business Continuity Planning Development & Acquisition E-banking Fedline Information Security Management Operations Outsourcing Technology Services Retail Payment Systems Wholesale Payment Systems Supervision of Technology Service Providers FFIEC Handbooks are a Guide to Examiners 15

Examination of IT Audit Examination Review Points: IT Audit Independence (FFIEC Handbook – Tier 1 Objective 5: Determine the level of Audit Independence) Audit Charter: Establishes the authority and mission of the Audit Function Defined by the Board Precludes Conflict of Interest Duties Authorizes full access to information, records and systems Auditors (Internal & External) report directly to the Board or Board-level Audit Committee Approval of the Audit Plan; Changes to the Audit Plan Approval of “Out-of-Scope” Management Requests Presentation of Audit Reports Auditor has the ability to escalate issues to the Board Through normal Audit Committee process (Audit Committee Executive Session) Through Direct Contact with AC Chairperson and Outside Directors 16

Examination of IT Audit Examination Review Points: IT Audit Independence Administrative Reporting – “Degree of Control” management has on: What is reported to the Board What is reviewed by Audit Approval of Audit Staffing & Contract Requests Department Compensation Levels • General Auditor compensation reviewed by Board/Audit Committee • Comparative data analysis by Audit Committee Performance Appraisals & Measurement Criteria • Based on Job Descriptions, Audit Charter and Audit Committee Directives Auditors are not responsible for ongoing control processes Audit Ratings are assigned based on a defined structure, and are not “Negotiated” with Management 17

Examination of IT Audit Examiner Review Points: IT Audit Board & Audit Committee Oversight (FFIEC Handbook Tier 1 Objective 2: Determine the quality of the oversight and support of the IT audit function provided by the Board and Senior Management. ) Board / Audit Committee: Defines the Authority & Mission of the Audit Function (Audit Charter) Reviews and Approves the Audit Plan • Ensures the Audit Plan provides proper risk-based coverage of the “Universe” of Audits – Does the Audit Committee know what is not being Audited? • Ensures performance of the Audit Plan & Schedule • Approves Major Deviations from the Plan Maintains proper awareness of audit conclusions, significant findings, and management progress on significant issues. • Reviews Audit Reports; Control Ratings Updates ; Issue Status Reports • Ensures an appropriate level of Committee reporting • Discussion is reflected in Board / Committee packages and meeting minutes 18

Examination of IT Audit Examiner Review Points: IT Audit Board & Audit Committee Oversight Board / Audit Committee: Approves the scope of engagement of External and Outsourced IT Auditors. • Ensures the audit resource are independent and qualified • Ensures the scope of review is adequate to support comprehensive assessment, including reviews of complex programs (e. g. , Information Security, Vendor Management, etc. ) 19

Examination of IT Audit Examination Review Points: IT Audit Staffing (FFIEC Handbook – Tier 1 Objective 4: Determine the qualifications of the IT Audit staff and its continued development through training and continuous education. ) IT Audit staff is adequate in number and is technically competent to accomplish its mission Staff level adequately supports the Audit Plan or adequate resources are secured through contract support Staff is qualified to perform duties • Education, Experience & Certifications • Qualifications vs. Job Descriptions • Staff is qualified in the Technologies used Specific expertise is secured where needed. Training program ensures ongoing technical competence, consistent with technologies in use/planned. Adequacy of Current Training Budget Record of Past Training 20

Examination of IT Audit Examiner Review Points: IT Audit Policies, Standards & Procedures Formal and comprehensive Policies, Standards and Procedures are established to guide IT Audit activities and ensure consistency Address who, what, where, when and how IT Audit activities will be conducted. Address all key Audit activities Controls are established to ensure adherence with policies, standards & procedures (e. g. , Review & Approval processes; Quality Assurance reviews; etc. ) Reliable processes are established to update Audit policies, standards and procedures. The expectation for well defined policies, standards and procedures applies to each of the following discussion topics. 21

Examination of IT Audit Examiner Review Points: IT Audit Defining the IT Audit “Universe” (FFIEC Handbook – Tier 1 Objective 8, Step #1: Determine if the audit universe is well defined. ) IT Audit “Universe” is Properly Defined Addresses the inventory of applications & platforms in use (e. g. , Applications; Operating Systems; RDMS; Utility Software; Network; Telecom; Hardware; Physical Locations; etc. ) Addresses the inventory of IT operations, functions & services (e. g. , Information Security; Network Security; BCP/DR; Project Management; Development & Change Management; Vendor Management; Production Control; System Operations; etc. ) Addresses outliers to central IT functions Reliable processes exist to identify & account for changes in the institution’s risk profile that may affect the Audit function (e. g. , changes in technologies & processes; new products; organizational changes; etc. ). Examiners may compare the defined IT Audit “Universe” with Systems Inventory (BCP); Network Diagrams; Organizational Charts; etc. 22

Examination of IT Audit Examiner Review Points: IT Audit Risk Ratings (FFIEC Handbook – Tier 1 Objective 8: Determine the adequacy of Audit’s risk analysis methodology in prioritizing the allocation of audit resources and formulating the IT Audit Schedule. ) Audit Risk Ratings: Analysis includes all appropriate risk factors. Strategic; Financial Impact; Operational; Transaction; Technology; Reputation; Legal/Regulatory; etc. ) (e. g. , Assigned ratings are appropriate & supported. • How do the assigned audit risk ratings compare with Business Impact Analysis (BCP) and Information Security risk rankings? Applied to the “Universe” of Audits Reliable processes exist to ensure audit risk ratings are consistently applied. (e. g. , review & approval process; Quality Assurance review; etc. ) Reliable processes exist to ensure that significant changes are identified and addressed to ensure continued reliability of the risk ratings. 23

Examination of IT Audit Examiner Review Points: IT Audit Plans (FFIEC Handbook – Tier 1 Objective 7: Determine the adequacy of the overall audit plan in providing appropriate coverage of IT risks. ) Audit is a process, not a single event. IT Audit activities are viewed as a whole when assessing the adequacy of coverage. • • Continuous Monitoring / Controls Validation Traditional Audits Integrated Audits Project Audits Audit Plan: Ensures proper risk-based coverage of the Audit “Universe” Provides for appropriate frequency of review for High and Medium risks. Does not exclude Low risk areas Meets appropriate (defined) standards for frequency of review Audit Delineation and Budgeted Hours support comprehensive review Is regularly met (without routine scope reductions) 24

Examination of IT Audit Examiner Review Points: IT Audit Reports (FFIEC Handbook – Tier 1 Objective 9: Determine the adequacy of the scope, frequency, and timeliness of IT-related audit reports. ) Auditors accurately identify and consistently report weaknesses and risks. Reports provide timely communication of issues: • Audit review period vs. report date • Critical issues are discussed at time of discovery Report Distribution: Addressed to an appropriate level of authority to affect corrective action. Distribution to other affected parties/stakeholders Audit Scope: Properly defined to provide clear understanding of coverage Scope limitations and significant “Out-of-scope” items are identified Appropriate to support conclusions • Core functions are included in scope of review • Compare with audit risk assessment 25

Examination of IT Audit Examiner Review Points: IT Audit Reports Audit Findings: Accurate, complete and clearly defined Properly identify Root Cause Repeat issues are identified Common issues are recognized across audits, and are linked for management attention. Issues are properly categorized based on associated risks and compensating controls. Audit Workpapers properly support decisions for issues not reported. Audit Recommendations: Properly address Root Cause Appropriate to prevent recurrence of issues 26

Examination of IT Audit Examiner Review Points: IT Audit Reports Management Action Plan: Appropriate to resolve issues in a timely manner and minimize likelihood of recurrence; or Auditor’s Note is provided to identify concerns with management’s action plan Audit Rating: Consistent with volume & severity of identified issues Consistent with stated conclusions. 27

Examination of IT Audit FFIEC Guidelines: IT Audit Reports FFIEC Basic Audit Report Guidelines – Report Should… Provide Scope and Objectives of the Audit Summarize all significant observations Provide written notification to senior management and Board Highlight exceptions, potential risk exposure and recommendations for remedial action State an overall opinion of the function, improvement or decline since last audited, reasons for changes Ensure timely written responses Satisfy audit objectives Ensure conclusions are appropriate for work performed Exercise sound judgment in separating significant/insignificant findings FFIEC Additional Audit Report Considerations – Report should… Provide an opinion on the adequacy of stated action plans Provide a definition of Control Ratings Provide a definition of Risk Ratings When appropriate, establish a timeline for follow-up audit 28

Examination of IT Audit Examiner Review Points: IT Audit Issues Tracking (FFIEC Handbook – Tier 1 Objective 6: Determine the existence of timely and formal follow-up and reporting on management’s resolution of identified IT problems or weaknesses. ) Reliability of the Issue Tracking Process Effectiveness in Securing Timely & Appropriate Corrective Action Extent and Age of Open Audit Issues; Repeat Issues Adequacy of Management Action (Addresses Root Cause; Timely Action) Prior Issues History Adequacy of Issue Escalation & Board Reporting Processes Process for Issue Closure: Audit approval/agreement required prior to Issue Closure Audit reviews/validates corrective action prior to closure (High Risk Issues) 29

Examination of IT Audit Examiner Review Points: IT Auditing Systems Projects (FFIEC Handbook – Tier 1 Objective 10: Determine the extent of Audit’s participation in application development, acquisition, and testing, as part of the organization’s process to ensure the effectiveness of internal controls. ) Policies regarding Audit participation in SDLC projects are clearly defined. Reliable processes are established to identify new projects. An appropriate Project Risk Rating methodology is established and consistently applied. Auditor is not simply repeating Project Management Office statements (“On-time” / “Within Budget”) 30

Examination of IT Audit Examiner Review Points: IT Auditing Systems Projects Appropriate risk-based audit coverage is provided for high-risk projects, including sufficient validation to conclude on the adequacy of project controls and testing activities. Validation that Project Methodology is followed Project status is accurately reported Appropriate testing is being performed • • Systems Integration Testing User Acceptance Testing Data Conversion Testing Sample Test Plans / Test Scripts Issues are followed-up / escalated The Audit Committee is properly informed of Project Audit activity, including high-risk projects that Audit is not reviewing. 31

Examination of IT Audit Examiner Review Points: IT Audit Process Audits and Technical Controls Validation Auditing of IT processes is expected; however, process auditing alone is not enough. An appropriate level of audit review and testing must be performed to validate the reliability of technical controls. The level of audit review and testing should be commensurate with the complexity and risk profile. Leveraging alternative control validation processes is acceptable (e. g. , third-party audits, vulnerability scans, penetration tests, etc. ). However, the Auditor must: assess the reliability of the alternative control validation; and ensure that the scope is sufficiently comprehensive to support Audit objectives and conclusions. 32

Examination of IT Audit Gaps in Technical Controls Validation – One Example Loan Application Audit included limited review of application security. • User Access Rights • Password Configuration The Auditor performed no validation of critical calculations. Auditor placed reliance upon a strong Change Management Process (recently audited). Process includes: Comprehensive Testing of System Calculations Quality Assurance Review & Approval for production system changes Restricted access to Production code, data, and interest rate tables. So what’s the problem? 33

Examination of IT Audit Gaps in Technical Controls Validation – One Example Loan Application Audit The application was executing from a Test Environment. Interest rate calculations were incorrect. The Auditor performed no validation to ensure that the Application was running from Production. (e. g. , review of Job Logs; review for Test Library References in Production Link List or JCL) 34

Examination of IT Audit Gaps in IT Audit Coverage – Other Examples Application Audits: Scope defined to include Application Security; however, Audit validation limited to user access provisioning. No review of security over application code and data. Network Security Audits: Reliance on Network Diagram (Visio Diagram) provided by Network Administrator without efforts to validate • Network Mapping Tools • Network Addressing • Review with Network Administrators Firewall Auditor received no training in FW type; could not interpret FW rule set. Failed to identify “Holes” in FW (Stateful Inspection FW allowing UDP traffic. ) 35

Examination of IT Audit Questions? ? ? Thank You! 36