Continuous audit today and tomorrow Miklos A Vasarhelyi

Continuous audit: today and tomorrow Miklos A. Vasarhelyi KPMG Professor – Rutgers University Senior Consultant- AT&T Laboratories

Continuous Audit and Reporting Laboratory Outline • An evolving framework • Some Key issues / the state of the art • Some CARLAB experiences • Six Steps in Implementing CA • Organizational Context • Opportunities and Challenges • Conclusions 2

An evolving audit framework

Continuous Audit and Reporting Laboratory An evolving audit framework Data level Process level Assurance of Data elements • XML/ XBRL datum • Generated and modified by different processes • Balkanization of data • Control / Assurance tags Assurance of Key Processes • Process reviews a la Systrust • Internal or outsourced • Third party processes are to become the norm • Intra and Inter process controls an issue Report level Assurance of Reports • Compliance reports becoming commonplace • Traditional audit is an instance of RLA • Generated and modified by different processes 4

Continuous Audit and Reporting Laboratory An evolving continuous audit framework • Automation • Sensoring • ERP Continuous Data Audit Continuous Control Monitoring • E-Commerce Continuous CA = CCM+ C(D)A CA -> Continuous Audit CCM -> Continuous Control Monitoring C(D)A -> Continuous Data Assurance Audit 5

Continuous Audit and Reporting Laboratory Some Key Issues • Two recent surveys (ACL and PWC) show that a large number of key companies are attempting to perform continuous audit like functions • An industry of software is evolving with ACL, IDEA, APPROVA, and others growing rapidly • Control Monitoring and Continuous Data Assurance are the main approaches • The first recorded application was AT&T Bell Laboratories CPAS effort in the 1986 -1991 period • The Rutgers Car. Lab is working in leading applications 6

Continuous Audit and Reporting Laboratory • Continuous Auditing Value Proposition – Improved business performance • Innovations in information technology & analytical modelling enable: – More frequent, timely, accurate & relevant business performance information – Lower compliance risk – Cost reduction 7

Continuous Audit and Reporting Laboratory CAR-Lab Experiences • Control monitoring at Siemens • Transaction monitoring at Unibanco • Continuous (data) assurance at HCA • Other – Conceptual developments – Simulating Liberty – EBR work – KPMG projects 8

Overview of Ca. R-Lab examples

Continuous Audit and Reporting Laboratory Siemens' – Project Value Proposition Expanded Audit Coverage Significant Cost Savings Automated Business Process Controls Monitoring Project 10

Continuous Audit and Reporting Laboratory Siemens' – Project Features • Formalize & automate internal audit procedures used for business process controls monitoring • Conduct “man vs. model” assessments • Calibrate “exception rules” to optimize model performance • Scale up to all SAP instances • Increase frequency of model application, where feasible • Transition to Approva application and extend the model where optimal 11

Continuous Audit and Reporting Laboratory A 3 pronged approach to audit automation • Automate audit plan using delivered Rule Sets: Est 25% of a typical manual audit plan • Automate using external data sets (Static & Variable): Est an additional 25% a typical manual audit plan • Re-enginer manual controls into automated controls with improved control precision: Est an additional 25% a typical manual audit plan • Total = Automation Opportunity ~75%!! 12

Continuous Audit and Reporting Laboratory MCP Management O pe ra tin g Auditor s w o l F Al A. A. S (audit Action Items) From Siemens Approva and other literature Master larm A Audit ting a r e Audit Program Op Parameterization Audit ar m Evidence Fl Receptacle ow s Tool CA Control Dashboard Other Static Parameters Inference Engine Evergreen Opinion Deterministic Data Extraction Stochastic Remote Audit Communic. Tool External Snapshot Table comparisons Interactive Mail Management Tool Sustainable Object Verification Tool Class of Other Auditable Actions ---- Other of Audit Processes 13

Continuous Audit and Reporting Laboratory IT / IA Continuous Auditing Program at Unibanco 14

Continuous Audit and Reporting Laboratory Unibanco – Some CA Program Features • Automated monitoring of over 5 million customer accounts on a daily basis using 25 automated procedures to: – – Detect errors Deter inappropriate events & behaviors Reduce or avoid financial losses Help assure compliance with existing laws, policies, norms and procedures • Examples of “low hanging fruit: ” – – – Customer advances Excess over credit limit Returned checks Federal tax payment cancellations TED emissions (should this be omissions? ) 15

Continuous Audit and Reporting Laboratory Unibanco – Advances to Clients Monitoring 16

Continuous Audit and Reporting Laboratory Continuous Data Assurance (CDA) at a Major Health Services Provides (HSP) • HSP is a large national provider of healthcare services, composed of locally managed facilities that include numerous hospitals and outpatient surgery centers. • IT internal audit provided access to unfiltered extracts from their transactional databases, comprising all procurement cycle daily transactions from October 1 st, 2003 through June 30 th, 2004: Over 500, 000 data points. • Dataset mimics what a CDA system has to deal with: highly disaggregate data flowing through CA system in real time. • Audit procedures have to be developed for this environment. 17

Continuous Audit and Reporting Laboratory Analytical Procedures in CA • • • Analytical procedures used in the planning, substantive testing, and reviewing stages of an audit. We focus on substantive testing. In conventional auditing first apply analytical procedures to identify potential problems, Then, focus detailed transaction testing on the identified problem areas. In CDA the sequence is reversed: 1. Use automated general transaction tests to all the transactions and filter out identified exceptions for resolution. 2. Apply automated analytical procedures to the filtered transaction stream to identify unforeseen problems. 3. Alarm humans to investigate anomalies. 18

Continuous Audit and Reporting Laboratory Continuous Data Assurance • Automation of Transaction Testing: – Formalization of business process rules as transaction integrity and validity constraints. – Verification of transaction integrity and validity detection of exceptions generation of alarms. • Automation of Analytical Procedures: – Selection of critical business process metrics and development of stable business flow (continuity) equations. – Monitoring of continuity equation residuals detection of anomalies generation of alarms. 19

Audit and Reporting Laboratory Continuous Data. Continuous Assurance System Automatic Analytical Monitoring: Continuity Equations Anomaly Alarms Automatic Transaction Verification Exception Alarms Responsible Enterprise Personnel Business Data Warehouse Enterprise System Landscape Sales Accounts Receivable Materials Management Human Resources Ordering Accounts Payable 20

Continuous Audit and Reporting Laboratory Establishing Data Integrity: A Procurement Example • Referential integrity along the business cycle and identification of completed cycles: P. O. Shipment receipt voucher payment. • Identification of data consistency issues and automatic alarms to resolve exceptions: – Changes in purchase order vendor numbers; – Discrepancies between the totals and the sums of line items; – Discrepancies between matched voucher amounts. 21

Continuous Audit and Reporting Laboratory Detection of Exceptions • Referential integrity violations – PO without matching requisition – Received item without matching PO – Payments without matching received items • Data integrity violations – PO has zero order quantity – Received item has negative quantity – Invalid payment check numbers (e. g. All 0 s) – Gross payment amount is smaller than net payment amount 22

Continuous Audit and Reporting Laboratory Continuity Equation Based CDA • Continuity Equations: – Stable probabilistic models of highly disaggregated business processes, uses as the expectation models for process based analytical procedures. – Originated in physical sciences (various conservation laws: e. g. mass, momentum, charge). • Continuity equations are developed using statistical methodologies of: 1. Linear regression modeling (LRM); 2. Simultaneous equation modeling (SEM); 3. Multivariate time series modeling (MTSM) using various Vector Autoregressive Models (VAR). 23

Continuous Audit and Reporting Laboratory Basic Procurement Cycle t 2 -t 1 P. O. (t 1) t 3 -t 2 Receive(t 2) Voucher(t 3) 24

Continuous Audit and Reporting Laboratory Ideal Continuity Equations of Basic Procurement Cycle Receive(t 2)= P. O. (t 1) Voucher(t 3)= Receive(t 2) • Aren’t partial deliveries allowed? • Are all orders delivered after exactly the same time lag? • Are there any feedback loops? 25

Continuous Audit and Reporting Laboratory Estimated Continuity Equations of Procurement Using VAR Model P. O. (t)= 0. 24*P. O. (t-4) + 0. 25*P. O. (t-14)+ 0. 56*Receive(t-15) + εPO Receive(t)= 0. 26*P. O. (t-4) + 0. 21*P. O. (t-6)+ 0. 60*Voucher(t-10) + εR Voucher(t)=0. 54*Receive(t-1) - 0. 17*P. O. (t-9) + 0. 22*P. O. (t-17) + 0. 24*Receive(t-17) + εV 26

Continuous Audit and Reporting Laboratory Detection of Anomalies • Anomalies are detected if: – Observed P. O. (t) < Predicted P. O. (t) - Var or – Observed P. O. (t) > Predicted P. O. (t) + Var • Similarly for: – Receive(t) – Voucher(t) • Var = acceptable threshold of variance. • If there is anomaly generate alarm! 27

Continuous Audit and Reporting Laboratory Measuring Anomaly Detection • False positive error (false alarm, Type I error): A nonanomaly mistakenly detected by the model as an anomaly. Decreases efficiency. • False negative error (Type II error): An anomaly failed to be detected by the model. Decreases effectiveness. • Detection rate is used for clear presentation purpose: The rate of successful detection of seeded errors. • A good analytical model is expected to have good anomaly detection capability: low false negative error rate (i. e. high detection rate) and low false positive error rate. 28

Continuous Audit and Reporting Laboratory Simulated Error Correction • Access to highly disaggregate data in real time makes it possible for CA system to detect, investigate and correct anomalies also in (nearly) real-time. • Real-time error correction enables utilizing the corrected rather than the erroneous data in revised continuity equation benchmarks. • Real-time error correction is likely to benefit future anomaly detection. We investigate the magnitude of this benefit using simulation. • Error correction raises important issues about auditor independence, and the line between auditing and monitoring of business processes. 29

Continuous Audit and Reporting Laboratory Benefit of Real-time Error Correction: MTSM 30

Continuous Audit and Reporting Laboratory Takeaways from HSP Study • Various statistical methods can be used to derive expectation models of acceptable quality. • But key is access to highly disaggregate data, not which benchmark is used. With such data, most reasonable continuity equation models give usable results. • Real-time error correction significantly improves error detection. • More disaggregated models are not always better: weekly data can be more stable than the daily one. • Alarms have to be managed – trade-off between Type I and Type II errors. 32

Implementation Issues in CA 10/27/2020

Continuous Audit and Reporting Laboratory • Background – While technologies of continuous audit have been extensively discussed and are progressively emerging the more mundane issues of their implementation in a socio-technical environment have been neglected – http: //www. theiia. org/itaudit/features/in-depthfeatures-2 -10 -08/feature-2/ 34

Continuous Audit and Reporting Laboratory 1. Priority 2. Areas 6. Action and Reaction 2. Rule Audit Control Panel 5. Follow-up 3. Frequency 4. Parameterization Six steps of process implementation 35

Continuous Audit and Reporting Laboratory – 1. Identification of Priority Areas • Modularize risk areas, rate these risks and evaluate the cost x benefits • Identify the basic audit objects • Choose critical business processes that will be the focus of continuous audit (low hanging fruit) • Identify key data in for the implementation of Continuous Audit in the mapped processes • Political Considerations 36

Continuous Audit and Reporting Laboratory • Key Objective of Audit Procedure – Detective – Deterrent – Financial – Compliance 37

Continuous Audit and Reporting Laboratory • 2. Rules of Monitoring and Auditing – Once an area of CA is chosen the “rules” of monitoring, alarming, and assurance must be established – These must take into consideration the legal and environmental issues as well as the objectives of the particular process – The CA process is established adopting certain rules, frequencies, and parameters. – e. g. we will monitor bank accounts in overdrafts or in excess limits 38

Continuous Audit and Reporting Laboratory • 3. Frequency – The natural rhythm of the process • Timing of computer processes • Timing of business processes – Cost benefit considerations – Nature of procedure objectives • Deterrence • Prevention 39

Continuous Audit and Reporting Laboratory – 4. Parameterization • Define parameter to analyze in accordance with the risk • eg. : Monitoring all accounts in overdrafts in daily basis , that have a balance of debt 20% larger than its limit of loan and bigger than 1000 USD 40

Continuous Audit and Reporting Laboratory • 5. Follow-up – Who will receive the alarm? • Management? • Audit leadership? • Immediate superior of the responsible for the data • The timing of the follow up – Pass the alarm along immediately – Reconcile the alarm prior to follow up – Wait for 3 sequential days of similar alarms to follow up • Escalation guidelines – E. g. after three days send to the immediate superior’s superior or wait for 3 days prior to the re-escalation 41

Continuous Audit and Reporting Laboratory • 6. Action and Reaction – Guidelines for dealing with auditees • Lack of bias • Consistency of response • Guidelines for individual factor considerations • Concern with collusion 42

Organizational Issues 10/27/2020

Continuous Audit and Reporting Laboratory • Organizational Structure for CA – – Is CA a part of the audit function or of management? Its part of the audit function Should there be a separate continuous audit group? Yes, to facilitate its implementation progressively in the many areas of continuous audit 44

Continuous Audit and Reporting Laboratory • Workforce Effects – Progressively labor requirements for the traditional audits supported by CA will reduce and deeper audit will become possible – Rebalancing of workforces – High technological competencies needed 45

Opportunities and Challenges 10/27/2020

Continuous Audit and Reporting Laboratory Opportunities for business and research (1) • Control system measurement – We are in a pre-paradigmatic stage of control documentation and measurement – We do not know how to monitor controls in large ERPs – We do not know how to provide a really supportable opinion on controls – We do not know how to rate combinations of controls 47

Continuous Audit and Reporting Laboratory Opportunities (2) • Business Process Monitoring and Alarming – Auditors have to carve a position on the new monitoring and control environment – Auditors can collect exception “alarms” as trusted parties and incorporate these into evidentiary matter – Auditors can be “trusted” 48

Continuous Audit and Reporting Laboratory Opportunities (3) • Automatic Confirmation Tools – Confirmations will have an increased evidentiary role with eventual elimination of population and integrity worries – Intelligent confirmatory tags can do much – Database to database hand-shaking will be medium – Business opportunity for auditors 49

Continuous Audit and Reporting Laboratory Opportunities (4) • Audit bots (agents) – Many of the basic audit functions can be emulated by software – These must be eventually developed by the profession to work hand-in-hand with human auditors in the new audit world – These agents will work on all areas including: 1) audit planning, 2) analytical reviews, 4) confirmations, and )5 evergreen opinions 50

Continuous Audit and Reporting Laboratory Opportunities (5) • Collecting forensic trails – Auditor “black” box • Publishing real-time authenticated reports for different compliance masters • Publishing FD independent compliance reports 51

Continuous Audit and Reporting Laboratory Challenges • Standards are needed for CA – Audit monitoring needs to be defined – Types of evidence are to change and must be reconsidered – Independence needs to be re-defined • The billing model has to be restructured to bill on function not hours 52

Continuous Audit and Reporting Laboratory Challenges • Audit firms must put improved knowledge collection and management processes to feed their audit analytic toolkit • Audit firms have to engage in auditor automation and pro-actively promote corporate data collection during-the-process • Value added must be justified in terms of data quality 53

Continuous Audit and Reporting Laboratory • Conclusions – Attention must be paid to the organizational processes that implement continuous audit – There are 6 key steps to progressively implement a CA program module by module – The CA process is dynamic and CA management will change schedule and parameters of each process – The organization of the audit process must be evolved progressively 54