Notice of Compliance Audit Phil ODonnell Manager Compliance

Notice of Compliance Audit Phil O’Donnell Manager, Compliance, Operations and Planning Audits and Investigations

2 Audit Frequency 3 Year Cycle Balancing Authority Transmission Operator Reliability Coordinator All other registered functions Subject to flexibility in the future as part of NERC’s Reliability Assurance Initiative

3 Compliance Audit (on-site vs. off-site) • On-Site – Documentation sent to WECC before audit for preliminary review – The audit team reviews evidence during off-site week or the first week of the audit and completes its review during the second week or on-site week – Data Requests or DRs – Tours to observe facilities – In-person interviews for clarification

4 Compliance Audit (on-site vs. off-site) • Off-Site – Documentation sent to WECC before audit for preliminary review – Data Requests or DRs – Entity may be present at audit if desired – Telephone interviews for clarification

5 Compliance Audit (on-site vs. off-site) • Primary difference is: – Location of audit conduct • Scope is typically smaller for off site • On Site – Required for RC, BA, TOP functions • Per NERC Rules of Procedure 403. 11. 2

6 Audit Timeline CIP v 5 Request for Information 145 days Notice of Audit 90 days Pre-Audit Survey Due Evidence Due 60 days 30 days Objections to Team Members 15 days AUDIT

7 Notice of Audit Packet Notice of Audit Letter ATT D: Audit Scope and WECC RSAWs ATT A: Compliance Monitoring Authority ATT E: Certification Letter ATT F: Pre-Audit Survey ATT B: Audit Team Biographies ATT G: Pre-Audit Data ATT C: Confidentiality Requests Agreements ATT H: Post Audit Feedback Form

8 Notice of Audit Letter 90 -Day Notice of Audit Letter Details of your specific audit • • Audit Engagement Dates Audit Period Registered Functions within Audit Scope Audit Team Composition Observers (if applicable) – May include FERC/NERC • Date/time of proposed Pre-Audit Conference Call • Links to reference documents

9 Attachments A, B and C Attachment A Explanation of Compliance Monitoring Authority Attachment B Short Biographies of the WECC Audit Staff Attachment C Signed Confidentiality Agreements of the WECC Audit Staff

10 Attachments D and E Attachment D Audit Scope Reliability Standard Audit Worksheets (RSAWs) Attachment E Certification Letter • Must be printed on your company letterhead and signed by an Authorized Officer

11 Attachment F Pre-Audit Survey • Verify Registered Functions • Audit Logistics • Signed by Authorized Officer • Please complete all applicable fields

12 Attachment G Pre-Audit Data Requests – Clarifications for Data Submittal • One Line Diagram • Delegation agreements (if applicable) • CCA and non-CCA lists • Public Key Encryption

13 Attachment H Audit Feedback • Sent with initial package Feedback is encouraged for all phases of audit!

14 Evidence Submittal WECC Enhanced File Transfer (EFT) https: //fileupload. wecc. biz Any questions regarding log in or user credentials please contact weccsupport@wecc. biz or call 1 -877 -937 -9722

15 Evidence Submittal File Folder COM-001 -1

16 Evidence Submittal Adobe Portfolios COM

Audit Approaches • We audit to the Requirements of the Standards. • General Approaches included in RSAW • RSAW may ask specific questions • Always includes the section: “Describe, in narrative form, how you meet compliance with this requirement. ”

18 Audit Approaches “Describe, in narrative form, how you meet compliance with this requirement. ” • Describe here how your company knows it is compliant with this requirement and how you know you have been compliant for the entire period of the audit • Your place to describe your internal controls • Your evidence should support your narrative

19 Audit Approaches • List the evidence provided in the RSAW • This road map is important • Compliance Assessment Approach in RSAW is used as a checklist • Data Request (DR) for gaps or samples • Document and records review are primary • Interviews and observations are usually for corroborating

20 Sufficient Audit Evidence Sufficiency of Evidence • The measure of the quantity of evidence • Quantity of evidence is dependent on the scope of the audit • Extra quantity does not make up for poor quality • Ensure you provide enough evidence to demonstrate compliance for the entire audit period

21 Sufficient Audit Evidence Sampling is used to limit the amount of detailed evidence provided • Normally used in conjunction with summary of a full set of data • Sampling used to assess details • Reduces the burden on the Audit Team but not really on the Entity • Audit Team must select the samples

22 Appropriate Audit Evidence Appropriateness The measure of the quality of evidence • Relevance • Validity • Reliability

23 Appropriate Audit Evidence Quality of Evidence • Good Internal Controls point to reliable evidence • Direct observation is more reliable than indirect observation • Examination of original documents is more reliable than examination of copies • Testimonial evidence from system experts is more reliable than from personnel with indirect or partial knowledge

24 Types of Evidence • Physical Evidence • Documentary Evidence • Testimonial Evidence Compliance Audits may use all three types but Documentary Evidence is by far the most frequent type of evidence assessed and relied on.

25 Testimonial Evidence • Attestations of Compliance or Statements of Compliance are generally not accepted as the only available evidence. • Attestations may be used to explain minor gaps in documentation or to state if no conditions occurred which are subject to a requirement. • Attestor must be knowledgeable and qualified.

26 Evidence for Procedural Documents The characteristics of a valid procedural or policy document include: – Document title – Definition or Purpose – Revision level – Effective dates – Authorizing signatures

27 Non Applicable Requirements Three instances are acceptable for use of term “Not Applicable” 1) Entity is not registered for the applicable function (only TOP responsible for TOP requirements) 2) Entity does not own, operate or maintain the equipment addressed by the requirement (UVLS, UFLS, SPS etc. ) 3) Entity does not use the program or process specified by the requirement (and is not required to… ATC, CBM, etc. )

28 Evidence for Tasks Performed • When the standard calls for a task to be performed it must be documented. – – – – Records Logs Reports Work Orders Phone recordings Transcripts of phone recordings Shift Schedules • Dates & Times are critical

29 Evidence of “Coordination” with other entities • Typical evidence provided initially is a single email. “…If you have any comments please contact ______” This alone is neither sufficient or appropriate to demonstrate coordination between two or more parties. • If emails or correspondence are used – Two way communications are needed • Better are: – Meeting Agendas – Meeting Minutes – Attendance Lists

30 Evidence of “Distribution” of information • Typical evidence provided initially is a single email with a large distribution list. “…please see attached” This alone is typically neither sufficient or appropriate to demonstrate distribution to others. • If emails or correspondence are used – Need clear identification of the personnel on the distribution list. • Even better is corroboration by receipt acknowledgement