Conformance IOP Certification Issues Strategies ENTSOe IOP July

Conformance & IOP Certification: Issues & Strategies ENTSO-e IOP July 9, 2013

Introduction • • ENTSO-e and EU Options UCA Testing Committee Terms NIST-UCA Pilot Test Program Proposal for ENTSO-e/EU-UCA Pilot Test Program Test Certification Issues Observations and Testing Assumptions Alternative to Testing Certification

ENTSO-e and EU Options • If Certification is desired – Need: – Define Program and how it will work – Define Entity to Manage the Program • Test Manager Entity must: – Taylor program to needs of ENTSO-e and TSOs in Europe – Already have experience in Testing Program development and implementation – Be able to provide assistance in the Europe Implementation of a Testing Program – Provide Tester Accreditation • Test Manager Administrative Tasks: – Maintain Conformance Certificates – Maintain testing documents – Maintain a list of Test Labs (TL) and provide Audit of TLs as needed

Testing Introduction • Goal of Interoperability testing to date validation/improvement of ENTSO-E profile and underlying CIM standard • Goal of Certification testing (conformance and/or interoperability) formal recognition of product functionality

UCA Testing Committee • Testing Committee created during UCAIug creation • Began work in 2005 • Technical Tasks – Test Procedures – Quality Assurance Program – Tester Accreditation • Administrative Tasks – Maintain Conformance Certificates – Maintain testing documents

Testing Subcommittee Charter • Composed of Chairs of each UG Testing Subcommittee • Single point focus for UCAIug • Arbitrates technical issues • Coordinates Test Subcommittee groups within UCA – 61850 Conformance and Interoperability – Open Smart Grid – Green Button ITCA – Conformance and Interoperability – CIM Conformance and Interoperability

Terms • • • ITCA – Interoperability Testing and Certification Authority IPRM – Interoperability Process Reference Manual SGTCC – Smart Grid Testing and Certification Committee SGIP – Smart Grid Interoperability Panel CB – Certification Body AB – Accreditation Body TL – Test Lab ILAC - International Laboratory Accreditation Cooperation IAF – International Accreditation Forum SDO – Standards Development Organization (IEC, ISO, etc) NIST – National Institute of Standards and Technology

SGIP-NIST IPRM – Tailored Program for NIST • Fall 2011 SGIP Meeting included an Session to discuss ITCA plans • SGIP-SGTCCWG 8 – Responsible for IPRM generation – Responsible for ITCA Set-up and Oversight – Defines the Rules for ITCA creation • What are processes and procedures for ITCAs • What is required by the Accreditation Bodies (AB) • What portions of ISO 67 must be implemented – CBs and TLs must implement ISO 65 and ISO 17025

UCA Pilot of NIST Test Program • UCA volunteered to pilot the Program • UCA has formal quality management system • UCA will issue (sign off on) certificates – Testers to become 17025 certified/compliant • Tester will not have final say on certificate issue • UCA will review the “tester files” before cert issued • Creating Test Scheme for ITCA – Team working on Procedures – EPRI started project to develop Test Procedures • ITCA Characteristics – Provides Certification Program Requirements [CPRs] used in administrating the Program – ITCA is the Test Scheme Owner

ENTSO-E & EU Testing Program

UCA Pilot for ENTSO-e and EU • UCA has volunteered to assist in development of the program • Need to identify program requirements • Need to identify Accreditation Bodies, Certification Bodies and Test Labs to execute the program (UCA would do this as part of the program implementation) • Complete discussions surrounding all Certification issues defined on the following slide.

Test Certification Issues • Need to identify what testing strategy is required for ENTSOe & Europe. – Conformance Testing (which exchanges? , how to test? ) – Interoperability Testing (which exchanges? , how to test? ) • Need to define what type of changes in the applications trigger the need for each type of certification (when? ) • Who Develops Test Cases and Procedures? • Who Develops the “Gold” Profiles, Instance Files, and other input files required for testing. Need files for both conformance and IOP.

Some Observations and Testing Assumptions • There should be NO code modifications to submitted applications during the testing. This is only allowed if the vendor has made arrangement for pre-testing with the test lab. • Any pre-test is to occur prior to submitting the product for conformance testing. • All Profiling tools used to create or generate profiles to be used in the testing must be certified. These tools must conform to the Guidelines defined in the new Profiling standard (62361 -101). In addition, any XSDs produced by this tool must conform to the XML-XSD Mapping standard (62361 -100). This profile tool testing will require some fully verified input and output files to compare against. Both positive and negative testing should be performed.

Some Observations and Testing Assumptions (Cont’d. ) • Validation tools must be tested for conformance to a “gold” model and a “gold” profile/schema. • A validation tool must be able to make two determinations at a minimum: – Verify the input/output files are valid CIM syntax (valid class/attribute/association, multiplicity, etc. ) Should be able to determine version of CIM and if all data conforms to the CIM standard – Extensions to the CIM must be accommodated. – Verify the data conforms to a given profile. – Verify the instance file is valid XML, RDF, XSD, etc. • All submitted applications must pass the conformance testing prior to entering an Inter-operability test. This would be true writher it is a tool, data manager, power flow or any other program.

Some Observations and Testing Assumptions (Cont’d. ) • All testing performed must use a single set of tools for conformance and inter-operability testing. The tool set can be changed but the tests would have to be executed again. • The test lab might use more than one ‘golden’ model during testing. Not all models need to be made available to the vendors prior to the official testing. • Models should include valid CIM objects that might not be included in the profile being tested. Other profiles might need this data. However, the application need only retain and handle the data that is part of the Profile under test; that is, the application only needs to export what is defined in the profile under test.

Some Observations and Testing Assumptions (Cont’d. ) • The entity that submits the application for test must identify the certification desired at the time of submission. • It is up to the lab to perform all requested testing. The lab must provide a list of documentation and other data that must be provided by the vendor. If this set of information is not provided, the lab will not execute the test and all applications and data will be returned to the vendor – untested. The vendor will not be executing the tests but may witness the testing activity along with any independent witness if desired. • A vendor could pass the conformance testing portion and still fail the IOP. Separate certificates will be issued for each type of test (conformance and IOP) and for each profile that is tested.

Alternative to Testing Certification If ENTSO-e or Europe does not participate in a formal Testing Certification process – we will need to define written rules and requirements to ensure files sent to ENTSO-e are valid and we will need tools to complete the validation of these files. There are many issues that will need to be addressed to ensure all vendors and TSOs have all the information and coordination required.

Responsibilities and Relationships Interoperability Testing and Certification Authority ITCA Standards Development Organization SDO Manage Test Lab accreditation/audit Maintain Conformance Certificates Maintain Testing Documents Accreditation Body AB Accredit Test Lab Develop standards Assist in Test Procedure creation Assist in Test Environment creation Test Lab TL Execute test according to Test Procedure Maintain Test Environment Certification Body CB

Who is who and could do what ILAC International Laboratory Accreditation Cooperation IAF – International Accreditation Forum SGTCC – Smart Grid Testing and Certification Committee AB – Accreditation Body ITCA – Interoperabili ty Testing and Certification Authority IPRM – Interoperabili ty Process Reference Manual SDO – Standards Developmen t Organization (IEC, ISO, etc) CB – Certification Body UCAI or TL – Test Lab SGIP – Smart Grid Interoperabili ty Panel NIST – National Institute of Standards and Technology ENTSO-E or TSO - user Vendor to be certified TSO – as vendor – developing CIM tool