USOAP CMA Workshop Module 5 Updates on CMA

USOAP CMA Workshop Module 5 Updates on CMA Online Framework and Electronic Filing of Differences (EFOD) system 18 June 2021 Page 1

Contents Ø Overview of CMA Online Framework Ø Functionalities of each component Ø Implementation of revised protocol questions Ø Update on EFOD Ø Finding of Differences Task Force (FDTF) 18 June 2021 Page 2

Overview of CMA Online Framework 18 June 2021 Page 3

CMA Online framework is…. q A suite of web-integrated applications and centralized database systems which includes: üCollection of safety-related information and documents from different sources üMonitoring and reporting of safety oversight activities by ICAO and Member States 18 June 2021 Page 4 4

Access through… q ICAO USOAP Restricted Website (www. icao. int/usoap) 18 June 2021 Page 5 5

Modules/Components 18 June 2021 Page 6 6

OLF Modules/Components Between State & ICAO Update on LEI & SSCs Conduct of onsite CMA activities Prioritize onsite activities By ICAO HQ and RO By ICAO HQ PQs MIR Onsite Audit Manager State Safety Risk Profile Offline documents for CCs and PQs F&Rs Collection of Safety Information Signing of USOAP CMA MOU From States From external stakeholders From ICAO HQ and RO NCMC Dashboard Access Control System MOU State Profile / SAAQ CC/EFOD SSC Planning & Scheduling CAPs Database for management of USOAP questions in each related documents module USOAP Library and Final Reports 18 June 2021 Database for collected information and States’ LEI Database for MIR, Findings, SSC, and CAP USOAP Live Reports Database for USOAP auditors and CMA activity schedule Page 7

States’ OLF functions q Manage OLF user accounts q Keep up-to-date SAAQ and CC/EFOD q Provide a CAP update and/or progress per corresponding protocol finding q Provide self-assessment of each “Not Satisfactory” PQ q Respond to MIRs 18 June 2021 Page 8 8

Functionalities of each component 18 June 2021 Page 9

State Dashboard q States (NCMCs) can view: 18 June 2021 Page 10 10

State Dashboard q States (NCMCs) can view: 18 June 2021 Page 11 11

Access Control q NCMCs ü Have a full control of user accounts for the State • Issuing additional user accounts • Deactivating invalid user accounts • Setting individual user rights/permissions q Individual users ü Manage user profile information ü Personalize password Note: If you are not an NCMC and want to have access to the online framework, please contact the NCMC(s) of your State 18 June 2021 Page 12 12

Access Control q User account ü composed of “ID or Email address” and password ü duplicate IDs or email addresses are not accepted q Access rights ü 3 types (limited, read-only or read/write) ü CC/EFOD : access rights can be granted per Annex ü PQ, CAP and F&R : access rights are permitted for all technical areas 18 June 2021 Page 13 13

State Aviation Activity Questionnaire (SAAQ) q States üComplete and update the State Profile and SAAQ • Most feedback received from users has been addressed • Feature to download and print completed SAAQ is under development 18 June 2021 Page 14 14

Compliance Checklist / Electronic Filing of Differences (CC/EFOD) q States ü Indicate compliance with and/or difference to SARPs with details ü This complies with both the Convention Art. 38 and CMA MOU q ICAO ü Reviews/monitors the level of global compliance/ difference ü Generate e-Supplement 18 June 2021 Page 15 15

Compliance Checklist / Electronic Filing of Differences (CC/EFOD) submit/update CCs Conduct a preliminary data entry Online/Offline Save a draft Online NOT shared with other States Work is in progress 18 June 2021 EFOD Submit/Update CCs Per Annex To be shared with other States Choose Standards only or All SARPs Per Annex Submit official notification to ICAO To be used for generation of Supplement ICAO to monitor and share information on implementation of SARPs Page 16

New functionalities of CC/EFOD q Preformatted word document Available ü Downloadable and can be worked on offline and put back into online system q Electronic Supplements Available ü Feature that users can search difference information by Annex provision and/or by States q Validation feature Being tested ü Support workflow of State’s internal coordination and approval process for completion of compliance checklist ü Enable an authorized user to notify differences to ICAO 18 June 2021 Page 17

Preformatted word document q The feature is available only to NCMCs ü Once the word document is downloaded, data in the EFOD system is locked ü In order to allow users to edit data online, NCMC should either upload the word document or unlock the data 18 June 2021 Page 18

Protocol Questions (PQs) q States ü View and search PQs ü Conduct a self-assessment on safety oversight system • Update implementation status of PQ(S/NS/NA) with evidence • Provide implementation status of new questions • Attach evidence documents q ICAO ü Keep Protocol Questions updated ü Update status of State’s PQ based on latest CMA activities ü Generate the State LEI ü Collect information from external stakeholders 18 June 2021 Page 19 19

Protocol Questions (PQs) Search a PQ Select a PQ Status of Implementation by ICAO – View-only State Assessment by States 18 June 2021 Page 20 20

Mandatory Information Request (MIR) q ICAO üRequests States to submit specific information and documentation ü Includes the following information when issuing MIRs • Relevant PQ(s) • Reason and reference for requesting information • Due date for requested information üRecords the result of review on submitted information and updates the status of MIR (closed/open/escalated to Findings/SSC) 18 June 2021 Page 21 21

Mandatory Information Request (MIR) q States üRequest extension of the due date üRespond with evidence requested üSupply requested information and/or documentation Note: From 1 Jan 2013, email will serve as a main tool for ICAO to issue MIR until the online application is in place 18 June 2021 Page 22 22

Findings & Recommendations (F&R) Change in methodology (From F&R based to PQ based) ORG/01 One CAP Assessment 18 June 2021 One CAP/CAP Update including: Item 1. . . Item 20 One Finding Remarks The curent organizational structure of the CAD is not determined in sufficient detail as to provide a clear definition of the regulatory and safety oversight functions and responsibilities related to PEL, IOS, AIR< ANS and AGA. In addition, the available organizational charts have not been approved yet. 2. 011 3. 103 4. 027 5. 021 5. 023 5. 045 7. 031 7. 053 7. 203 7. 271 7. 327 … One Finding Remarks One CAP/CAP Update One CAP Assessment 2. 001 PEL One Finding Remarks One CAP/CAP Update One CAP Assessment 3. 101 . . . ANS One Finding Remarks One CAP/CAP Update One CAP Assessment 7. 327 Page 23

Findings & Recommendations (F&R) q States ü View and search Findings by Area, CEs, PQ number ü Provide comments to ICAO on the preliminary findings PQ finding by ICAO – View-only Comments by State 18 June 2021 Page 24 24

Findings & Recommendations (F&R) q ICAO üUpdate the status of existing F&R(s) • F&R(s) can only be closed when a CAP addresses all relevant PQs and it is fully implemented and validated • When all relevant PQs of a F&R are validated as satisfactory, then the LEI will be updated accordingly üIssue new Finding(s) including: • A PQ and applicable CE found be to not-satisfactory; (With one finding addressing only one PQ) • A description of the identified deficiency üEscalate the Finding(s) to an SSC 18 June 2021 Page 25 25

Corrective Action Plan (CAP) q States ü View and search Finding(s) issued to a State ü View CAPs submitted during the CSA audit cycle ü Submit/update a CAP to address a Finding • • A list of proposed actions Action office(s) to the proposed actions Estimated implementation date (dd/mm/yyyy) Revised implementation date, when required ü Provide regular progress reports on a CAP • Completion (%) for each proposed action • Date of completion • Supporting objective evidence (reference) 18 June 2021 Page 26 26

Corrective Action Plan (CAP) Select a PQ Finding by ICAO – View-only CAP provided by States 18 June 2021 Page 27

Corrective Action Plan (CAP) q ICAO üRecord status of a CAP review (In progress/Completed) üRecord result of a review on a proposed CAP with comments üMonitor progress made to a CAP 18 June 2021 Page 28 28

CAP in i. STARS q CAP data has been migrated into CMA online framework’s CAP module by dividing States into two groups q CAP in i. STARs will progressively be phased out as data migration is completed 18 June 2021 Page 29 29

CAP in i. STARS § Group A States: Start using CMA Online Framework CAP Module as of 1 Jan 2013 § Group B States: Remain in i. STARs and move to CMA online framework progressively as their ICVM final reports are published and data migration is completed. CSA Group F&R Based CSA Completed 1 Yes 2 PQ Based CSA Completed ICVM CAP Updates in ISTARS Yes 4 A 5 6 7 Yes Yes Yes B 9 18 June 2021 ICVM Planned In Dec. 2012 ICVM Planned in Jan. ~Apr. 2013 Yes 3 8 ICVM PQ based CAP ICVM Report Result provided in Planned In In Progress Published ISTARS Nov. 2012 Yes Yes Page 30 30

USOAP report q Final report (static) üFinal reports based on activities conducted since the first cycle of USOAP üPublished according to predetermined timelines after an activity 18 June 2021 Page 31 31

USOAP Report q USOAP live charts (dynamic) üBased on the most recent online and/or onsite activity for each audit area/CE Note: When you click the icon, you will be requested to enter a user account for ICAO Portal since the report resides on i. STARS. 18 June 2021 Page 32 32

Static and Dynamic Reports - Onsite Preparation Online Dynamic Report Onsite Activity Manager Onsite Mission Validation Reporting CSA - Live CSA Report - Live (Based the latest activity) Prepare PQ checklist Complete selfassessment Static Report TL Submission SPO Validation Prepare Draft Report 45 days prior 1~2 weeks ICVM - Preview TL Acceptance Updated Report Live TL updates ICVM result to incorporate feedback from States Finalize and send Draft report to State 90 days after Publish Final Report to State 165 days after Note: Live mode is available to all users of all States while preview mode is limited to the NCMC of the concerned State and ICAO users 18 June 2021 Page 33 33

USOAP Library q A centralized library for documents related to USOAP CMA including: üAssembly and Council Working Papers üDoc 9734 – Safety Oversight manual and Doc 9735 - USOAP CMA manual ü Generic MOU approved by the Council üMaster copies of PQs and SAAQ üState Letters and Electronic Bulletins üUSOAP CMA activity schedule 18 June 2021 Page 34 34

Tutorials and Help q Provides access to all training/tutorial material related to USOAP CMA including: ü Tutorial videos/help files on how to use USOAP CMA online framework ü CBT on how to develop an effective CAP ü Up-to-date workshop/seminar materials in available languages 18 June 2021 Page 35 35

Feedback q States ü Report issues and send queries related to CMA online framework q ICAO ü Keeps track of all reported issues and manages the resolution of such issues 18 June 2021 Page 36 36

CMA related services in i. STARs q i. STARs currently hosts four services : - USOAP results reporting To REMAIN - Corrective Action Plan - ICVM self-assessment checklist - Onsite activity tools To Online Framework https: //portal. icao. int/istars F&R USOAP Results by State ICVM CSA Result F&Rbased Based. CAP 1 AFDD 3 5 4 PQ based CAP PQ 6 7 CAP/CAP Updates ICAO CAP Assessment CAP and CAP Assessment Sharepoint List MSQL 2 PDF/ Word 18 June 2021 Infor Path Info Path Page 37 37

Updating EFOD 18 June 2021 Page 38 38

Verification by States q In April 2011, States were requested to: (SL AN 1/1 -11/28) ü Recognize EFOD as an alternative means of filing of differences ü Verify and confirm the data in EFOD which had previously been submitted through CC under USOAP • Over 100 States have been updating data in EFOD • 45 States sent replies (34 States completed verification, 11 States submitted plan for completion) 18 June 2021 Page 39

Procedures on the use of EFOD q Considered and approved by Council at its 195 th Session q To be published by Secretary General in 2 nd Quarter 2013 q Table of Contents 1. 2. 3. 4. 5. 6. 7. 18 June 2021 Introduction/Scope Access to the System Languages Annex Information Compliance/Differences Information System performance References Page 40

Filing of Differences Task Force (FDTF) 18 June 2021 Page 41

Background q Unclear requirements regarding filing of differences from ICAO provisions ü According to USOAP report, more than 70% audited States were found to be not satisfactory in fulfilling requirements of filing and publication of differences. Convention Article 38 and Assembly Resolution • Notification to ICAO of ALL differences from Annex provisions 18 June 2021 Annex 15 • Publication in the AIP of SIGNIFICANT differences from Annex, PANS and SUPPs provisions Page 42

Establishment of TF Deliverables • Draft policy on the use of EFOD – Completed • Comprehensive study on issues regarding filing of differences in general with a view to improving relevant rules and procedures and providing additional guidance to States – In Progress 18 June 2021 Page 43

Comprehensive study on issues regarding filing of differences q Being finalized by FDTF q To be considered by Council at its 198 th Session in 2013 q Table of Contents 1. 2. 3. 4. 5. 6. 7. 8. 18 June 2021 Executive summary Introduction ICAO’s past efforts Current situation Analysis of differences Known issues Importance and benefits of filing and publication of differences Future strategies and recommendations Page 44

Known EFOD issues ICAO Misleading, inconsistent, insufficient guidance Limited value of Supplements Annex Amd’t frequency and unrealistic applicability dates 18 June 2021 States • Not all differences filed • Differences not filed correctly • Significant differences not published in AIPs Annex provisions comprehension and interpretation difficulties Lack of State commitment Lack of effective national process and procedures Lack of knowledge and expertise Page 45

FDTF Recommendation q Improve and develop ICAO documents q Improve SARPs development and amendment process q Increase transparency of differences information q Enhance assistance programme in cooperation with various entities q Introduce a quality control process wherein ICAO reviews differences and provides timely feedback q Improve the EFOD system to ease the work of States 18 June 2021 Page 46

Review Ø Overview of CMA Online Framework Ø Functionalities of each component Ø Implementation of revised protocol questions Ø Update on EFOD Ø Finding of Differences Task Force (FDTF) 18 June 2021 Page 47

Review 1. The online Framework is a web-integrated system which includes: _____ of safety-related information and documents from different sources, ____ and _____ of safety oversight activities by ICAO and Member States a. Collection, Monitoring, Reporting (Slide 4) 18 June 2021 Page 48

Review 2. Name 3 of the 5 functions that a State can perform using the OLF. • • • Manage OLF user accounts Keep up-to-date SAAQ and CC/EFOD Provide a CAP update and/or progress per corresponding protocol finding Provide self-assessment of each “Not Satisfactory” PQ Respond to MIRs Slide 8 18 June 2021 Page 49

Review 3. Who has full control of user accounts for the State? • NCMC (Slide 12) 4. Which icon is used to access filing of differences? • “Manage CCs/EFOD” icon (Slide 15) 5. Name 3 of items on the OLF CMA library? • Assembly and Council Working Papers • Doc 9734 – Safety Oversight manual and Doc 9735 - USOAP CMA manual • Generic MOU approved by the Council • Master copies of PQs and SAAQ • State Letters and Electronic Bulletins • USOAP CMA activity schedule (Slide 34) 18 June 2021 Page 50

Review 6. i. STARs currently hosts four services. Which one will remain on i. STARS • USOAP results reporting (Slide 37) 18 June 2021 Page 51