Unified modular methodology for calculating noncompliance score and

Unified modular methodology for calculating non-compliance score and accessibility rating Vojtech Regec Milan Regec SLOVAKIA

Background & Experience 500 450 400 Number of tested websites 1773 tested websites during 10 years 450 320 300 250 200 150 100 150 130 100 50 0 23 2005 2007 2008 2009 year 2010 2011 2013 2015

Milestones | Legal Framework • Act No. 275/2006 on Information Systems of Public Administration • Decree of Ministry of Transport, Posts and Telecommunications of the Slovak Republic No. 1706/M 2006 on Standards for Information Systems of the Public Administration • Decree of Ministry of Finance 013261/2008 -132 on public administration information system standards • Decree of Ministry of finances 312/2010 col. about standards for information systems of public administration • Decree of Ministry of Finance 55/2014 Coll. , on standards for Public Administration. Information Systems. • Act No. 95/2019 Coll. on Information Technologies in Public Administration EN 301549 V 2. 1. 2

Milestones | Legal Framework • Act No. 275/2006 on Information Systems of Public Administration • Decree of Ministry of Transport, Posts and Telecommunications of the Slovak Republic No. 1706/M 2006 on Standards for Information Systems of the Public Administration • Decree of Ministry of Finance 013261/2008 -132 on public administration information system standards • Decree of Ministry of finances 312/2010 col. about standards for information systems of public administration • Decree of Ministry of Finance 55/2014 Coll. , on standards for Public Administration. Information Systems. • Act No. 95/2019 Coll. on Information Technologies in Public Administration

DIRECTIVE (EU) 2016/2102 on the accessibility of the websites and mobile applications of public sector bodies (50) Conformity with the accessibility requirements set out in this Directive should be periodically monitored. A harmonised monitoring methodology would provide for a description of the way of verifying, on a uniform basis in all Members States, the degree of compliance with the accessibility requirements, the collection of representative samples and the periodicity of the monitoring. Member States should report periodically on the outcome of the monitoring and at least once on the list of measures taken in application of this Directive.

DIRECTIVE (EU) 2016/2102 on the accessibility of the websites and mobile applications of public sector bodies (50) Conformity with the accessibility requirements set out in this Directive should be periodically monitored. A harmonised monitoring methodology would provide for a description of the way of verifying, on a uniform basis in all Members States, the degree of compliance with the accessibility requirements, the collection of representative samples and the periodicity of the monitoring. Member States should report periodically on the outcome of the monitoring and at least once on the list of measures taken in application of this Directive.

DIRECTIVE (EU) 2016/2102 on the accessibility of the websites and mobile applications of public sector bodies (51) The monitoring methodology to be established by the Commission should be transparent, transferable, comparable and reproducible. The reproducibility of the monitoring methodology should be maximised while taking into account the fact that human factors, such as testing by users, might have an influence on that reproducibility. To improve comparability of data between Member States, the monitoring methodology should describe the way in which the outcomes of different tests need to be or can be presented. In order not to divert resources from the task of making content more accessible, the monitoring methodology should be easy to use.

DIRECTIVE (EU) 2016/2102 on the accessibility of the websites and mobile applications of public sector bodies (51) The monitoring methodology to be established by the Commission should be transparent, transferable, comparable and reproducible. The reproducibility of the monitoring methodology should be maximised while taking into account the fact that human factors, such as testing by users, might have an influence on that reproducibility. To improve comparability of data between Member States, the monitoring methodology should describe the way in which the outcomes of different tests need to be or can be presented. In order not to divert resources from the task of making content more accessible, the monitoring methodology should be easy to use.

Goals To quantify the level of accessibility in a representative way with the focus on real accessibility Modular but objective score applicable in different environments Fair comparison of scores between different countries or between different time periods Meaningful statistical analysis and identification of problematic areas Independence of particular automated testing tool In line with W 3 C WAI Accessibility Conformance Testing approach

Background • the result of the automatic check (evaluation) Premise 1 is just the very beginning of testing • the automated tool cannot fully and reliably Premise 2 evaluate any single success criterion Premise 3 • human-based testing is essential determine the real impact on accessibility

Accessibility Conformance Testing (ACT) Rules Format 1. 0 (Oct 2019)

Failed outcome: further testing still needed

Failed outcome: further testing still needed

Strengths and weaknesses of accessibility verification methods METHOD automatic (automated tool) ASPECTS OF OBTAINED RESULTS reliability quantity quality time req. – +++ + +++ manual (evaluator) +++ + +++ – user (person with disability) ++ –

COMMISSION IMPLEMENTING DECISION (EU) 2018/1524 • The simplified monitoring shall aim to cover the following user accessibility needs to the maximum extent it is reasonably possible with the use of automated tests: • • • (a) usage without vision; (b) usage with limited vision; (c) usage without perception of colour; (d) usage without hearing; (e) usage with limited hearing; (f) usage without vocal capability; (g) usage with limited manipulation or strength; (h) the need to minimise photosensitive seizure triggers; (i) usage with limited cognition. Member States may also use tests other than automated ones in the simplified monitoring.

Homepage: National Council of the Slovak Republic

ERRORS • Empty Button • Empty Links • Low Contrast

Classification level of infringement with accessibility requirements

In Depth vs Simplified Verify compliance • Making sure criteria is met across sample • Additional scenario-based testing • Estimated average time: 8– 12 hours Detect noncompliance • Detecting first / most significant infringement • No scenario-based testing • Estimated average time: 5– 7 hours

In Depth vs Simplified Verify compliance • Accessibility Rating Detect noncompliance • Non-Compliance score

Sampling of website In Depth Testing • Homepage – level 0 • All pages – level 1 • Mandatory sequences (4) – variable level • Contextual sequences – variable level Simplified Testing • Homepage – level 0 • All pages – level 1

Levels of Infringement In Depth Testing • Severe infringement • Moderate infringement • In compliance • Not applicable Simplified Testing • Severe infringement • Moderate infringement • Infringement not detected • Not applicable

Levels of Infringement In Depth Testing • Severe infringement • Moderate infringement • In compliance • Not applicable Simplified Testing • Severe infringement • Moderate infringement • Infringement not detected • Not applicable

50 Success Criteria evaluated [VALUE] Robust [VALUE] Understandable [VALUE] Operable WCAG 2. 1 Level A, AA [VALUE] Perceivable

Application of Priority multiplicator PENALISATION 1 × 4 2 2 × 8 4 3 × 12 8 In compliance (infringement not detected) 0 0 0 Not applicable 0 0 0 Priority multiplicator Severe infringement Moderate infringement

List of criteria with added priority multiplicator based on real-life accessiblity impact 3 × 2 × 1. 1. 1. A 1. 3. 3. A 1. 3. 1. A 2. 1. 2. A 2. 1. 1. A 2. 4. 6. AA 1 × A (25) AA (19)

Calculation of Accessibility Rating (AR) for In-depth monitoring model 1 2 3 4 Segment Weight points Score portion perceivable 80 (20 × 4) 40% operable 68 (17 × 4) 34% understandable 40 (10 × 4) 20% robust 12 (3 × 4) 6% AR = p. AR 1 × 0, 40 + p. AR 2 × 0, 34 + p. AR 3 × 0, 20 + p. AR 4 × 0, 06

Calculation of Non-Compliance score (NCS) for Simplified monitoring model 1 2 3 4 Segment Weight points Score portion perceivable 80 (20 × 4) 40% operable 68 (17 × 4) 34% understandable 40 (10 × 4) 20% robust 12 (3 × 4) 6% NCs= p. NCs × 0, 40 + p. NCs 2 × 0, 34 + p. NCs 3 × 0, 20 + p. NCs 4 × 0, 06

Rating levels for in-depth (AR) and simplified monitoring (NCS) AR (%) Conclusion 100 96 Fully Accessible 95 91 90 81 80 71 High Level of Accessibility Moderate Level Accessibility Low Level Accessibility 70 and lower Inaccessible NCS (%) 0 1 - 7 8 - 25 26 - 50 Conclusion Fully Compliant Low level of noncompliance Moderate level of noncompliance High level of noncompliance 51 and more Severe non-compliance

Example of results from simplified monitoring Partial results Segment perceivable operable understandable robust 2 4 9 5 1 2 14 0 0 3 7 0 0 2 1 0 Partial score of non-compliance 35% 24% 15% 33% Degree of non-compliance High Moderate High Severe infringement Moderate infringement Infringement not detected Not applicable Overal non-compliance score Conclusion 27% High level of non-compliance

Discussion Could we rely today just on automated tools in simplified monitoring? How to achieve the objectivity and consistency in reporting severity of infringement? Should the scores of in-depth and simplified monitoring be comparable?

Thank you for our attention! Vojtech Regec • Associate Professor & Vice-Dean for Science, Research and Doctoral Studies • Palacký University Olomouc • Tel: +420 774 239 737 • Email: vojtech. regec@upol. cz Milan Regec • Director, Modrý anjel, n. o. • Comenius University in Bratislava • Tel: +421 903 420 043 • Email: milan. regec@uniba. sk