Measuring IT Accessibility Sharon Laskowski Visualization and Usability

Measuring IT Accessibility Sharon Laskowski Visualization and Usability Group sharon. laskowski@nist. gov October 16, 2000

NIST Mission n To strengthen the US economy and improve the quality of life by working with industry to develop and apply technology, measurements and standards n n n Information Technology Lab - To develop measurements and standards to support the IT industry and key users of it Information Access Division - To accelerate the development of technologies that allow intuitive, efficient access, manipulation and exchange of complex information by facilitating the creation of measurement methods and standards Voluntary consensus IT standards and how to measure and evaluate technology Note: NIST does not do individual product test and evaluation a la Consumer Reports

How did I get involved in Section 508 activities? n n DOC asked NIST to provide some technical expertise to the EITAAC and the 508 Ad Hoc Committee The VUG has some relevant research projects: n n n Web usability evaluation tools and benchmarks Industry Usability Reporting (IUSR) Project Participation in NCITS V 2 Access Interfaces Technical Committee NOT NIST compliance with ADA, 508 requirements BUT rather, what does it mean to measure the accessibility of an IT product?

This Presentation n n Some ideas about what it means to measure accessibility and to determine whether a product is accessible and provides comparable access A few suggestions for how to test and evaluate once we understand what the definitions and metrics are…

Quick Intro to Conformity Assessment n Conformity Assessment (ISO/IEC) - any activity concerned with determining directly or indirectly that relevant requirements are fulfilled n n n Can verify that a product meets a given level of quality or safety and provide the user with information about its characteristics, the consistency of the characteristics, and/or performance of the product Includes: sampling and testing, inspection, certification, and quality and environmental system assessment and registration Standards underlie these activities and can have a major impact on the outcome

Intro: Standards n Standard: a prescribed set of rules, conditions, or requirements concerning definitions of terms; classification of components; specification of materials, performance or operations; delineation of procedures; or measurement of quantity and quality in describing materials, products, systems, services, or practices n n Voluntary or mandatory should be clear, concise, precise, credible, unambiguous design vs. PERFORMANCE standards should specify all essential characteristics necessary for achieving the objective of the conformity assessment

Intro: Testing n Test: technical operation that consists of the determination of one or more characteristics of a given product, process, or service according to a specified procedure n n n Accuracy (or bias) - departure from “true value” Variability (or precision) - repeatability, reproducibility Need accreditation procedure, especially if a regulatory requirement must be met

Intro: Certification n Certification - process of assuring product conforms n n n Self-certification Second Party certification - buyer requires and certifies that product meets certain standards Third Party certification n n Independent lab Professional or technical society Consumer organization Gov’t organization (e. g. for building codes)

Testing Accessibility of IT n Accessibility depends on successful USER interaction n Important to measure BOTH design and performance considerations n n n Design: text tags on images for web pages Performance: task completion (efficiency, effectiveness, and satisfaction are potential metrics) Testable, accurate, reproducible, think about: n n Screen flicker specs Color contrast “Consistent” images Complete typical task in n minutes, with < m errors

So, what do we need in order to test the user interaction for accessibility? n n n [Uniform interpretation of 508 standard and clarification of the 508 standard to insure testability] Ways to capture performance with users Accurate, repeatable, reproducible tests Tests should be low cost and quick to apply Industry needs to be able to apply tests during development process

Capturing and Measuring User Interaction n Usability engineering has techniques for evaluating the usability of products n n n Performance-based: user testing, formative and summative User testing typically specifies the tasks and user demographics Metrics: efficiency, effectiveness, satisfaction Usability data rarely shared between vendors and software purchasers NIST has developed a common usability reporting format to encourage exchange of usability information

Can this be applied to accessibility? n n n We can report on testing with users using the common industry format (CIF) as a guideline, BUT, this is not sufficient as the basis for a reproducible, repeatable, low cost set of tests for accessibility The CIF is a starting point for discussion between vendor and consumer companies n n n They must agree that the tasks and the user demographics tested are the right ones for the consumer company If not, additional testing is needed However, it does provide some assurance that basic usability principles were adhered during design and development

A CIF for “accessibility”? n Reporting on accessibility requires a more rigorous form of the CIF n n Specifications for which disabilities and what ranges should be tested Reporting on product in combination with other software or devices, e. g. screen readers Low cost options for testing that might not involve testing with people with disabilities directly (e. g. screening techniques) Benchmarks derived from expected performance by users without disabilities n n How bad can performance be and still be acceptable? Some of this requires research; it is not a trivial extension

My Conclusions n n Performance-based evaluation of accessibility need to be addressed An approach which combines an “accessibility” CIF along with benchmarks for acceptable performance has potential n At a minimum, producing such a CIF shows that a vendor has at least done some basic testing for accessibility For more info: http: //www. nist. gov/iusr http: //www. section 508. gov n