ECDL Test Development and Validation Process Paul Davis

ECDL Test Development and Validation Process Paul Davis, Oxford University Computing Services Garry Cleere The European Computer Driving Licence Foundation Ltd. CAA Conference

Test development process CAA Conference 2

Syllabus Definition (Expert Opinion) CAA Conference 3

Soft Test Characterisation • Non-threatening test experience • Predominantly performance based • Straight forward language • No trick questions • Coherent outputs • Sustains the metaphor of the driving licence • Ample time CAA Conference 4

Forum score CAA Conference 5

Test characteristics – real life CAA Conference 6

Forum duration of tests CAA Conference 7

Mean duration CAA Conference 8

Criticality (Core Item Identification) Exercises CAA Conference 9

Model Tests Exercise • • • SMEs create 4 streams of tests Spread core items across streams Provide a reasonable coverage in each test Provide syllabus coverage across streams Average number of items CAA Conference 10

Model Tests Grids Scheme REF. TASK ITEM 3. 1. 1. 1 Open (and close) a word processing application. 3. 1. 1. 2 Open one, several documents. 3. 1. 1. 3 Create a new document (based on default, other available template). CAA Conference MT 1 MT 2 MT 3 MT 4 NE NE 1 1 11

Item Count 36 Items • Item Count dictated by coverage requirement • Dictated by congruency and criticality in tests • 100% Syllabus coverage throughout ECDL tests • Balanced, discriminating test profiles for Syllabus V. 4 • Tests Syllabus Version 4. 0 rigorously • High reliability in tests and between tests • Item Count standardisation in all Modules CAA Conference 12

Result against completion time CAA Conference 13

Cut Score / Pass Mark • Systematic approach and Industry based cut score technique (modified Angoff Method) • Underpinned by expert opinion • Cut Score exercises carried out for each module • Borderline Candidate, or Minimally Competent Candidate (MCC) • Meaningful Cut Scores based on profile of ECDL tests derived from ECDL V 4 Syllabus • Item Count standardisation in all Modules • An auditable, standard cut score process • Common cut score across all ECDL core level modules, (M 1 - 7) CAA Conference 14

Cut Score Exercises CAA Conference 15

Model Tests Collation and Balancing CAA Conference 16

Characterisation Test Template Version 4. 0 CAA Conference 17

Item / Test Development Dependant upon: • Syllabus • Soft Test Characterisation • CTT • ECDL-F Evaluations • Outcome Probability Grids (expected pass rate from question) CAA Conference 18

Item / Test Development CAA Conference 19

Expert Review of Items / Tests • Syllabus Congruence • Syllabus Coverage • Compliance with CTT • Appropriateness of Item format • Technical Accuracy • Language & Clarity • Text Content CAA Conference 20

Expert Review : ATES Systems • • • CAA Conference Test Presentation / Interface Features Timing & Information Appropriateness of ATES Test Content Marking Algorithm Candidate Information and Guidance Practice Tests Documentation and Training Materials ATES Technical Support Reporting Test Generation Security Features 21

Perceptions of tests CAA Conference 22

Choice of test engine (V 3) Simulation In. Application Rich Simulation Web-based MCQ/HS MQTB First 12 12 1 3 3 Second 8 7 5 5 5 Third 5 4 11 6 3 Fourth 3 5 12 6 3 Fifth 1 1 0 9 15 CAA Conference 23

Expert Review : Test Design / ATES Systems • • • ATES System Overview ATES System Hardware Requirements ATES Operating System Requirements Other Software Requirements Minimum Screen Resolution Assistive Technologies • Screen Design and Visual Impairment CAA Conference 24

Assembly and Administration of Pilot Tests • Tests are made available for Pilot Test activity • Qualitative analysis • Test taker and administrator observation sheets / questionnaires • Test data requirements Remember: • Across 4 streams for 7 tests across 100 countries and many languages. CAA Conference 25

Pilot Tests Review Pilot Test results are reviewed by the ATES Evaluations Group. Any changes requested in the tests based on Pilot testing are then undertaken before the live certification tests are administered. To ensure statistical validity some pre-testing on the fly is used until results don’t change. Maintenance then takes over. CAA Conference 26

Empirical Evidence Evaluation ECDL Tests are evaluated and monitored on an ongoing basis by the ATES Evaluations Group using recognized psychometric measurements. CTT Version 4. 0 - Test behaviour statement CAA Conference 27

Ongoing Maintenance of Tests ECDL tests are monitored on an ongoing basis by the ECDL Foundation ATES Evaluations Group. Tests are updated in line with Syllabus revisions and also where test results or empirical evidence indicates that items or tests are not functioning in line with test behaviour parameters. CAA Conference 28

Process Steps and the Validity Argument • • • Documented process - expert opinion Detailed test data reporting requirements Data fed back to the ECDL Foundation Industry Standard techniques Process Loop Valid ECDL Tests CAA Conference 29

Contacts Dr Paul Davis Deputy Head, LTG, OUCS 13 Banbury Road Oxford OX 2 6 NN UK Tel: 01865 283414 paul. davis@oucs. ox. ac. uk CAA Conference Garry Cleere Programmes Manager ECDL Foundation 107 The Windmill Sir John Rogersons’ Quay Dublin 2 Ireland Tel. 00 353 1 679 2847 Tel. 00 353 1 679 2921 garry. cleere@ecdl. com 30