Using Verbal Reports for Data Collection and Analysis

Using Verbal Reports for Data Collection and Analysis Maja Gjurovikj STANAG 6001 testing team Ministry of Defence Republic of Macedonia

Why trialling? Collect data to be analyzed: Do items test what they were meant to test? Do they distinguish between masters or non- masters? Are the ambiguous? Inform about test administration procedures: time instructions answer keys Bank items to generate new tests: determine characteristics of items keep/revise/reject

Test item characteristics CODE IMMIGRATION CODE CAR ALARMS 2 DATA FROM Pretesting Feb. 8 2011, Perugia DATA FROM Pretesting Oct. 2011, Bi. H AS NO. 32 AS NO. 40 GROU P A B C D ? T 4 13 4 4 0 M 4 7 10 4 0 B 12 7 4 2 0 % 26% 36% 24% 13% 0% GROU P A B C D ? T 3 17 4 5 2 M 8 10 6 8 3 B 3 8 6 7 7 % 14% 36% 16% 21% 12% FV DI 26% -0, 36 FV DI 36% 0, 30

Psychometric approaches vs. mentalistic approaches empirical validity theoretical validity • concurrent validity • face validity • predictive validity • content validity • construct validity QUANTITATIVE APPROACHES EXAMINE PSYCHOMETRIC PROPERTIES OF THE TEST QUALITATIVE APPROACHES PROVIDE MORE INFORMATION ON TEST CONTENT, TEST TASK PROPERTIES AND TESTTAKING PROCESSES

What is verbal report analysis? a special label used to describe the verbal data gathered a technique that draws inferences about the cognitive processes that produced the verbalization comprised of ‘protocols’ - utterances made by an individual invaluable at the trialling and pretesting stage supplement other quantitative data

Variations on verbal report procedure Form of Report Temporal Variations Concurrent TALK ALOUD THINK ALOUD Retrospective Procedural Mediated Non-med. Mediated Variations Non-med. Concurrent Retrospective Mediated Non-med.

Stages in collecting and analyzing verbal reports DATA COLLECTION STAGES • Task identification • Task analysis • Selecting an appropriate procedure • Selecting subjects • Training subjects • Collecting verbal reports • Collecting supplementary data • Transcribing verbal reports DATA ANALYSIS STAGES • Developing an encoding scheme • Segmenting protocols • Encoding protocols • Calculating encoder reliability • Analyzing data Source: Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate

Examples of verbal report methodology Cohen (1994) five test-takers 2. a two-part test 3. concurrent reports 4. observe test-takers during the test-taking process 5. give test-takers a questionnaire 6. conduct in the test-takers’ L 1 Buck (1994) 1. six test-takers 2. 54 -item test based on a single listening text 3. 13 short sections, one at a time 4. attend a post-test interview 5. take the test items again 1.

Advantages vs. disadvantages reveals test validity provides direct information has wide application commitment in time disruption of behaviour individual differences

Some qualitative data analysis softwares Coding Analysis Toolkit (CAT; https: //cat. texifter. com) Weft QDA (http: //www. pressure. to/qda) Text Analysis Markup System (TAMS; http: //tamsys. sourceforge. net) Transana (https: //www. transana. com) QDA Miner Lite (https: //provalisresearch. com/products/qualitative-data -analysis-software)

Points to note reveals the dynamics of the test taking processes uncovers students’ unobservable behavior resourceful feedback concerning the mental processes or test-taking strategies complements the analysis generated from quantitative investigation a developing and maturing method

References: Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate. Grotjahn, R. (1986). Test validation and cognitive psychology: Some methodological considerations. Language Testing 3: 159– 185. Zheng, Y. (2009) Protocol Analysis in the Validation of Language Tests: Potential of the Method, State of the Evidence. International Journal of Pedagogies and Learning 5: 124– 137 Council of Europe (2009). Relating Language examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment Cambridge: Cambridge University Press. (https: //www. youtube. com/watch? v=Bwp. PIi. BK 0 c. A)