Unit 4 Data data collection and the research

Unit 4. Data, data collection and the research process • In this unit we will focus on the following important concepts: – Reliability (consistency) of the way data are collected. – The research process. © 2015 Dávid & Ryan 1

Reliability – essentially consistency • Questions to consider – What would you do to increase the consistency of (lesson) observations (if it is your chosen instrument)? – What would you do to increase the consistency of your questionnaire/ your test? • Class discussion to clarify answers, alternatives. • Look at the list of techniques again (next slide) and choose one or two other instruments. – How do you think you could increase consistency in these cases? © 2015 Dávid & Ryan 2

Instruments and data collection techniques 1. Observation – 5. Repertory grids Done in Focus on the Learner. – 2. Interviews 3. Questionnaires, feedback and tests – Some of this done in Focus on the Learner. Done in Focus on the Learner. 6. Text (discourse) analysis – – Conversation Analysis Diary studies 7. Case studies 4. Introspection – Concurrent/retrospective? • Verbal protocols © 2015 Dávid & Ryan 3

Reliability – consistency (contd. ) • Consider the cases to the right. A good measure of reliability can only be achieved in one. – Which one? – How is the validity of conclusions threatened in all the other cases? Validity or reliability itself is the problem? 1. A long questionnaire. Some questions are leading, misleading or miss the point. 2. Brilliant interview questions capture the problem very well. A considerable number of interviews made. 3. Excellent ideas and questions. The number of questions is 3, and the qnnaire was done by 7 out of 15 students. © 2015 Dávid & Ryan 4

Two important terms • Remember these terms from your studies earlier (Focus on the Learner)? • T_ _ _ • S_____ • Can you recap what they mean? • Can you guess or shall we play „Hangman”? • This is the tradition from action research. • • How are these two terms related to reliability? What important aspect of research do they miss? © 2015 Dávid & Ryan 5

Background In surveying, triangulation is the process of determining the location of a point by measuring only angles to it from known points at either end of a fixed baseline, rather than measuring distances to the point directly as in trilateration. The point can then be fixed as the third point of a triangle with one known side and two known angles. © 2015 Dávid & Ryan 6

The research process 1 • Class question: Assuming you have • selected the research instruments appropriately, • you have done the operationalisation successfully, – what other principle is important? © 2015 Dávid & Ryan 7

The research process 2 • The importance of keeping to the stages of the research process – What do you think the stages are? • Discussion in pairs or groups – What sort of validity are you likely to get? © 2015 Dávid & Ryan 8

Stages of the research process 1. Formulating the • Questions: question(s) and reading – Is the ordering important? 2. Operationalisation: Design – What important of instruments quality aspect will 3. Moderating and probably be affected pretesting (piloting) the if the steps are ignored or some get instruments skipped? 4. Data collection – What sort of validity 5. Analysis would be harmed? 6. Reporting research (write. How would you describe it? up), drawing conclusions. © 2015 Dávid & Ryan 9

A summary • What is reliability a property of? • What is your observation about the relationship between reliability and validity? – The relationship between reliability and validity is asymmetrical (Bachman 1990, Messick 1989). • The possibility of being „consistently wrong” about otherwise right goals, things to do, etc. • What follows from this? – Popular beliefs: The more you increase reliability, the weaker validity becomes (Gardner et al. 2010) • Comment. © 2015 Dávid & Ryan 10