6 Data collection EDN 6015 F Cheryl HodgkinsonWilliams

6. Data collection EDN 6015 F Cheryl Hodgkinson-Williams 3 -8 February 2014

1. Trigger problem 2. Context & Rationale 3. Concepts & literature 5. Research Questions 4. Conceptual frameworks & theories 6. Data collection

Once upon a time … • Other researchers used … (approaches & methodologies) to find out what was happening and this given me the idea to use … in my study I will approach … to participate in my study

From the UCT guidelines (1) • Masters and Ph. D research studies invariably involve some kind of empirical work. This means that the researcher enters a field to collect data, the analysis of which will allow the research question to be addressed. Such a field may comprise part of an archive of, say, historical and/or policy documents. In either case, a number of crucial decisions need to be made about conducting this empirical work:

From the UCT guidelines (2) Sampling: Where will you collect data from? If you are doing archival research of an historical nature, over what time period will your study extend? What documents will form part of your data? Which will you leave out? On what basis will you make this decision? [Who will participate in your study? How will you select them? ] A range of important decisions need to be made, about how to make the project meaningful, but at the same time manageable in terms of the time frame and resources available.

From the UCT guidelines (3) Data collection instruments: Developing a research design means making decisions about who (or what, in the case of textually-based studies) to approach for the purposes of data collection. It also means making decisions about what [and how much] data to collect and [how best to collect this data]. These decisions need to be constantly referred back to your research question: what kind of data do you need in order to address it? Collected how? Different kinds of instruments are available; will you use (maybe adapt) some instrument presently in use, or develop your own? (UCT Faculty of Humanities Guidelines for Research)

Terminology problematic • The terminology used to describe methodological choices is not standard or consistently used and can be very confusing. – Broad types of research = strategies (Twining 2010), approaches, styles, methodology (Cohen et al. 2007) – Specific research methods or process = methods, tools, instruments (Cohen et al. 2007)

Terminology in this course Type of research Research participants Research strategy Research method Research instrument • Types usually include: Quantitative, qualitative or mixed-methods • Those whom you will interview, observe or ask to participate in a survey • Broad types of research = strategies, approaches, styles e. g. Quasi-experiment, survey, ethnography, case study, action research, historical studies • Specific research methods or process = methods, tools, instruments e. g. Pre-tests/post-tests, non participant observation, questionnaires, telephone surveys, interviews • Associated research instrument – the way in which the data is physically collected e. g. Test paper, observation schedule, field notes, paper-based or electronic questionnaire, audio transcript, interview transcript

Define understanding of methodological choice • In your proposal and dissertation make sure that you define WHOSE understanding of a research ‘type’ ‘’strategy’, ‘method’ or ‘instrument’ you are using (e. g. Cohen et al. 2007) so that you aren’t blamed inconsistent use of terms!

Types of research • Broadly speaking research is often referred to as falling into three types of research: – Quantitative – Qualitative – Mixed Methods Much of the research in the ICT in Education field is either qualitative or uses mixedmethods

Sampling / Participant selection Maxwell (2008: 234 -235) identifies the key sampling issues: Who What Sampling When Where • Who – are the individuals you want to observe or interview? How many will be enough? • Where – in what setting will the study take place? • When – at what time will the study take place? • What – other sources of information do you require?

Negotiating a research relationship • “Your relationships with the people in your study can be complex and changeable, and these relationships will necessarily affect you as the ‘research instrument’, as well as have implications for other components of your research [e. g. ethics and validity]” (Maxwell 2008: 234) • You need to consider how to build a relationship with your participants – trust, honesty and respect are important

Research strategy: Experiment • Purpose: – – Compare under controlled conditions Measure objectively Establish causes Predict behaviour • Assumptions: – A real world exists & there is a correct way of describing it • Methods: – Tests/Assessments – Non-participative observation • Not widely used in ICT in Education research

Research strategy: Quasi-experiment • Purpose: – – Compare under natural conditions with existing groups Endeavours to measure as objectively as possible May establish causes May predict behaviour • Assumptions: – A real world exists & there rigorous way of describing it, but acknowledged that natural science methods not as appropriate in social sciences. Useful if ‘causes’ and ‘predictions’ are tentative • Methods: – Pre-test/posttest – Non-participant observation • Used in early in ICT in Education research – calls for more use (Underwood & Underwood in Twining 2010)

Research strategy: Correlation • Purpose: – Uncover possible relationships between variables – Explain the resultant correlations according to an existing theoretical model or develop a tentative theory • Assumptions: – A real world exists & there is a correct way of describing it (Positivist) – A real world exists & there rigorous way of describing it, but acknowledged that natural science methods not as appropriate in social sciences. Useful if ‘causes’ and ‘predictions’ are tentative (Post-positivist) • Methods: – Analysis of secondary data (e. g. Access to computers at school and computer literacy tests at university) • Often misused to endeavour to indicate causal relationship (Twining 2010: 161)

Research strategy: Survey • Purpose: – Gather opinions, scores, conditions etc. from a sample of participants to make generalisations of the broader population • Assumptions: – A real world exists & there is a correct way of describing it (Positivist) – A real world exists & there rigorous way of describing it, but acknowledged that natural science methods not as appropriate in social sciences. Useful if ‘causes’ and ‘predictions’ are tentative (Post-positivist) – A real world exists, but everyone experiences it differently (Interpretivist) • Methods: – Questionnaires – Telephone or Internet surveys • Commonly used by ICT in Education research (Twining 2010: 163)

Research strategy: Ethnography • Purpose: – Construct in-depth depictions of the every day life events of people, through active researcher participation and engagement (Crichton & Kinash 2003) – Elicit perceptions and views of participants • Assumptions: – A real world exists , but everyone experiences it differently (Interpretivist) – A real world exists, but people without power experience it differently (Critical theoretical) • Methods: • – Participant observation – Interviews (semi-structured, unstructured) ICTs offering new ways of undertaking ethnography, e. g. ‘Virtual Ethnography’ (Crichton & Kinash 2003)

Research strategy: Case study • Purpose: Describe and interpret unique individuals, courses, institutions in depth • Assumptions: – A real world exists & there rigorous way of describing it, but acknowledged that natural science methods not as appropriate in social sciences. Useful if ‘causes’ and ‘predictions’ are tentative (Post-positivist) – A real world exists, but everyone experiences it differently (Interpretivist) – A real world exists, but people without power experience it differently (Critical theorist) • Methods: – – Non-participant or participant observation Interviews (semi-structured, unstructured; individual or focus group) Questionnaires Documentary resources/artefacts • Used extensively in ICT in Education field

Research strategy: Action research • Purpose: – Plan implement, monitor and evaluation an intervention to improve practice or solve a local problem – Empower participants to be involved in intervention and research process • Assumptions: – A real world exists, but everyone experiences it differently (Interpretivist) – A real world exists, but people without power experience it differently (Critical theorist) • Methods: – – Non-participant or participant observation Interviews (semi-structured, unstructured; individual or focus group) Questionnaires Documentary resources/artefacts • Used extensively in ICT in Education field in schools and HE

Research strategy: Historical • Purpose: – Reconstruct a chain of events to identify those events that caused of triggered other events • Assumptions: positivist, interpretivist; critical theoretical; post modern (especially feminist) and critical realist • Methods: – Documentary resources/artefacts • Not used extensively in ICT in Education field.

Research strategy: Literature review • Purpose: – Provide an overview of scholarship in a certain discipline through an analysis of trends and debates (Mouton 2001: 179) • Assumptions: Interpretivist; critical theoretical; and critical realist • Methods: – Critical literature review – Meta-analysis • Used in ICT in Education field

Activity 1 • Referring to one of the articles you located yesterday identify the overall research strategy within your group and then report back to the class.

Mapping strategies to methods • Research strategies – – – – Experiment Quasi-experiment Survey Ethnography Case study Action research Historical studies Literature review • Research methods – – – – – Tests/Assessments Pre-tests/post-tests Non participant observation Questionnaires Telephone/Internet surveys Participant observation Interviews Documents and/or artefacts Critical literature review

Activity 2 • Referring to one of the articles you located yesterday identify the various methods adopted in this article within your group and then report back to the class.

Research instruments • Research instruments (existing or developed) – Test paper – Assessment checklist – Observation schedule – Questionnaire – Interview schedule – Audio transcript – Usage logs (e. g. logins, downloads, links, page views) – Conceptual framework

Activity 3: Mapping methods to instruments - Which fits where? • Research methods – – – – – Tests/Assessments Pre-tests/posttests Non participant observation Questionnaires Telephone/Internet surveys Participant observation Interviews Documents and/or artefacts Critical literature review • Research instruments (existing or developed) – – – – Test paper Assessment checklist Observation schedule Questionnaire Interview schedule Audio transcript Usage logs Conceptual framework

Instrument-specific techniques • In developing your instruments there a number of instrument-specific techniques you will need to master e. g. , – Developing questionnaires with appropriate question types (e. g. open/closed; Likert scale; categories) – Framing meaningful interview questions (Openended, clarifying) – Developing observation schedules (e. g. determining focus areas)

Assumptions underpinning methodological choices • The way in which we decide to ‘collect and interpret our data’ is dependent on our underlying philosophical assumptions: • “. . . methods are embedded in commitments to particular versions of the world (an ontology) and ways of knowing that world (an epistemology). These commitments are always held by the researcher, mostly tacitly” (Usher 1996: 13)

Bisman, J. (2010). Postpositivism and Accounting Research : A (Personal) Primer on Critical Realism, Australasian Accounting Business and Finance Journal, 4(4), 2010, 3 -25. Available at: http: //ro. uow. edu. au/aabfj/vol 4/iss 4/2 [2 February 2013]

Methods are not paradigmatic • However, “research strategies and methods are not paradigmatic in themselves” (Twining 2010: 157), instead it is the way in which these methods are used to gather or elicit data and how the data is interpreted that indicates the explicit or implicit philosophical assumptions.

References • • Cohen, L, Manion, L & Morrison, K (2007) Research Methods in Education. 6 th edition. London: Routledge. Crichton, S & Kinash, S (2003) Virtual Ethnography: Interactive Interviewing Online as Method. Canadian Journal of Learning and Technology Volume 29(2) Spring. Avaialable online: http: //www. cjlt. ca/index. php/cjlt/article/view. Article/40/37 de. Waard, I. , Abajian, S. , Gallagher, M. S. , Hogue, R. , Keskin, N, Koutropoulos, A. & Rodriguez, O. C. (2011). Using m. Learning and MOOCs to understand chaos, emergence, and complexity in education. The International Review of Research in Open and Distance Learning, 12(7), 94 -115. Maxwell, J. A. (2008). Designing a qualitative study. In Bickman, L. & Rog, D. J. (2008) The Sage handbook of applied social research methods, (pp. 214 -253). London: Sage. Available online: http: //www. corwin. com/upm-data/23772_Ch 7. pdf (Last accessed 12 January 2013). Twining, P (2010) Educational information technology research methodology: Looking Back and moving forward. In A. Mc. Dougall, J. Murnane, A Jones & N. Reynolds, Researching IT in Education: Theory, Practice and Future Directions. London: Routledge. 153 -168. UCT Faculty of Humanities Guidelines for Research. Available online: http: //www. humanities. uct. ac. za/downloads/humanities/postgrad/gradschool/navthesis/start/res earch_proposal. pdf Usher, R (1996) A critique of the neglected epistemological assumptions of educational research in Scott, D & Usher, R (Eds) Understanding educational research. London: Routledge.

Written by Cheryl Hodgkinson-Williams cheryl. hodgkinson-williams@uct. ac. za in 2013 and adapted in 2014 This work is licensed under the Creative Commons Attribution-Share Alike 2. 5 South Africa License. To view a copy of this license, visit http: //creativecommons. org/licenses/bysa/2. 5/za/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.