The Craft of Interviewing in Realist Evaluation Dr

The Craft of Interviewing in Realist Evaluation Dr Ana Manzano a. manzano@leeds. ac. uk @Realism_Leeds School of Sociology & Social Policy University of Leeds- United Kingdom 03. 10. 14 European Evaluation Society 2014 -Dublin

Summary � Background � The study � The craft of interviewing

Background � In the last two decades, commissioners around the world have been funding realist evaluation studies and this (Pawson & Tilley, 1997) is now a well-established approach to programme evaluation. � Realist evaluation is a mixed-method strategy which proposes- among other methods of data collection- the use of theory-driven interviews to 'inspire/validate/falsify/ modify” hypothesis about how programmes and interventions work. � “Theorizing the interview” (Pawson, 1996): the teacher-learner cycle.

• Pawson, R. and Tilley, N. (1997) Realistic Evaluation. London: Sage • Pawson, R. (2006) Evidence-based Policy. A Realist Perspective. London: Sage • Pawson, R (2013) The Science of Evaluation. A Realist Manifesto. London: Sage

The process of theory testing. . . Programmes are ‘theories’ and so evaluation becomes a process of ‘theory testing’. But to do so theories need to be articulated in a particular manner � Ingredients are Contexts, Mechanisms and Outcomes � � Realist Evaluation begins with a period of ‘theory elicitation’ using ◦ ◦ Programme documentation Interviews with key stakeholders Existing ideas about the family of programmes Existing social / psychological theory Data construction should be theory-driven (and not datadriven) � The subject matter of the interview is the researcher’s theory, The interview will confirm, falsify, modify, refine the researcher’s theory. � What theory? ? A realist theory of the Ms, Cs and Os which define how programmes work. �

The Context of the Study � Fake handbags- Spin –off the Realist Evaluation Workshop paper (Pawson & Manzano-Santaella, 2012) � The need for standards? � The purpose of advancing realist methods � The need to do evaluation on evaluations � OBJECTIVE: To explore the empirical application of the realist interview

Methodology � 1) What are the research methods used in RE studies? (2) If interviews were used, how do these studies apply the teacher-learner cycle? and (3) What methodological problems - related to interviewing- do these papers raise? � Literature review of realist evaluations in applied health � Studies published between years 2004 -2013. � A total of 40 papers were found and of those, 32 included interviews as one of the methods of data collection

Interview Method in RE 32 used interviews 40 Realist evaluations 8 did not use interviews 5 only used interviews 27 used interviews in combination with other methods Most common model: documentary analysis (it could include quant data), ethnographic observation 8 combined with explicit quantitative methods

Findings � Interview is the most common method used in RE � It is normally used in combination with other methods � It is not clear how quantitative data is used in the documentary analysis (frequent method) � Sometimes RE is used as a “more prestigious” label to do qualitative research

Interview method seems an unproblematic: � Etheridge et al. 2013: “we used an interview guide derived from the main concepts in our theoretical framework” � Rycroft-Malone et al. 2013: “interviews were conducted using a topic guide based on the study programme theory and the contents of the evaluation framework”

And. . . � We found that, generally, little attention has been given to fieldwork processes with authors highlighting lack of methodological guidance at the analysis stage (Cs, Ms, . . . ) � Tendency to reflect/offer veredict on the realist approach � The process of data collection is not recognised as analysis. . . Still treated as lineal

BUT our practitioner knowledge (old fashioned “anecdotal evidence”) tells us that evaluators do find them problematic � Typical team: queries addressed to the Realism@Leeds How do you interview like a realist? How do you do realist topic guides? How do you present topic guides to ethics committees? How do I do iterative analysis when I am only doing one lot of interviews? ◦ I have prepared a table with CMOs, should I show it to the participants? ? ◦ Can you use NVIVO? ◦ Can I see your topic guides? ◦ ◦

The Craft “I’ll show you my theory if you show me yours” (Pawson & Tilley, 1997: 169)

Why? � � � � Theories are placed before the interviewee for them to comment on with a view to providing refinement The subject matter of the interview is the researcher’s theory/ The subject (stakeholder) is there to confirm, falsify and basically, refine theory. Teacher-learner relationship in which the medium exchange is the CMO configuration 2 way flow of information The realist approach assumes that interviewees accounts have a direct relationship with the real experiences in the world (beyond the interview situation). . . The research should have “Hypothesis seeking behaviour” (Pawson, 1996) during the interview In practice: ◦ Semi-structured interviews ◦ Theory-driven interviews

Who knows What? Subjects Practitioners Evaluators • More likely to be sensitized about Mechanisms (M) • Than to its contextual constraints (C) and outcome patterns (O) • Will have specific ideas on what is within the programme that works (M) • Likely to have experienced successes and failures (O) • Some awareness of people and places (C) for whom and in which the programme works • They carry theories into the encounter with programmes • Embryonic or well developed theories • Knowledge from similar and previous programmes • Knowledge from social science theory • Wisdom stronger on form than content

Some tips: � Preparing for the interview: ◦ Go to the interview with some specific theories to test. Your evidence based (observations, documents, other evaluations) guesses of “contexts”, “mechanisms”, and “outcomes” ◦ Not necessarily all of them � Phrasing questions: �The programme says that. . . But I can see that. . . �I have read some studies that say. . . �When they implemented this programme in York, they had this problem with. . . Have you seen that? �How did you do it before the programme was implemented?

Advantages &/or disadvantages? � Lots of work needs to be done before the interview � You have to start thinking CMOs at the first literature review � Then while you are collecting data via the other methods (observations, analysis of secondary quantitative data, etc. . . ) � Then at the interview. . . And in the next interviews � Then at the analysis (Common mistake is to start looking for CMOS here!). . . Then maybe interview again. . . � It is hard work!!!!!! Realist interviewing “involves a highly specific and carefully planned route march which goes between the qualitative and quantitative traditions “ (Pawson, 1996)

Warnings!! � Participants will not tell you: “Yes, well done! That is a mechanism! You got it!”. � The interview is only a step in the search for “nuggets of evidence”. . . � One of the key mechanisms in one of my projects was built up of one sentence said to me in a Staff Christmas party. . . This comment was not coded with NVIVO!

Example: � � � � � Grey literature Researcher: In this case I am interested on the fact that she (the patient) was self-funder and if it affected the way the discharge went and everything. . . The (“fine”) policy says that with self -funders, social services are only responsible for the assessment process. But what I have found out is really… the hospital doesn’t sort of remember that, because if the patient is selffunding, it is sort of…just… CM: Umm…Yes, they can just get on with it, go and do it. Observation Researcher: In the team meeting (that I observed), the physio was saying “maybe in three or four months the patient will be able to recover”. And then the physio asked you ‘Is she [the patient] very self-funder? ’ And you said, ‘Oh, yes, the son says she was quite over the limit’. So why do you think the physio was asking that? Why will it make a difference if the patient wasn’t self-funder? CM: Well, they shouldn’t, should it? But they often want them out, don’t they? Out of the way… Researcher: I still don’t understand what she meant! Do you want to translate it? CM: I think what she [the physio] is getting at is, if she wasn’t self-funding, they’ll be saying well… maybe a rehabilitation bed, either move her up to a ward or… Researcher: Ah, OK. Yes… But if she is a self-funder… CM: If they go self-funding, if they get her going there [in the nursing home] brilliant. If not, well, she can stay there. Which isn’t really fair to her! Because maybe she is not getting [rehabilitation] Researcher: In a different way, I’ve seen with some of other self-funders that I’ve done, not with you, with the other care managers, that the promotion of independence is a bit less than with social services patients. Because they [hospital staff] tend to say ‘The family is going to find a place and that place is ready. So they can go’. So in a way they, as you said, they know that they can get them out quicker. CM: Yes, they can. Interview with frontline practitioner Cross-case comparison