Policy Relevance of Increased Rigor in Qualitative Methods

  • Slides: 15
Download presentation
Policy Relevance of Increased Rigor in Qualitative Methods: The Case of Experimental Evaluations in

Policy Relevance of Increased Rigor in Qualitative Methods: The Case of Experimental Evaluations in Education Hannah Betesh Berkeley Policy Associates bpa

Goals of the session • Invite the qualitative research community to embrace transparency in

Goals of the session • Invite the qualitative research community to embrace transparency in analysis as one form of rigor (‘showing’ as well as telling our work) • Unpack our team-based analytical approach • Translate a theoretical big idea into the nuts and bolts of qualitative analysis

Policy Context • Emphasis on accountability under No Child Left Behind • Increased use

Policy Context • Emphasis on accountability under No Child Left Behind • Increased use of “medical model” in evaluation of educational programs. • Heightened focus on quantitative outcomes, to the exclusion of qualitative findings on implementation.

Research Context • Very recent emergence of qualitative tradition in policy research (Rist 1994)

Research Context • Very recent emergence of qualitative tradition in policy research (Rist 1994) • Qualitative research often impugned by many policy researchers as insufficiently valid and/or reliable (Howe & Eisenhart 1990) • There is, sometimes, a perceived lack of ‘rigor’ in qualitative educational research (Guba 1981)

Importance of Rigorous Qualitative Analysis in Policy Research • Mixed-methods research can generate richer

Importance of Rigorous Qualitative Analysis in Policy Research • Mixed-methods research can generate richer findings than either qualitative or quantitative analysis alone (Creswell and Plano Clark 2007) • Qualitative inquiry can help unpack implementation issues to inform quantitative findings • Capture perspectives of marginalized populations to elucidate differences in quantitative data (Tillman 2009)

Transparency as Rigor The ways in which researchers “account for and disclose their approach

Transparency as Rigor The ways in which researchers “account for and disclose their approach to all aspects of the research process are key to evaluating their work substantively and methodologically” (Anfara, Brown, and Mangione, 2002).

Facets of Rigorous Qualitative Research • Transparent and public coding scheme • Interim research

Facets of Rigorous Qualitative Research • Transparent and public coding scheme • Interim research products to track evolution of analysis • Explicit links to any quantitative component of the research

Components of Rigorous Analysis • Team-based data analysis • Use of software • Multi-stage

Components of Rigorous Analysis • Team-based data analysis • Use of software • Multi-stage analysis process • Laying a strong foundation

Components of Rigorous Analysis Team-based data analysis

Components of Rigorous Analysis Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Examining relationships Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Examining relationships Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Examining relationships Team-based data analysis Reliability-enhancing

Components of Rigorous Analysis Identifying salient conceptual categories Examining relationships Team-based data analysis Reliability-enhancing mechanisms

Components of Rigorous Analysis Salient conceptual categories Examining relationships Team-based data analysis Reliability-enhancing Going

Components of Rigorous Analysis Salient conceptual categories Examining relationships Team-based data analysis Reliability-enhancing Going beyond mechanisms code and retrieve

Identifying Salient Categories Researc h questio ns Data Di sc us sio n

Identifying Salient Categories Researc h questio ns Data Di sc us sio n

What Now? • Study description • Specific application of process • Software demonstration

What Now? • Study description • Specific application of process • Software demonstration