Bringing Rigor to Qualitative Data Qualitative Comparative Analysis

Bringing Rigor to Qualitative Data Qualitative Comparative Analysis Ann M. Doucette, Ph. D 25 October 2013 Midge Smith Center for Evaluation Effectiveness The George Washington University Ann M. Doucette, Ph. D

Development as a Complex Adaptive System (CAS) ● ● Non-linear, dynamic, adaptive, emerging Outcomes better understood through dynamic analysis ─ ─ Individual action changes the context for other individuals • Individuals include: beneficiaries (direct and indirect) other community members, organization members (NGOs), etc. • Diversity and Individuality: beneficiaries have unique and shared perspectives, values, norms Interconnections between individuals and systems • Systems embedded within systems § ─ ● ● ● Example: beneficiaries, within communities, within sectors, within countries, within national policies (not always in agreement) Study of the interaction of individuals and systems evolving behavior – adaptation to novelty and the unexpected Predictability is not expected at the detail level Broader-focused predictability is possible - patterns Order is natural and not imposed Ann M. Doucette, Ph. D 2

Causal Density - Intervention Complexity • Extent of complexity o Number of person-to-person transactions o Diversity of front-line implementing agent preferences o Influence of distracting conditions on implementing agents and program quality o Solution addressing concerns and challenges • Known – best practices, prior experience, etc. • Need for innovation, adaptation, modification • Implementation capacity o Level of fidelity that can potentially be reached given resources • Expectations o Adherence, integrity, quality of Potential impact understood within a framework of what can be program/intervention achieved – time to expectancy implementation – protocol Ann M. Doucette, Ph. D 3

Analytic Approach Qualitative Comparative Analysis (QCA) • Theory-based • Qualitative Comparative Analysis (QCA): analytic technique using Boolean logic as a method of quantitative analysis of macro (crosscase) phenomena yielded from a case study • Different paths lead to the outcome of interest (coverage – equifinality) o Counter to traditional statistics where factors are assumed to have the same incremental effect on outcome across cases (additivity) • Cases are sorted in terms of causal and outcome conditions o Small “N” is addressed through the maximization of comparison causal and outcome variables – cases either having or not having (“ 0” – “ 1”) the condition estimate of a dependent variable o Combination of “A” + “B” outcome “X” is obtained by adding together “A” + “B” + “C” outcome “X” the appropriately computed effects (independent variables) • Cluster analysis conducted to examine emerging shared properties (consistency - proportion of cases characterized by pattern) Ann M. Doucette, Ph. D 4

QCA Approach • Certain aspects of cases co-occur o Not deterministic (e. g. , avoid infection by using hand sanitizer, reduce corruption via transparency, etc. ) • Necessity: almost always present if outcome occurs, but does not produce the outcome (e. g. , micro-finance community economic development, HIV infection AIDS) o Outcome is a subset of condition o The frequency to which a condition occurs relative to outcome • Consistency: degree to which subset relationship (configuration) is linked to outcome conditions o Relevance of program aspects to outcomes o Participation in program activities to awareness building • Sufficiency: causal complexity – the degree to which a condition is present relative to other conditions and the outcome o Many solutions are possible – several sufficiency conditions o Use of theory is helpful Ann M. Doucette, Ph. D 5

Necessity – Sufficiency Example: Addressing Corruption • Outcome: decreased corruption • Conditions o “A” Transparency o “B” Enforcement of sanctions o “C” Written procedures • Path 1: A + B sufficient in leading to decreased corruption • Path 2: A + C sufficient in leading to decreased corruption • A is a necessary condition o Linked with both “B” and “C” o It is not sufficient as it does not yield the outcome in the absence of “B” and/or “C” Ann M. Doucette, Ph. D 6

Configuration Sets • Crisp (cs. QCA) o Each case is assigned membership • 1 = yes - present • 0 = no – absent • Fuzzy (fs. QCA) o Each case is assigned the degree to which the case fits • • • 1. 00 = fully in 0. 80 = mostly in 0. 60 = more in than out 0. 40 = more out than in 0. 20 = mostly out 0. 00 = fully out • 1. 00 = fully in • . 75 = more in than out • . 50 = neither in nor out • . 25 = more out than in • 0. 00 = fully out Ann M. Doucette, Ph. D 7

Truth Tables • Rows: logical combination of independent variables (input, suspected causes) o 0 = does not occur o 1 = occurs • Each row is assigned an output value in terms of the dependent variable o 0 = does not occur o 1 = occurs • Each row represents multiple cases (summary) having certain combinations • Combinations are reduced, representing sufficient causal configurations Ann M. Doucette, Ph. D 8

Example: Cases (Crisp Set) Ann M. Doucette, Ph. D 9

Looking for Matches Ann M. Doucette, Ph. D 10

QCA - Example Case study: Examining imposed economic austerity measures* Truth Table Example • Four conditions were thought to be linked to the outcome (protest versus no protest) yielding 16 possible configurations (4 x 4 – selected data shown) o Condition candidates established using qualitative/quantitative data o Conditions identified using cluster analysis • Consistency is shown for the first seven configurations • Estimation of the extent to which a configuration is associated with outcome (protest versus no protest) *Ragin, C. , 1987, The Comparative Method. Moving beyond qualitative and quantitative strategies, Berkeley/Los Angeles/London: Univ. of California Press Ann M. Doucette, Ph. D 11

QCA and Its Advantages • Addresses challenges of small-n studies o Number of comparisons - insufficient for probabilistic statistics o QCA maximizes number of comparisons within/across cases • Addresses complexity of issues found in development studies o Additivity not assumed – each causal condition has an independent association with the case outcome o Uniformity is not assumed – conditions may be positively or negatively associated with outcome in combination with other conditions o Outcome asymmetry is assumed – presence or absence of outcome may have different explanations • No assumption of linearity • Permanency of outcomes is not assumed • Parsimony • Case-oriented approach facilitates a form of counterfactual analysis – logical analysis Ann M. Doucette, Ph. D 12

Addressing the Counterfactual Exploration of plausible counterfactual configurations • # cases need to explore conditions geometrically increases according to function 2 k o 10 condition = 1024 cases to express all configurations o Many configurations would likely not be empirically supported counterfactual configurations • Possible configurations, but not observed in the data o Differentiation of strong versus weak counterfactual configurations • Theoretical consistency • Logical consistency (each outcome is explained by the configurations of conditions • Theory-informed simulations Ann M. Doucette, Ph. D 13

QCA - Disadvantages • Need for a strong theory – prior causal knowledge is essential • Small-n study o QCA does not seek to identify central tendencies, but to identify causal pathways linked to individuals cases • Emphasis on dichotomizing variables (crisp set) loss of information o Fuzzy set approach allows measurement at set intervals between 0 and 1 • More flexibility, but can be subjective and not standardized appropriately • Coding rules must be transparent • Measurement error can erode conclusion based on deterministic methods • Variable selection bias o Relevant variable selection is not particular to QCA, but to all analytic approaches • Average effects of conditions are not estimated • Outside mainstream science -- approach differs from traditional additive causation associated with probabilistic statistics • Does not allow for time dimension does not address process o Temporal sequencing is a separate exercise – could be incorporated in case selection Ann M. Doucette, Ph. D 14

Links Software http: //www. u. arizona. edu/~cragin/fs. QCA/software. shtml Websites http: //www. u. arizona. edu/~cragin/fs. QCA/ http: //poli. haifa. ac. il/~levi/method. html Ann M. Doucette, Ph. D 15

Ann Doucette, Ph. D. The George Washington University 2147 F Street NW, Suite B-01 Washington, DC 20052 Tele: 202. 994 -8112 Email: doucette@gwu. edu Ann M. Doucette, Ph. D