Qualitative Methods/ Case Studies

Qualitative Methods/ Case Studies

---

Are Case Studies Different?  A quantitative v. qualitative divide is emerging in political science.  KKV sparked some of this debate by forcefully presenting a statistical foundation for case study research.  CQRM, George & Bennett, and others are attempting to develop a distinctive qualitative methodology.

---

No…but  The purpose of case studies is often misunderstood in political science, and cases are often chosen for the wrong reasons.  Case studies are (or should be) simply quasi- experiments in which cases are intentional y selected for certain traits or characteristics.  Case selection is not only non-random, but purposive.  Threats to internal validity are often magnified because of intentional selection and (typical y) smal n.

---

Bottom Line  In principle, more cases are always better than fewer cases.  Use case studies primarily when it is too costly to collect data on more cases.  Nonetheless, a well-designed quasi- experiment with only a few cases is often preferable to a non-experiment with many cases.

---

Case Study Designs That are common in the political science literature.

---

Configurative/Heuristic/ “Building Block” Case Studies  In the process of generating theory, there is no substitute for knowing one or more cases well.  Contrary to George and Bennett, exploratory cases should (appear to) be modal or common events, not outliers.  These cases cannot, of course, then serve as tests of your theory.

---

Critical Case Studies  May be useful in disconfirming theories. We predict: if X, then Y, if not X, then not Y. We observe: X and not Y or not X and Y. Then theory may be disconfirmed.  Test of association. If failed, theory lacks conclusion validity.  How much weight to attach to a single case (or even a small handful of cases)?

---

Eckstein’s Crucial Case Studies  Most likely case studies: if theory holds anywhere, it should hold in this case.  Failure to support theory counts disproportionately against theory.  Least likely case studies: if theory works in this case, it should work in all cases.  Support for theory counts disproportionately in favor of theory.  If a case is more or less likely, theory contains unstated premises.

---

Most Similar Case Studies  John Stuart Mil s’ “method of difference.” Choose two or more cases that are as much alike as possible except for variable of interest.  A NEGD with a treatment and comparison group. N O X O N O O  Rather than control for possible group differences statistical y, intentional y pick cases with identical (or similar) values on the covariates that vary in the treatment.

---

 Can be a powerful research design when it is difficult or costly to study a large number of cases. When carried out correctly, can be internally valid. Do not need a large number of cases for a proper test.  Implicit foundation for “area studies.” Belief that regions share many similarities, and that these similarities are related to similar outcomes (weak test) and not related to dissimilar outcomes (stronger test).

---

How Similar is Similar?  In most similar designs, which covariates should you try to match?  Similar but irrelevant covariates do not add anything to the test. Likewise, dissimilar but irrelevant covariates do not detract from the test. Both reduce degrees of freedom.  Covariates that are related to both the treatment and outcome variables must be included whether similar or not – otherwise, omitted variables bias.

---

Problems of Inference  Must include covariates that correlate with X and Y.  If select cases with similar (relevant) covariates, likely to be similar in X as well. Indeed, since covariates and X are correlated, “naturally occurring” cases with similar covariates and different treatments may be outliers.  In “real world” cases, treatment effect is likely to be small, hard to identify, and uncertain.

---

Most Different Case Studies  Mills’ “method of agreement.” Choose cases that are as different as possible except for the variable of interest (i.e., all receive same treatment).  If X and Y occur despite different covariates, X and Y may be related.  Appears to be a NEGD, but actually a single group, non-experimental design. N O X O

---

Process-Tracing/ Causal Mechanisms/ Within-case Comparisons  George and Bennett advocate process- tracing as a valid scientific or causal test.  Outline an approach that seeks to move beyond “cause” to “causal mechanism.”  Claims it is superior: “process-tracing is the only observational means of moving beyond covariation alone as a source of causal inference” (p.224).

---

Process-Tracing in Notation  Single case, single treatment, many “post- tests” in a pre-specified sequence: O X O → O → O → O 1 2 3 4  Strength lies on the jointly predicted effects of a single treatment. All steps must be as predicted by hypothesis.  Often used to assess theories of decision- making or policy process.  Non-experiment; not a causal test.

---

Equifinality  Two or more causal processes that lead to the same result. If X, then Y If not X, then stil Y  George and Bennett claim that process-tracing can differentiate between multiple theories of Y.  Rather, should refine theory of Y. If X , then Y; If X , then Y; If X , then Y 1 2 3 If not X , X , X , then not Y 1 2 3  In this design, other Xs are covariates of X.

---

Most Similar Process-Tracing  Process-tracing is never, by itself, a causal test. Can be valid when combined with a most similar design: O X O → O → O → O 1 2 3 4 O O ~ O ~ O ~ O 1 2 3 4  Because of multiple opportunities to “fail,” constitutes a stronger test than a most similar design alone.  But requires a ful y specified theory linking each post-test observation to the next.

---

Potentially Va V lid Single Case Designs That are nonetheless seldom used in political science.

---

Non-equivalent Dependent Variables Design  Theory predicts treatment effect on one outcome variable but not on another similar outcome variable. O X O 1 1 O O 2 2  Other outcome serves as “comparison group.” Strength comes from “pattern matching” across different outcomes.

---

Interrupted Time Series  Single case observed over time, pre- and post- treatment (aka: regression point displacement). OOOXOOO  Analogous to a most similar design, with the case as its own “comparison group.”  The narrower the window around the treatment effect, the more powerful the test.  When to start and stop the series can be problematic. Need good estimate of functional form.

---

Problems Frequently Encountered in Case Study Research

---

Threats to Internal Validity  History: Many case studies are conducted over time. Need to consider other variables/events that may affect outcome.  Maturation: Again, change over time within the cases selected is likely to confound results. Need to model functional form.  Testing: Since case studies are typically given by nature, not likely to be a threat; but if pre- and post-tests are administered, testing threats may exist.

---

Threats to Validity, continued  Instrumentation: Often a problem. Cases selected because data is costly or difficult to obtain. Typically present verbal descriptions of the variables. Operationalization is even more important than in large-n designs.  Regression: Possible, especially if case selected is a prominent “outlier.”  Mortality: unlikely, since cases intentionally selected.

---

Selection Bias  In case studies, we intentionally select some (small) number of cases for our sample.  If we select cases from limited ranges of the outcome variable, we “truncate” that variable and introduce selection bias.  Truncating the outcome variable produces (on average) an underestimate of the treatment effect.

---

Example of Selection Bias  KKV give example of business school student who wants a high paid job and selects for his study sample only those graduates earning high salaries. He then relates salary to number of accounting courses.  By excluding graduates with low salaries, he paradoxical y underestimates the effect of additional accounting courses on income.

---

Underestimating the Value of  Accounting

---

Selection Bias in NEGD  KKV demonstrate selection bias in (as usual) a single group design.  In a NEGD, however, the effect is the same. Truncating the outcome variable (on average) underestimates the treatment effect.  Democracy (treatment) on life expectancy (outcome). Democracy is Polity score 6 and above. Covariate is GDP per capita.

---

If DV is truncated, estimate of treatment effect is smaller. Life Only Only expectancy countries countries in al with life with life countries expectancy expectancy in 1987 < 66 years in > 65 years in 1987 1987 Democracy 7.19*** 4.72* 3.02*** (Polity score (1.70) (2.25) (0.68) ≥ 6) N=110 N= 58 N= 52

---

Avoiding Selection Bias  Selection bias is found in both large- and small-n studies, but more likely when intentionally selecting a small number of cases.  If you select cases on the dependent variable, maximize variation.

---

Complex Causal Models  All of several conditions must be true for treatment effect to occur. If X1 + X2 + X3, then Y If not X1 or not X2 or not X3, then not Y  When faced with such a complex causal structure, many analysts adopt a “narrative” method of inquiry that examines a few cases in depth. This is common in historical-sociological or path dependent approaches.

---

Too Complex…?  In actuality, the number of cases needed for valid causal tests increases exponential y with the number of conditions.O[X , X , X ]O O[X ]O 1 2 3 1 O[X , X ]O O[ X ]O 1 2 2 O[ X , X ]O O[ X ]O 2 3 3 O[X , X ]O O O 1 3  With three necessary treatments, need at least eight groups.  Skocpol has six necessary conditions for revolution in three cases (plus several implicit comparisons).

---

Deterministic Theories and Case Study Designs  Some theories offer deterministic causal predictions: If X, then Y If not X, then not Y That is, a treatment must lead to a specified outcome.  A deterministic theory can be disconfirmed by a single “critical” case.

---

Probabilistic Theories and Case Study Designs  Other theories offer probabilistic causal predictions:  If X, then Y is more likely If X, then Y wil occur 60% of the time With a power ratio of 1:1, each of two countries have a 50% chance of winning a war; with a 3:1 ratio, the more powerful country has a 90% chance of victory.

---

 Outcomes may occur probabilistical y because they are natural y subject to chance.  Flaws in a research design may also make deterministic relationships appear as if they are probabilistic:  The same outcome can result from multiple treatments, and these are imperfectly control ed in the research design  Omitted variables bias  Measurement error To safeguard against “type II” errors, typical y test deterministic theories as if they are probabilistic theories.

---

 Due to random variation in a probabilistic setting, relying on a small number of cases can produce incorrect inferences.  Case studies are not well suited to testing probabilistic theories.  This is especially true when examining prominent outliers (e.g., World War I, the French Revolution, the U.S. Civil War).  Yet, these “deviant cases” often attract the most attention.

---

Conclusion  Case studies are intentionally selected observations.  Case studies can be used in effective quasi-experimental designs  two or more in most similar design (NEGD) one or more in NEDV one or more in interrupted time series.  Small-n makes all dimensions of research design more not less important.

---