Qualitative Methods/ Case Studies
Are Case Studies Different? A quantitative v. qualitative divide is emerging in political science. KKV sparked some of this debate by forcefully presenting a statistical foundation for case study research. CQRM, George & Bennett, and others are attempting to develop a distinctive qualitative methodology.
No…but The purpose of case studies is often misunderstood in political science, and cases are often chosen for the wrong reasons. Case studies are (or should be) simply quasi- experiments in which cases are intentional y selected for certain traits or characteristics. Case selection is not only non-random, but purposive. Threats to internal validity are often magnified because of intentional selection and (typical y) smal n.
Bottom Line In principle, more cases are always better than fewer cases. Use case studies primarily when it is too costly to collect data on more cases. Nonetheless, a well-designed quasi- experiment with only a few cases is often preferable to a non-experiment with many cases.
Case Study Designs That are common in the political science literature.
Configurative/Heuristic/ “Building Block” Case Studies In the process of generating theory, there is no substitute for knowing one or more cases well. Contrary to George and Bennett, exploratory cases should (appear to) be modal or common events, not outliers. These cases cannot, of course, then serve as tests of your theory.
Critical Case Studies May be useful in disconfirming theories. We predict: if X, then Y, if not X, then not Y. We observe: X and not Y or not X and Y. Then theory may be disconfirmed. Test of association. If failed, theory lacks conclusion validity. How much weight to attach to a single case (or even a small handful of cases)?
Eckstein’s Crucial Case Studies Most likely case studies: if theory holds anywhere, it should hold in this case. Failure to support theory counts disproportionately against theory. Least likely case studies: if theory works in this case, it should work in all cases. Support for theory counts disproportionately in favor of theory. If a case is more or less likely, theory contains unstated premises.
Most Similar Case Studies John Stuart Mil s’ “method of difference.” Choose two or more cases that are as much alike as possible except for variable of interest. A NEGD with a treatment and comparison group. N O X O N O O Rather than control for possible group differences statistical y, intentional y pick cases with identical (or similar) values on the covariates that vary in the treatment.
Can be a powerful research design when it is difficult or costly to study a large number of cases. When carried out correctly, can be internally valid. Do not need a large number of cases for a proper test. Implicit foundation for “area studies.” Belief that regions share many similarities, and that these similarities are related to similar outcomes (weak test) and not related to dissimilar outcomes (stronger test).
How Similar is Similar? In most similar designs, which covariates should you try to match? Similar but irrelevant covariates do not add anything to the test. Likewise, dissimilar but irrelevant covariates do not detract from the test. Both reduce degrees of freedom. Covariates that are related to both the treatment and outcome variables must be included whether similar or not – otherwise, omitted variables bias.
Problems of Inference Must include covariates that correlate with X and Y. If select cases with similar (relevant) covariates, likely to be similar in X as well. Indeed, since covariates and X are correlated, “naturally occurring” cases with similar covariates and different treatments may be outliers. In “real world” cases, treatment effect is likely to be small, hard to identify, and uncertain.
Most Different Case Studies Mills’ “method of agreement.” Choose cases that are as different as possible except for the variable of interest (i.e., all receive same treatment). If X and Y occur despite different covariates, X and Y may be related. Appears to be a NEGD, but actually a single group, non-experimental design. N O X O
Process-Tracing/ Causal Mechanisms/ Within-case Comparisons George and Bennett advocate process- tracing as a valid scientific or causal test. Outline an approach that seeks to move beyond “cause” to “causal mechanism.” Claims it is superior: “process-tracing is the only observational means of moving beyond covariation alone as a source of causal inference” (p.224).
Process-Tracing in Notation Single case, single treatment, many “post- tests” in a pre-specified sequence: O X O → O → O → O 1 2 3 4 Strength lies on the jointly predicted effects of a single treatment. All steps must be as predicted by hypothesis. Often used to assess theories of decision- making or policy process. Non-experiment; not a causal test.
Equifinality Two or more causal processes that lead to the same result. If X, then Y If not X, then stil Y George and Bennett claim that process-tracing can differentiate between multiple theories of Y. Rather, should refine theory of Y. If X , then Y; If X , then Y; If X , then Y 1 2 3 If not X , X , X , then not Y 1 2 3 In this design, other Xs are covariates of X.
Most Similar Process-Tracing Process-tracing is never, by itself, a causal test. Can be valid when combined with a most similar design: O X O → O → O → O 1 2 3 4 O O ~ O ~ O ~ O 1 2 3 4 Because of multiple opportunities to “fail,” constitutes a stronger test than a most similar design alone. But requires a ful y specified theory linking each post-test observation to the next.
Potentially Va V lid Single Case Designs That are nonetheless seldom used in political science.
Non-equivalent Dependent Variables Design Theory predicts treatment effect on one outcome variable but not on another similar outcome variable. O X O 1 1 O O 2 2 Other outcome serves as “comparison group.” Strength comes from “pattern matching” across different outcomes.
Interrupted Time Series Single case observed over time, pre- and post- treatment (aka: regression point displacement). OOOXOOO Analogous to a most similar design, with the case as its own “comparison group.” The narrower the window around the treatment effect, the more powerful the test. When to start and stop the series can be problematic. Need good estimate of functional form.
Problems Frequently Encountered in Case Study Research
Threats to Internal Validity History: Many case studies are conducted over time. Need to consider other variables/events that may affect outcome. Maturation: Again, change over time within the cases selected is likely to confound results. Need to model functional form. Testing: Since case studies are typically given by nature, not likely to be a threat; but if pre- and post-tests are administered, testing threats may exist.
Threats to Validity, continued Instrumentation: Often a problem. Cases selected because data is costly or difficult to obtain. Typically present verbal descriptions of the variables. Operationalization is even more important than in large-n designs. Regression: Possible, especially if case selected is a prominent “outlier.” Mortality: unlikely, since cases intentionally selected.
Selection Bias In case studies, we intentionally select some (small) number of cases for our sample. If we select cases from limited ranges of the outcome variable, we “truncate” that variable and introduce selection bias. Truncating the outcome variable produces (on average) an underestimate of the treatment effect.
Example of Selection Bias KKV give example of business school student who wants a high paid job and selects for his study sample only those graduates earning high salaries. He then relates salary to number of accounting courses. By excluding graduates with low salaries, he paradoxical y underestimates the effect of additional accounting courses on income.
Underestimating the Value of Accounting
Selection Bias in NEGD KKV demonstrate selection bias in (as usual) a single group design. In a NEGD, however, the effect is the same. Truncating the outcome variable (on average) underestimates the treatment effect. Democracy (treatment) on life expectancy (outcome). Democracy is Polity score 6 and above. Covariate is GDP per capita.
If DV is truncated, estimate of treatment effect is smaller. Life Only Only expectancy countries countries in al with life with life countries expectancy expectancy in 1987 < 66 years in > 65 years in 1987 1987 Democracy 7.19*** 4.72* 3.02*** (Polity score (1.70) (2.25) (0.68) ≥ 6) N=110 N= 58 N= 52
Avoiding Selection Bias Selection bias is found in both large- and small-n studies, but more likely when intentionally selecting a small number of cases. If you select cases on the dependent variable, maximize variation.
Complex Causal Models All of several conditions must be true for treatment effect to occur. If X1 + X2 + X3, then Y If not X1 or not X2 or not X3, then not Y When faced with such a complex causal structure, many analysts adopt a “narrative” method of inquiry that examines a few cases in depth. This is common in historical-sociological or path dependent approaches.
Too Complex…? In actuality, the number of cases needed for valid causal tests increases exponential y with the number of conditions.O[X , X , X ]O O[X ]O 1 2 3 1 O[X , X ]O O[ X ]O 1 2 2 O[ X , X ]O O[ X ]O 2 3 3 O[X , X ]O O O 1 3 With three necessary treatments, need at least eight groups. Skocpol has six necessary conditions for revolution in three cases (plus several implicit comparisons).
Deterministic Theories and Case Study Designs Some theories offer deterministic causal predictions: If X, then Y If not X, then not Y That is, a treatment must lead to a specified outcome. A deterministic theory can be disconfirmed by a single “critical” case.
Probabilistic Theories and Case Study Designs Other theories offer probabilistic causal predictions: If X, then Y is more likely If X, then Y wil occur 60% of the time With a power ratio of 1:1, each of two countries have a 50% chance of winning a war; with a 3:1 ratio, the more powerful country has a 90% chance of victory.
Outcomes may occur probabilistical y because they are natural y subject to chance. Flaws in a research design may also make deterministic relationships appear as if they are probabilistic: The same outcome can result from multiple treatments, and these are imperfectly control ed in the research design Omitted variables bias Measurement error To safeguard against “type II” errors, typical y test deterministic theories as if they are probabilistic theories.
Due to random variation in a probabilistic setting, relying on a small number of cases can produce incorrect inferences. Case studies are not well suited to testing probabilistic theories. This is especially true when examining prominent outliers (e.g., World War I, the French Revolution, the U.S. Civil War). Yet, these “deviant cases” often attract the most attention.
Conclusion Case studies are intentionally selected observations. Case studies can be used in effective quasi-experimental designs two or more in most similar design (NEGD) one or more in NEDV one or more in interrupted time series. Small-n makes all dimensions of research design more not less important.