Quantitative Political Science Research is Greatly Underpowered; most methodologists greatly overestimate the statistical power of political science research

Quantitative Political Science Research is Greatly Underpowered. Vincent Arel-Bundock et al. Jul 5 2022. https://osf.io/hsgkp

Abstract: We analyze the statistical power of political science research by collating over 16,000 hypothesis tests from about 2,000 articles. Even with generous assumptions, the median analysis has about 10% power, and only about 1 in 10 tests have at least 80% power to detect the consensus effects reported in the literature. There is also substantial heterogeneity in tests across research areas, with some being characterized by high-power but most having very low power. To contextualize our findings, we survey political methodologists to assess their expectations about power levels. Most methodologists greatly overestimate the statistical power of political science research.

---

On average, political methodologists believe that 66% of studies have at least50% power, and 43% have at least 80% power. We demonstrate that these areoverly optimistic estimations. On average, experts overestimate the share ofstudies powered at the 50% level by 48 percentage points, and the share of studiespowered at the 80% level by 32 percentage points. Political science researchsuffers from low power and this problem is not sufficiently appreciated.