Why Too Many Political Science Findings Cannot Be Trusted and What We Can Do About It: Assessing, Explaining and Improving the Credibility of Our Discipline’s Evidence Base

Wuttke, Alexander. 2018. “Why Too Many Political Science Findings Cannot Be Trusted and What We Can Do About It: Assessing, Explaining and Improving the Credibility of Our Discipline’s Evidence Base.” SocArXiv. September 11. doi:10.31235/osf.io/dq87w

Abstract

While other scientific disciplines are currently undergoing “credibility revolutions”, political science should also face the uncomfortable question of how much we can trust the findings published in the discipline’s journals and presses. Theoretically, this commentary uses the concept of social dilemmas to show how scientists ¬in academia’s current incentive system work against their self-interest if they prioritize research credibility. Empirically, this article conducts a comprehensive review of meta-scientific research with a focus on quantitative political science, demonstrating that threats to the credibility of political science findings are systematic and real. Building on a framework to assess the credibility of scientific literatures, the review shows that a large fraction of the published literature makes itself inaccessible for intersubjective credibility assessments, fails tests of analytical robustness or shows patterns of systematic deviation from true population estimates. Still, the review also highlights the notable progress the discipline has made in recent years to secure research credibility. This commentary urges further steps toward this direction and proposes concrete institutional measures to better align individual researcher rationality with the collective good of verifiable, robust and valid scientific results, such as dedicated verification procedures and the implementation of pre-registration and results-blind peer reviewing.