Replication in experimental economics: Not good

Replication in experimental economics: A historical and quantitative approach focused on public good game experiments. Nicolas Vallois, Dorian Jullien. GREDEG Working Paper No. 2017-21. 2017. https://halshs.archives-ouvertes.fr/halshs-01651080/

Abstract : We propose a historical perspective on replication in experimental economics focused on public good games. Our intended contribution is twofold: in terms of method and in terms of object. Methodologically, we blend traditional qualitative history of economics with a less traditional quantitative approach using basic econometric tools to detect unnoticed historical patterns of replication. In terms of our object, we highlight a type of replication that we call " baseline replication ", which is not present in explicit methodological discussions, yet central in the specificity of experimental economics regarding replication in economics.

---
Conclusion
The goal of this paper was to propose a historical perspective on the practice of replication in experimental economics. Explicit methodological discussions of replication in economics highlight several meanings of \replication". However none of these meanings correspond to the most widely practiced type of replication in experimental economics, which we called "baseline replication": that similar baseline conditions in different experiments have to yield similar results.

We proposed a quantitative approach to highlight the historical dynamics of replication in public good games experiments. We found that results in baseline conditions converge over time, which suggests that baseline replications are effective over time, but they also decline over time, which suggests a counter-intuitive (i.e., non-stable) notion of replication. We prefer to interpret our results as a sign of the standardization of experimental practice over time which favors internal validity, rather than a sign of scientic progress towards a \true" baseline result in PG games. We hope that our blend of qualitative and quantitative approach to history of recent economics can motivate discussions with economists, both regarding our econometric method and our empirical results. In terms of further work, our results suggest that  finding a measure of standardization of experimental protocols can provide an interesting variable to study long-term patterns in experimental economics. Finally, baseline replication in other types of experiments than PG games could be studied.