A Framework for Moving Beyond Computational Reproducibility: Lessons from Three Reproductions of Geographical Analyses of COVID-19
We have published a new preprint in the MetaArXiv collection of the OSF Metascience Community!
Kedron, Peter, Sarah Bardin, Joseph Holler, Joshua Gilman, Bryant Grady, Megan Seeley, Xin Wang, et al. 2023. “A Framework for Moving Beyond Computational Reproducibility: Lessons from Three Reproductions of Geographical Analyses of COVID-19.” MetaArXiv. May 29. doi:10.31222/osf.io/7jqtv.
In this paper, we make the case for an approach to reproduction studies that goes beyond assessing computational reproducibility to thoroughly evaluate the internal validity of prior studies. For a study to be computationally reproducible, all of software, data, and code should be accessible enough for independent researchers to regenerate the same results. However, computationally reproducible studies may still contain flaws.
Therefore, we proposed an approach to reproduction studies in which researcher decisions are all critically reviewed, especially when they cause unplanned or unanticipated deviations to the reproduction study analysis plan. We implemented our approach with students in geography methods courses, illustrated by findings from three reproduction studies of COVID-19. We enhanced the computational reproducibility of each of the three studies by publishing our efforts in open science research compendia. We also learned more about the research design and implementation of each research project than one would discover through a normal peer review or computational reproducibility audit. Through the reproduction studies, we highlighted and discussed methodological decisions in the prior studies with implications for the internal validity, especially geographic threats to validity. In many cases, we went beyond identifying and reviewing key decisions to also reanalyze the original study with improvements and robustness checks.
The three original studies and reproduction study compendia are:
| Original Study | Reproduction Compendium DOI |
|---|---|
| Mollalo et al 2020 | 10.17605/OSF.IO/E43KQ |
| Vijayan et al 2020 | 10.17605/OSF.IO/MY5DZ |
| Saffary et al 2020 | 10.17605/OSF.IO/QFKG4 |
Sarah Bardin presented this paper at the AAG in Denver.