Unintended Code Errors and Computational Reproducibility

Transparent and reproducible science requires that data and code underlying published results be made available to allow regeneration and verification of the results. But how easy is it to reproduce computational analyses using these materials? This technical report describes efforts to computationally reproduce results at one established repository as part of a routine internal pre-publication review. We conduct a structured content analysis of twenty-six “author reports'' generated through a standard review process and categorize identified issues. We find that four in five studies include code errors that prevent execution or produce inconsistent results. Although these errors are generally minor, we interpret our findings to indicate that review and correction can help resolve many of the errors and facilitate computational reproducibility.