Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.
Libraries
askus Ask us
 

Research Data Services

Framework for Evaluating the Reproducibility of a Data Set
Organization Yes / No / Maybe? (explain if necessary)
Are all files encapsulated within one directory?  

Is the sub-directory structure clear and easy to navigate?

  • Are the names of subdirectories self-explanatory?
  • Is the raw / input data separated from the derived data?
  • Is the data separated from the code?
  • Are any outputs (figures, tables) provided? Are they contained in their own subdirectory?
 
Are file names self-explanatory? If not, how could they be improved?  
Are there multiple versions of a file? If yes, are versions clearly enumerated?  

Is there a README file?

  • If yes, does it specify author contact information, file contents, directory overview, dependencies, etc? What other information could it provide to improve reproducibility?
 
Document Software Yes / No / Maybe? (explain if necessary)
Is the software environment specified?  
Are dependencies needed to run scripts specified clearly?  
Are relative paths used in scripts (vs. absolute paths)?  
Are all file conversion, data cleaning and analysis steps documented by scripts?  
Is the execution of all code automated by a master script?  
Are decisions behind data cleaning, analysis, and other scripts well documented within the code as annotations, or as a reproducible report (e.g. R markdown (*.Rmd))?  
Document Data Yes / No / Maybe? (explain if necessary)
Are the raw data provided? If only processed data are provided, is there sufficient description to understand transformations made to raw data?  
Are all data files necessary to rerun analyses provided? If not, are links to containing repositories specified?  
Are data provided in open file formats?  
Is sufficient documentation provided to understand the data? (e.g. data dictionary, code book)  
Licensing and Sharing Yes / No / Maybe? (explain if necessary)
Is a license specified for the software? (for e.g. either in a README file or a separate license text file?)  
Is a license specified for the data?  
Is the repository(ies) containing the data and code registered with a unique DOI?  
Are the repository(ies) and published article cross linked with metadata?  

References:
Broman, K. (n.d.). Initial steps toward reproducible research. Accessed September 05, 2019 from https://kbroman.org/steps2rr/

Clyburn-Sherin, A. (2019). Preparing data and code for reproducible publication using container technology. Workshop presented at the Research Data Access and Preservation Summit 2019, Coral Gables, FL. Slides accessed September 05, 2019 from http://bit.ly/rdap-workshop

Rokem, A., Marwick, B., & Staneva, V. (2018). Assessing reproducibility. In Kitzes, J., Turek D., & Deniz, F. (Eds.) The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences. Accessed September 05, 2019 from https://www.practicereproducibleresearch.org/core-chapters/2-assessment.html

Creative Commons License
This work by The University of Victoria Libraries is licensed under a Creative Commons Attribution 4.0 International License unless otherwise indicated when material has been used from other sources.