CODECHECKAdvancing reproducible research by assessing core software quality indicators in CODECHECK
The Dutch Research Council (NWO) funds this project in the programme “Research on open science” from September 2026 to September 2028 (file number 110.110.017, grant).
Summary
Reproducibility is a cornerstone of open science, yet it remains unclear which elements of research software most directly influence it. This knowledge gap limits effective institutional support and policy development. The project addresses the gap by identifying the factors that cause issues during a CODECHECK, using CODECHECK as an analogue for reproducibility. CODECHECK is offered as a service by several institutions in the Netherlands, and their checks provide the data for the study. The identified factors, or failure modes, can serve as targets for interventions to improve reproducibility, such as training, support services, and policy measures.
Approach
- A structured template for documenting the issues encountered during a CODECHECK, used in all checks conducted during the project.
- Institutional CODECHECKs at Amsterdam UMC and the TU Delft Digital Competence Centre - with a goal of about 60 checks during the project - possibly supplemented by retrospective checks and by community codecheckers.
- A classification of the reported issues into the failure modes proposed by Samuel et al. (2026) - failure during dependency setup, unavailable software components, unavailable data, and errors in the code - extended with missing code, missing output, and outputs that do not match the manuscript.
- An analysis of whether reproducibility success can be predicted before a check, e.g., from a reproducibility readiness score, from the output of MetaCheck, or from the indicators of the Research Software Quality Kit (RSQKit).
Project team
- Merel C. Postema, Research Data Management, Amsterdam UMC
- Thomas Pronk, Research Data Management, Amsterdam UMC
- João Guimarães, Library, Delft University of Technology
- Yasel Quintero, ICT, Delft University of Technology
Outputs
- Project proposal: https://doi.org/10.5281/zenodo.22123712
- Poster presented at the TU Delft Open Science event, October 2026: https://doi.org/10.5281/zenodo.23476687
The certificates of the participating institutions are in the CODECHECK register: Amsterdam UMC and TU Delft DCC.
Funding
The project is supported by the NWO programme “Research on open science” from September 2026 to September 2028 through the project Advancing reproducible research by assessing core software quality indicators in CODECHECK at Amsterdam UMC and TU Delft (file number 110.110.017, grant).