Benefits of CODECHECK as part of peer reivew

Learn more about CODECHECK in the published paper: Nüst, D. & Eglen, S. J. (2021). CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility. F1000Research, 10:253. https://doi.org/10.12688/f1000research.51738.2.

Why a CODECHECK adds value to the review process

A CODECHECK complements traditional peer review by addressing an aspect that reviewers are rarely able to cover: whether the computational results can actually be reproduced. The process is guided by five core principles that are designed to be practical and compatible with existing editorial workflows:

  1. Codecheckers record but do not fix. The codechecker follows the author’s instructions to run the code and records whether the outputs are reproduced. If problems arise, they are reported back to the author. This makes the process constructive: the conversation helps authors improve their documentation to a level that enables future reuse, without placing a burden on the codechecker to debug code.

  2. Communication between humans is key. Direct communication between the codechecker and the author ensures that minor technical issues (e.g., platform-specific dependencies) can be resolved efficiently. This keeps the process lightweight and avoids delays that would arise from routing all communication through the editorial office.

  3. Credit is given to codecheckers. Codecheckers receive formal credit for their work, comparable to a peer review contribution. You publish reviewer contributions to ORCID, please do so for codecheckers. The CODECHECK certificate is published with a DOI, and the codechecker is named. This makes the role attractive to early career researchers and research software engineers, broadening the pool of contributors to the review process.

  4. Workflows must be auditable. A successful CODECHECK means that the code was executed at least once using the provided instructions, that all code and data were available, and that the material could be investigated or extended in the future. The goal is practical auditability, not full automation.

  5. Open by default and transitional by disposition. Code and data associated with the CODECHECK certificate are made openly available unless there are strong reasons to the contrary (e.g., sensitive data). Importantly, this openness requirement applies to the computational materials, not the paper itself, so it is compatible with journals at any stage of the transition to Open Access.

A CODECHECK does not increase the overall workload for peer reviewers or the editorial team. The codechecker is a separate role, and the resulting certificate provides the editor with an additional, independent signal about the quality and transparency of the submission.

The seperation of the expertise around the data and code into a seperate role even opens up new resources and can simplify matching suitable reviewers. First, consider training early career researchers and students as CODECHECKERs, or invite/hire research software engineers as experts. These students can eventually grow into experienced (scientific) reviewers with high value to your journal community. We’d be interested to set up training and mentoring for such an additional pool of contributors together with your. Second, you can match methodological expertise independently of familiarity with tools, programming languages, or current availability. Your requirements on anonymity for the scientific review can even be handled seperately from the CODECHECK, enabling direct contacts for efficiency and keeping distance where each is more useful. Finally, you may place the CODECHECK flexibly in the process wherever it provides most value for your journal’s or conference’s review and publication model, from a staff-based smokescreen directly after submission to a post-publication additional merit.

Journals and conferences across several disciplines have successfully integrated CODECHECKs into their workflows - the the list of partners.