CODE CHECK Processes

The CODE CHECK principles can be implemented in different ways. On this page we describe a few abstract alternatives developed during conversations with our partners to elaborate on the options and opportunities the principles provide. You can take a look at the default process for a concrete implementation of a CODE CHECK principles.

In our discussions with editors and publishers, we found that the point in time during the review process when a code check happens, the person who takes the role of codechecker, and the openness of the codechek, can vary. For example, the check may happen before peer review or after, the codechecker can be staff, a regular reviewer, or a member of a special reproducibility committee, the check may be open or blind, either matching or differing from the scientific review, and the results may be published or confidential.

Post review with extra role

  1. A manuscript with computations passes scientific peer-review and is assigned a codechecker with matching expertise (first priority: code, second priority: domain).
  2. The codechecker must understand the instructions and executes the workflow using the CODE CHECK infrastructure. They contact the manuscript author if needed and collaborate based on the CODE CHECK infrastructure.
  3. A CODE CHECK evaluates if the code and data produce a core output (figure, dataset) - it is not a result check or judgement of scientific merit.
  4. The codechecker makes the final call on the result within a week, publishes the checked code and data together with the environment in a public repository (with DOI), and puts their name on it. A CODE CHECK badge badge on the published article points to the repository with the checked files.

Pre-review with extra role on staff

  1. An editorial staff member/ECR with technical expertise checks if something is “mechanically wrong” before sending out a submission to review. This is worth the time together with other technical checks! The codechecker can reach out to the author to fix problems. It is also a sanity check for the author and a measure of seriousness for the journal.
  2. If a code check is successful, start the regular review.
  3. After acceptance, add CODE CHECK badge badge to the published article.

Parallel reproducibility reviewer

  1. A manuscript is submitted to a journal or conference.
  2. It is assigned a scientific reviewer with the suitable domain expertise, and a codechecker with matching technical expertise. The scientific review is conducted open/single-blind/double-blind, while the codechecker knows the author and can reach out directly through the best channel, e.g. email, GitHub/GitLab issues, or discussion forums.
  3. Both reviewer and codechecker follow the same deadline.
  4. Both the scientific review and the CODE CHECK result are submitted to the handling editor or programme committee member, who makes the decision on how to proceed.
  5. Independent of the submissions acceptance, the CODE CHECK findings are sent to the submitting author.

Regular reviewer

The review process is conducted as usual, but the editor does not only match the domain expertise but also technical skills. The reviewer conducts the CODE CHECK and submits the resulting certificate as part of his report. This variant requires an open peer review and a broad pool of reviewers.

Independent community pre-review

An author may also reach out to a colleague or find a Code Copilot on Twitter, who can give feedback but also create a CODE CHECK report according the a journal’s/conference’s implementation of the principles. This report certainly increases trust in a workflow and may help authors to include code and data in their submission. Such an independent report may even be taken up during review and take the place of a report usually created by a different process described on this page.

More?

You have ideas about how a CODE CHECK would work for your journal or conference? Let us know (see contact options below) !