CODECHECK community workflow for codecheckers

See also the CODECHECK community workflow overview and discuss your issues. This guide has two main parts - a short community workflow list of steps, and an extended version which may be used as a reference. Are you checking a paper with CHECK-NL? See their [workflow for in-person events}(/nl/workflow/).

Codechecker tasks - short version

Before you start, an author created a pre-producible workflow: all data and code, plus a README file detailing the content, a manifest file detailing the output in the CODECHECK configuration file specification, and a license file; this is ideally bundled in a single repository or archive file and accompanied by a (pre-published) paper. The author or editor then created a new issue in the CODECHECK register to start a new community CODECHECK. Now it’s your turn.

Accept codechecking invitation
Create a project in an online repository, either by forking existing code repository or creating new project
[In case you forked] Create a directory codecheck where your files will go
Create a new document to write the CODECHECK certificate and start documenting the ongoing codecheck now with everything relevant that you do; the exact form of a codechecking procedure and form of documentation vary greatly, but there are some tools, such as an R package to automate some steps, including an Rmd template and word processor templates in .odt and .docx formats; all of that is optional, as long as the final certificate contains the mandatory information
Open the manuscript and follow the instructions to reproduce the workflow
During the CODECHECK, contact the authors in case of problems; keep in mind the general CODECHECK principles, especially “the codechecker records but does not fix” – unless it is trivial change; the authors can provide updated versions of code and documentation; however, the entire procedure should not be much more time-intensive than a normal peer review of a paper and not involve more than a few code revisions; the codechecker can always stop the process after a reasonable effort and document on the issue as not successfully reproduced.
Summarize the process and outcomes in your certificate and upload it as PDF to your project; add edited files and intemediary as well as output files as you see fit; the certifiate must at least contain the information on who checked what and how; the ambition should be to document for future self and other researchers; have a look at the available certificates.
Notify the editor about the completion

Codechecker tasks - extended version

Prerequisites

The codechecker in general is not there to fix things, but to document how far they got. The result is either is a completed CODECHECK, or a documentation why the check was found impossible to complete (see principle 1). Codecheckers should give feedback to the author and definetely allow workflows to be fixed. It is very hard to put a precise number on the amount of work you should put into a CODECHECK. You’re not expected to spend more time on a CODECHECK than you would on peer-reviewing a manuscript. You should take advantage of the fact that you can talk to the author and feel free to reach out early and often, when you think that issues can be resolved quickly. Depending on your familiarity with the used programming language and specific tools at hand, a very rough experience value could be 30 minutes of reading documentation, downloading and installing software, and another 30 minutes to write up the CODECHECK certificate. The time in between for running the actual workflow will vary greatly, from minutes to hours, and hopefully can be run in the background. In case the computations run longer than your regular working day, consider asking the author to prepare a subset of the workflow.

However, a codechecker may, for example out of personal interest in the research at hand, invest additional efforts. In any case, the overall goal is to leave the workflow repository in the same or better condition.

Some further tips:

Every CODECHECK is unique, just as the associated research article. If this guide feels like it doesn’t work for your case, you are likely still on the right track.
Reach out to fellow codecheckers in the public discussion forum if you face any problems.
A familiarity with make is helpful to provide an easy entrypoint and build up useful code snippets for your CODECHECKs, see https://book.the-turing-way.org/reproducible-research/make and https://swcarpentry.github.io/make-novice/reference

CODECHECK steps

Comment on the issue in the CODECHECK register repository to notify author and editor that you are accepting, and actually starting, the CODECHECK.
Depending on where the code and data are, create a new project or fork the author’s code repository, e.g., to the codecheckers organisation on GitHub.com or GitLab.com; supported repositories for a citable certificate: Zenodo, OSF, ResearchEquals. If there is no code repository, it is strongly recommended to create a new code repository for the source code of your check which is the basis for publishing the certificate in a repository; usethe naming scheme Lastname-YYYY using the family name of the corresponding author. Please take care to follow the terms and conditions of the workspace’s licenses; stop your CODECHECK if the licensing is unclear and contact the author to fix the documentation.
[In case your forked] Create a directory codecheck to not interfere with original files. This is the check directory. If you start a project from scratch, the project root is likely to be your check directory. You can use .codecheck if codecheck exists in submission for some reason. All files created by you and strictly related to the check should go into this directory. The exception are files that could be useful for the author and which you suggest the author may transfer into their own repository after the check (see “leave in a better condition” above).
1. [Optional] Start a Makefile or a script to re-run the workflow based on provided documentation, i.e., recreate all files listed in the manifest by runnign the command make codecheck. This Makefile target or script should run the whole or most suitable subset of the workflow and create the certificate. This may also be realised by putting in code chunks in a computational notebook, either independent of or as the basis for the CODECHECK certificate, e.g., R Markdown, Quarto, or Jupyter.
2. [Optional] More contents of the check directory.
  - Document the used computing environment, see CODECHECK bundle guide.
  - Create a notebook as the basis for the certificate (see below), e.g. codecheck.Rmd.
  - Make the repository Binder-ready; put all Binder-related files into .binder/ directory to separate them from the author’s content.
3. Write the CODECHECK certificate and save it as a PDF file named codecheck.pdf in the check directory. The certificate should cover at least WHO (you, but maybe with support from others?) checked WHAT, and HOW. There are not strict requirements on the form, but you’re welcome to use our word processor templates in .odt and .docx formats or the .Rmd template from our R package. Imagine the certificate as a reminder for future you so you will be to re-check the workflow in two years time - what help would you need to do that? There is no need to document every detailed step if that is not helpful for you. Include a full citation of the certificate, because your CODECHECK is a valuable contribution to science (see below for reserving the DOI). Take a look at the example CODECHECKs for existing certificates to serve as templates.
4. [Optional] Add more certificate sections depending on interest, time, and skills:
  - Do the generated outputs match the ones in the original manuscript? Are the differences relevant or not?
  - Are there any parts of the workflow where the author could improve the code?
  - How long did it take you to conduct the CHECK, and where did you struggle?
  - Are used pieces of software and data properly CITED and publicly DEPOSITED und suitable LICENSES?
  - Are open formats (text-based etc.) used for input and output data?
  - Is the data and software FAIR?
5. Add mandatory codechecker-contributed information to the codecheck.yml file, see spec
6. [If applicable] Wait for the article DOI.
7. [Recommended] Submit the unpublished draft of the Zenodo record for the certificate (see next step) for review to the CODECHECK community on Zenodo; this allows a CODECHECK editor to give feedback before you continue with the deposition and avoids direct updates of documents that just received a DOI.
Deposit the CODECHECK certificate on Zenodo using your own Zenodo account and following the community curation policy (which is replicated here for convenience):
- Reserve a DOI
  - Add the DOI to the codecheck.yml file.
  - Add the DOI to the codecheck.pdf CODECHECK certificate, which should include a full citation of itself.
- Files
  - codecheck.pdf (mandatory)
  - Optional: You can add any material to this record that you see fit, especially things that helped you with your reproduction, i.e., the CODECHECK bundle.
- Communities: Search for “codecheck” to add the record to the CODECHECK community on Zenodo.
- Authors: Add all codecheckers as authors.
- Title: "CODECHECK Certificate YYYY-NNN" (certificate number issued via the register ticket above, optionally you may add the submission’s title).
- License: Use Creative Commons Attribution 4.0 International if you only upload the CODECHECK certificate, otherwise use Other (Open) or Other (Attribution) and document the licensing of the different parts in an Additional notes field.
- Description: Copy the summary of the check here.
- Contributors: Add the original authors as contributors (see Zendo Metadata form section “Contributors (optional)”) with a suitable role (e.g., “Researcher”).
- Add a Relationship in metadata between the certificate and the original paper/submission.
  - Relation: Reviews (= the certificate reviews the article)
  - Identifier & Scheme: the identifier (ideally the article’s DOI)
  - Resource type: Publication (with clarification as fitting, e.g., Publication / Journal article)
- Add the certificate identifier as an Alternative identifier, e.g., https://zenodo.org/records/14576035
  - With schema URL using http://cdchck.science/register/certs/<CERT ID>
  - With schema Other using cdchck.science/register/certs/<CERT ID>
- Optionally, add extra metadata as you see fit (fields such as Version, Language, Keywords).
  - Connect the Zenodo record to the GitHub repository with a Relate/alternate identifier.
  - Connect the Zenodo record to the article/preprint with a Related/alternate identifier.
If possible, coordinate with the original author(s) to add a CODE WORKS badge to their repository, e.g., by sending a pull request on GitHub, a merge request on GitLab, or sending them an HTML snippet for their personal website. The badge should link directly to the Zenodo record via the DOI. The following snippet should work in Markdown:
```
 [![CODECHECK](https://codecheck.org.uk/img/codeworks-badge.svg)](https://doi.org/<DOI HERE>)
```