CODECHECK community workflow for CHECK-NL

The following steps are required to complete a CODECHECK as part of a workshop organised by CHECK-NL. See also the CODECHECK community workflow overview for a broader approach. Before you start, note that every CODECHECK is unique, just as the associated research article. Reach out to fellow codecheckers in the public CODECHECK discussion forum if you face any problems, or use the internal team discussion if you need to consult in private. You are probably doing fine even if you digress from this documentation.

Steps

  1. Authors create a pre-producible workflow: all data and code, plus a readme file detailing the content, a manifest file detailing the output CODECHECK configuration file, and a license file; this is ideally bundled in a single repository or archive file and accompanied by a (pre-published) paper
  2. Authors send their request for a CODECHECK to project e-mail address codechecknl@gmail.com
  3. The CodecheckNL project team accepts the request for the workshop or advises to follow the normal community workflow (see above)
  4. During the workshop, codecheckers download materials or clone the a repository
  5. The workshop codecheckers create a new directory in their working environment where all new files go, and start documenting the ongoing codecheck; exact form of codechecking procedure and form of documentation vary greatly, but there are some tools, such as an R package to automate some steps, including a Rmd template; all of that is optional, as long as the final report contains the mandatory information
  6. During codecheck, the workshop codecheckers can ask the authors (if present at the workshop) in case of encountered problems, keeping in mind the general Codecheck philosophy (especially “the codechecker records but does not fix” – unless it is a very trivial bug like pathnames)
  7. The codecheckers summarize the process and outcome in a report and bundle it with all input and output files; this workshop codecheck bundle is then shared with the CodecheckNL project team via email or repository; the report should at least contain the information on who checked what and how; document for future self and other researchers; have a look at the available reports; most contain also optional information (compare CODECHECK community workflow guide)
  8. The CodecheckNL project team checks the bundle and report, and together with the workshop codecheckers, revise where necessary; once ready, either the CodecheckNL project team or a corresponding codechecker upload the file on Zenodo or OSF, and [optionally] adds a pull request to original repository for the Codecheck badge
  9. The CodecheckNL project team adds the new codecheck to the registry.