CODECHECK community workflow onboarding for new codecheckers

New codecheckers usually have an onboarding call with a CODECHECK editor before their first check. This page lists the topics of that call in the order in which you meet them during a check, so you can prepare for the call and look things up afterwards. It complements, but does not replace, the workflow for codecheckers.

We like to compare a CODECHECK to a craft: you learn it by doing it, with a mentor at your side, more than by reading about it. So don’t worry if not everything below makes sense right away - it will after your first check. Your CODECHECK editor is your partner in your first check and your go-to person for any question - quite often, your check raises one nobody has asked before.

Onboarding topics

Getting to know each other

When and why did you register as a codechecker?
What is your background - are you a researcher, a developer, a research software engineer, a student, or a data steward? Which programming languages do you use?
Have you already been involved in a CODECHECK, as an author or as a codechecker? Your earlier certificates might already be in the register. If you have done reproducibility checks that are not in the register yet, let us know - we may be able to add them.

Roles

A CODECHECK involves three roles:

the author provides the workflow, i.e., code, data, and instructions (author guide);
the codechecker runs the workflow following the author’s instructions and writes the certificate (codechecker guide);
the CODECHECK editor assigns codecheckers, mentors them, gives feedback on the certificate, and looks after its publication and its integration into the register (editor guide); the editor does not run the workflow a second time.

The codechecker complements the scientific peer reviewers with a specific task and a different kind of expertise. If you don’t get many review requests yet, codechecking is also a good entry into peer review and scholarly communication.

What a CODECHECK is - and what it is not

A CODECHECK tests whether the instructions provided by the authors lead to the results the authors report. It is one reproduction, at one time, by one person. It does not assess whether the method is a good idea, the parameters are well chosen, or the code is good or correct - we do not review code quality, naming, or modularisation. The code can be a total mess - if the mess comes with a Makefile or a README that lets you run it with a couple of commands, that is good enough for a CODECHECK. For the same reason, it doesn’t matter much if large parts of the code were generated by AI.

Think of the regular safety inspection of a car: the inspector does not care whether the seats are comfortable or how fast the car goes, but checks that it starts and that the brakes work. In the same way, you take the authors’ documentation, turn the key, and see whether the engine starts.

This may sound like a low bar, but nobody knows yet whether the work you check will turn out to be important - if it does, the CODECHECK has preserved the state of the materials for a closer look later. Keep the CODECHECK principles in mind, especially the first one: codecheckers record but don’t investigate or fix. A CODECHECK is not about catching cheaters - authors are usually happy that somebody looks at their work and answer questions quickly and in detail.

The results do not have to match to the last pixel or the tenth decimal. Figures often differ in legend position, colours, or aspect ratio when the code runs on another computer. We trust a human comparing the original and the reproduced figure side by side more than an algorithm: what matters is that they show the same data, and that you record the differences you notice.

Partially successful checks are fine. You may recreate six of eight tables but not the other two, because they rely on sensitive data, or only the last step of a long pipeline, because training the model took the authors weeks. Some check is a lot better than no check at all, as long as the certificate summary says clearly what was checked and what wasn’t.

Where your check comes from

Many checks are proposed by authors in the community workflow, e.g., for a published preprint. Others come from a collaboration with a journal or a conference. For example, CODECHECK is one of the evaluation services of the Lifecycle Journal, where submissions are published as preprints, several services evaluate them publicly, and the authors decide which feedback to take up before they create the version of record.

For such checks, you follow the same community workflow and start from the public preprint. The authors may never have seen our author guide, so don’t be surprised by a sparse README; you and the CODECHECK editor pick up some of the authors’ tasks, and the editor takes care of the metadata that the venue needs.

Your first steps

Look at the paper’s landing page, read the abstract, and find the links to code and data. Then scroll through the paper to see which figures and tables are in the results - these are the outputs you will try to recreate. You don’t need to read more of the paper, unless the topic interests you.
Preserve what you check. Code and data may come from GitHub or GitLab, or from a repository such as Zenodo or OSF. A record with a DOI is already preserved, and you can’t fork it. A GitHub or GitLab repository is not, because it can be changed, renamed, or removed, so fork it into the codecheckers organisation on GitHub or the cdchck group on GitLab as a snapshot under our control. The CODECHECK editor adds you to the organisation; let them know if you haven’t forked into an organisation before.
Open the certificate template and use it as your notebook from the start, see what goes into the certificate.
Read the README and run the first commands. Your first email to the authors often goes out 10-20 minutes after you start, about a missing dependency, a hard-coded path, or files you can’t match to the paper, e.g., three versions of a script and a folder named “final”. If there are no step-by-step instructions at all, write to the authors right away and ask for them. This is often where a CODECHECK has its biggest effect: nobody asked the authors for these instructions before, and they are happy to write them.

Time horizons

There is no fixed time budget for a CODECHECK, but these numbers help to set expectations - spend more if you are excited about the work and can learn something useful for your own projects:

Active time is similar to a peer review, around two to three hours.
Computation time should fit into a working day: set up the environment in the morning, let the computations run while you take notes, and polish the certificate in the afternoon. A run overnight is fine, too, if you tested the workflow with a smaller configuration before; if a computation started in the morning sits at a few percent at lunchtime, abort it. If the workflow needs much longer, or a GPU or an HPC cluster, ask the authors for a subset of the data, a configuration that runs on a CPU, and the matching reference results. Recreating the results for the subset is a successful CODECHECK, as long as the certificate summary states this limitation.
The gray area of fixing things: if a fix takes you about as long as writing the email about it, e.g., a single hard-coded path or an undocumented dependency, go ahead and record the change. If it would take you as long as it would take the authors, it is their job, not yours. Fixing things is fun, so when you notice you have been at it for ten minutes, remind yourself to write to the authors instead.

See also the prerequisites in the codechecker guide.

Communication

You can talk directly to the authors, and you should do so early and often. Email is the lowest bar and the most reliable way to reach researchers. Some authors are happy to work in GitHub issues or pull requests, and you can link to those in the certificate, but we don’t require it - we want to impose as little technology on the authors as possible.

We manage the process in the issues of the CODECHECK register on GitHub, where the CODECHECK editor keeps the information about your check up to date. Post updates or questions there as much as it helps you, and mention the editor; if the authors are active on GitHub, you can mention them, too. Feel free to copy the CODECHECK editor into your emails to the authors, especially if you want them to step in. If you don’t hear back within a day or so, write an email - GitHub notifications do get lost.

What goes into the certificate

The certificate documents who checked what and how. Think of it as a reminder for your future self: what would you need to re-run the check in two years?

A summary of two or three sentences at the top, e.g., “The workflow was well prepared, I could run all commands with only minor warnings, and the results look the same.”
The steps you took, as briefly as possible: “I followed steps 1, 2, and 8 of the README and they worked” is enough when they did. Write down the things that made you think, tinker, or search, and copy the commands you had to adapt, e.g., because the authors used Windows and you use Linux, and the error messages you ran into.
The outcomes, i.e., the files from the manifest you recreated; ideally show the original and the reproduced figures or tables side by side.
Recommendations for the authors, e.g., to name or number the scripts so that the order of the steps is clear, to state the hardware they used and the expected execution time of each step, or to add a licence, because a repository without one can’t legally be reused. You are the first person looking at the workflow from the outside, so your recommendations are valuable - but write them down instead of implementing them, and don’t hold up the check over formatting or style.

Keep a feedback list and update it, don’t delete from it: when the authors fix an item, strike it through and add “fixed by the authors”. This short story of questions and fixes shows the positive impact of the check.

In the end, we need a PDF file; there are templates for word processors, R Markdown, Quarto, and Jupyter, but you don’t have to use them. The R Markdown, Quarto, and Jupyter templates read the metadata from the codecheck.yml file (see below), insert the outputs listed in its manifest, and check the file for you, so you enter the summary and the outputs only once. In a word processor document, nothing is linked, and you keep the summary and the list of outputs in sync with codecheck.yml by hand. If you write the certificate as a notebook, you can keep private notes in comments in the source file, and we publish the source file, too, as a more machine-readable version. For inspiration, look at the example certificates and at the good practices and common pitfalls we have collected from them.

The `codecheck.yml` file

The CODECHECK configuration file captures the metadata of a check in one place, especially identifiers, so that we can connect the certificate, the paper, the authors, and the codecheckers. Ideally, the authors create a first version of the file; in practice, the codechecker usually does. Looking up DOIs, titles, and ORCIDs is tedious, and AI coding assistants do it well (see AI in codechecking), but it is worth doing it by hand once to learn the mechanics. Validate the file with codecheck::validate_codecheck_yml_rules("codecheck.yml") from the codecheck R package, and let the CODECHECK editor know about error messages you don’t understand - the validation is work in progress. If you don’t use a notebook template, focus on the certificate first - the CODECHECK editor edits codecheck.yml files all the time and is happy to help.

Share the first draft of the certificate with the CODECHECK editor, who can give you feedback, and then with the authors. A draft is also a good way to ask the authors for help when you are stuck: send it with the steps you took and the error message, and decide after their reply what stays in. Authors often take up the feedback right away - out of ten items, nine may be fixed the next day. Updating the certificate is some extra work, but rewarding: the original research got better because you checked it.

Publication

The onboarding call often leaves this topic for later, when your certificate is done.

The certificate is published with a DOI, usually in the CODECHECK community on Zenodo; the codechecker guide describes the metadata to fill in. You can also publish on ResearchEquals; OSF is no longer an option, because it maintains the existing records but no new ones can be created. Don’t click “publish” on Zenodo too quickly: submit the draft record for review to the community instead, so that a CODECHECK editor can check the metadata with you before the DOI is final. Fixing a record after publication is a hassle. Files you created during the check, e.g., a small script that solved a problem, can go into the certificate deposit, too.

The codecheck.yml file is published in your fork of the authors’ repository. If the code is not on GitHub or GitLab, e.g., in a data repository that can’t be forked, the file can also be published together with the certificate, e.g., on Zenodo.

Afterwards, the CODECHECK editor adds the certificate to the register, which generates a landing page for each certificate and exports the information to Wikidata - there is nothing you need to do, see the editor guide for the details.

AI in codechecking

How to handle AI in CODECHECKs is work in progress, and we are gathering experience with it ourselves. You are welcome to use AI tools for the check and for writing the certificate, but you don’t have to - if you haven’t used coding assistants before, doing your first check “old school” is the best way to learn. So far, we have learned the following:

AI is a big help with metadata, e.g., to fill in the codecheck.yml file or to verify that all ORCIDs are correct.
AI needs instructions, too. An assistant can work through a README, run the commands, and take notes for you - but without a README, it can’t do much more than you can.
Coding agents tend to run off and fix things. Asked to look at a workflow, an agent may start writing a Dockerfile and refactoring the code without being told to. Give the agent the right context, e.g., our assistant guide for CODECHECKs, a ready-to-use context file that explains the role of the assistant and the CODECHECK principles: codecheckers record but don’t investigate or fix.
Don’t accept any change that you don’t understand. A human has to verify every change, and a human codechecker signs off on the certificate.
Run the commands yourself. Even if an assistant ran them, run them as well and see that the outcome is what the assistant reported.
Be transparent. Describe in the certificate how you used AI, and feel free to try things out - we are all still learning what works.
A well-prepared workflow is easier to check - for humans and for AI. Authors can use AI coding assistants to prepare their workflows, see using AI coding assistants in the author guide.

Relevant for institutional efforts

Some institutions offer CODECHECKs as a service for their researchers, see the institutional workflows. If you are going to conduct checks as part of such a service, the following topics are relevant, too.

There is no strict process. An institutional check can start in many ways, e.g., for a paper that is about to be submitted, for work in progress, or as part of a competition. Define a workflow that works for you and for the research you check. You don’t need to write it down right away; after the first few checks you will know which steps are worth documenting, and we are happy to give feedback.

Flexibility, but with transparency. Within an institution, authors may share data privately, sit next to you, or let you work on their computer because the data cannot leave it. You may also use institutional resources, such as an HPC cluster or special hardware, and you might go further than a volunteer codechecker, e.g., by actively helping the authors to fix their workflow. All of this is fine, as long as the certificate makes the setting transparent: state that the check was conducted as part of your institution’s service, which resources you used, and where the codechecker was not independent of the authors. Readers of the certificate can then judge for themselves.

Your own template and name. Institutions may use their own certificate template, put their own logo up front, and call the certificate differently, e.g., “reproducibility report” - about half of the certificates in the register come from the reproducibility reviews of the AGILE conference and carry that name. We care about the content.

Where to publish. The register’s automation, e.g., the preview images and the metadata on the landing pages, works best for certificates published on Zenodo. If your institution publishes certificates in a different repository, talk to the CODECHECK editor early, so that we can extend the register for it.

Register issues are public. The register issue for a check normally names the authors and links to their code or data. Talk to the authors before you open an issue, and until then, keep in touch with the CODECHECK editor by email. Over time, institutions can run the process themselves, i.e., open the register issue, reserve a certificate identifier, and submit the certificate to the Zenodo community, where the CODECHECK editor only reviews and accepts it.

A separate list of codecheckers. Institutional codecheckers are listed separately from the volunteers in the codecheckers repository, with a mapping of GitHub handle, ORCID, and institution so that we can connect the certificates correctly. We don’t contact institutional codecheckers with requests for volunteer checks. You are of course welcome to also sign up as a volunteer, and then be listed twice.

Your institution in the register. The register lists the certificates of each institution on its own page under venues. Once the first certificates are published, let the CODECHECK editor know how your institution and its collection of certificates should be named.