CODECHECK community workflow for CODECHECK editors
See also the CODECHECK community workflow overview and discuss your issues.
This page is for CODECHECK editors, i.e., the volunteers who look after community CODECHECKs in the register. Are you a journal editor, publisher, or conference organiser who wants to introduce CODECHECKs at your venue? Start with getting involved as a journal, publisher, or conference, the benefits of a CODECHECK in a review process and how it can be implemented, and the existing workflows at other venues. The tasks below show what an editor does in one concrete implementation and may serve as a template for your own. If you would like to become a CODECHECK editor, get in touch.
CODECHECK editor tasks
The main tasks for the CODECHECK editor are assigning and assisting (mentoring) codecheckers, and validating the publication of the certificate and all metadata.
If you create a new check issue on the register yourself, e.g., for a journal submission or on behalf of an author, the preferred way is the CODECHECK Launch Pad. It looks up the next available certificate identifier and sets the correct labels for the type of check.
When a new issue is assigned to a codecheck editor in the register, here are a few things you need to do.
- First checks
- Briefly check the submitted repository - does the workflow look at all codecheckable?
- Ensure that a preprint is linked or a real manuscript is published within the repository, or that a reference to a journal submission is provided to which you have contacts.
- Make sure the author has completed the required author tasks.
- Manage CODECHECK workflow
- Assign the next available Certificate identifier in
YYYY-NNNformat: it is one more than the highest identifier in the titles of all issues on the register, open and closed, which end with| YYYY-NNN. Do not go by the register itself, because it only lists finished checks, and identifiers of checks still in progress are missing from it. Add the identifier to the issue title and to the first comment of the issue, and add the labelid assigned- the Launch Pad counts only issues with this label when it looks up the next identifier. Issues created with the Launch Pad already carry the identifier, so check only that it is still the next free one. - Find a codechecker and invite them by @-mentioning in the register issue. Remove the
needs codecheckerlabel when you found one who accepted the task. Good job so far! - Onboarding calls with new codecheckers: if the codechecker has not done a CODECHECK before, offer them an onboarding call before they start; the onboarding page lists the topics to cover.
- Use the following labels to document the current state of the check:
work in progress,metadata pending - Support the codechecker as needed (send reminders, technical support, mediate between author and codechecker, et cetera); all communication should happen within the GitHub issue on the register unless there are sensitive information to share.
- Ensure that a reference to the certificate is/will be added to the manuscript.
- Assign the next available Certificate identifier in
- Certificate publication and register
- Wait until the article is published in a citable form.
- Ask the codechecker to update all required metadata in the
codecheck.ymland to update the certificate report (especially the final DOIs!) - Review the Zenodo record against the CODECHECK community curation policy, which is the most up-to-date list of the required metadata, e.g., the title
CODECHECK Certificate YYYY-NNN, only the codechecker(s) as authors (the original authors are credited via the citation of and relation to their work, not as authors or contributors), a CC-BY 4.0 licence for the certificate, aReviewsrelation to the checked paper, and the register landing page as an alternative identifier. Codecheckers submit the draft record for review to the CODECHECK community instead of publishing it, so the review happens while the DOI is not yet final: you find the request under the community’s requests (as a community curator), can comment on it to give feedback, and can curate the metadata and files of the draft directly. - Double-check the information in the metadata and the actual certificate, then accept the review request, which publishes the record with its own DOI. If a codechecker published the record directly instead, curate it as described in the next point.
- Alternatively, you may make edits to the metadata in the forked repository yourself, or handle edits via Zenodo yourself (metadata editing in CODECHECK community, or even at the stage of a shared draft of a Zenodo record). A published record belongs to the codechecker’s Zenodo account; to edit it, ask the codechecker for edit access (on the record page: Share → People → Add people → Can edit). Note that a new version of a Zenodo record starts from the metadata of the first version, so curation edits must be repeated on it.
- If the certificate is published on ResearchEquals instead, review everything before the codechecker publishes the module: ResearchEquals modules cannot be edited after publication, not even their metadata, and the only remedy is a new version with a new DOI. Check especially that the DOI of the checked paper is added as a reference of the module, see the codechecker guide for the full list.
- Add the CODECHECK as a new row at the end of the
register.csvfile, with the columnsCertificate,Repository,Type,Venue,Issue,Wikidata(see the register README for all details):Certificate: the identifier, e.g.,2026-001;Repository: where thecodecheck.ymlfile is published, e.g.,github::codecheckers/Piccolo-2020,gitlab::cdchck/<name>, orzenodo::<record ID>; append|<path>if the file is not in the repository root;Type:community,journal,conference, orinstitution;Venue: anamefromvenues.csv, e.g.,codecheckorpreprintfor community checks, or the journal’s short name;Issue: the number of the register issue;Wikidata: leave it empty, it is filled in when the certificate is exported to Wikidata.
- A new venue needs a row in
venues.csvfirst (at leastname,longname, andlabel), and a register issue label named after it, unless it is the only venue of its type. The labels of the issue must match the row, i.e., one type label equal toTypeand a venue label for theVenue. - Validate the new row with
codecheck::validate_register_rules()from thecodecheckR package, then commitregister.csvto themasterbranch with a commit message such asadd 2026-001, closes #N, which also closes the issue. A GitHub Action then renders the register automatically - check the new certificate page athttps://codecheck.org.uk/register/certs/<YYYY-NNN>/once it has finished. - “Archive” the repository clone in the codecheckers organisation on GitHub/the cdchck organisation on GitLab (instructions for GitHub, instructions for GitLab)
- Post two closing comments on the register issue, so that authors and codecheckers can find the results (see register#196 for an example): first the citation of the certificate, e.g.,
Family, I. (YYYY). CODECHECK Certificate YYYY-NNN. CODECHECK Community on Zenodo. https://doi.org/10.5281/zenodo.NNNNNNN, preceded by a short note on any curation changes you made to the record; then the link to the certificate landing page, e.g.,Here is the certificate landing page in the register: https://codecheck.org.uk/register/certs/YYYY-NNN/. Close the issue, if the commit did not already do so; the labels stay as they are. - If a community check on a preprint is eventually mentioned in the published peer-reviewed article, you can update the reference DOI to the final paper, see this example commit.
Automation with the Chekhov bot
Chekhov is a bot that takes over repeated steps of this workflow. It is controlled by mentioning it on the first line of a comment, one command per comment, and works on two repositories:
- on the register, where it validates the
codecheck.ymlearly, helps find and assign codecheckers, tracks where a check stands, opens the pull request adding the certificate to the register, and announces it; - on the codecheckers repository, where people register as codecheckers.
To see what it can do on an issue, comment
@chekhovbot commands
New codecheckers
When somebody opens a registration, the bot answers with what their row still needs: the handle against the issue’s author, the ORCID, a way to reach them besides GitHub (an email on the ORCID record when they wrote see ORCID page), the early-career columns, and whether they are listed or in a team already. When the registration is complete:
- Comment
@chekhovbot includeon the registration. The bot opens a pull request adding the row tocodecheckers.csvand asks you to review it - or whom the bot’s settings name, never the registrant - and it checks the organisation: somebody outside it is asked of the organisation’s owners, an invitation not yet accepted is said, and a member is put in the codecheckers team. - Review the pull request and merge it; the bot never merges, because a person confirms an ORCID and a contact address.
- Once the row is merged and the person is in the team, the bot posts the welcome and closes the issue - the next night, or at once when you comment
@chekhovbot includeagain.
Anything personal - a first name, an event in common, an answer to their question - is your own, separate comment: the bot posts fixed text only. @chekhovbot include open @handle opens a registration for somebody who has joined the organisation without filing one, and the bot does the same by itself when a codechecker it asked the owners to invite accepts. The nightly run reminds the owners of an invitation not sent and of a pull request waiting a week, and tells the person once about an invitation waiting for them.
The registration runbook describes what the bot checks, and the procedure by hand. Progress and feedback: register#209 for the register, chekhov#49 for registrations.
CODECHECK