Mardy the Math Bunny

This is our colleague Mardy.

Mardy is an ordinary Math Bunny who loves carrots and doing mathematical research. Mardy is reading a very interesting paper titled:

A proof of the Riemann hypothesis by Bernhard Riemann's greatest fan

Even though she is a bit sceptical about the paper Mardy finds that the paper is really well written. This might contain an actual proof! But then she encounters the following sentence:

In order to complete the proof we performed the remainder of the calculations using a computer.

Excitedly, Mardy tries to check out the corresponding software and....

.

Pick your favourite reproducibility issue!

  • The paper does not contain a link to the code at all.
  • The paper contains a link, but: ERROR 404, Web page does not exist. (Which usually means that the repository is private or the author left academia.)
  • It is unclear which files in the repository correspond to which parts of the paper. You'll have even more fun if the repository is gigantic and unstructured.
  • The code is dependent on an outdated package and no longer works.
  • The code has no annotation, bad variable naming and is completely incomprehensible.
  • The software is impossible to install as the installation instructions are incomplete.
  • The software has no license, which means you're legally not allowed to reuse it.
  • No documentation.
  • No examples.
  • The implemented algorithm is claimed to be faster than previous algorithms. But it is not possible to check this as:
    • No bench-marking tools are provided to compare them and
    • The system specs used for comparing aren't given either.
  • The code runs, but the output is not what the authors claim it should be.
  • The code is dependant on other people's packages, but those packages aren't referenced or cited, so you need to google them yourself.
  • The code is in a repository, but saved in an Excel sheet, a Word document or a pdf.
  • The repository contains a dataset, but it isn't written down anywhere what the data represents. A list of numbers without meaning is just a list of numbers.

Why is this the case?

It is not really a surprise that the software is in this state. Computers have rapidly become more and more important in science, but we haven't quite adapted to using them properly yet. This is reflected by the following issues:

  • The focus of most publications is the paper.
  • The quality and correctness of code is sometimes given less attention as everyone wants to publish as quickly as possible.
  • Writing good mathematical software does not always get the acknowledgement it deserves.
  • Writing software properly is difficult and a skill we aren't really being taught.
  • Citation and acknowledgement of software is difficult and often done improperly.

Focus of a software review

The main goal of a software review should be to ensure that the code is reproducible. When doing a review one should check:

  • If the code is available on a long lasting repository.
  • If the code comes with a license.
  • If the repository comes with a Readme that explains the structure of the repo, what the code does, and what files do what.
  • If the repository has installation instructions and if following those instructions actually allows you to run the code.
  • Whether the code runs (after installing it) without any errors.
  • If the code actually does what is claimed in the paper or in accompanying examples. (This is not always possible of course. Especially, when computations take a long time. In that case small examples and tests verifying computed output can help convince others of the correctness of the results.)
  • If the code is written in such a way that other people will be able to read it.
  • Depending on the paper (e.g. when benchmarking is important or if the computations done take a lot of time), it is also checked whether the specs of the system used to do computations are given.
To see what a software review could look like click here