Lessons Learned

What are “lesson learned” and reproducibility documents?

This article category will now include two general classes of article types: negative results and reproducibilty studies. These are defined in more detail below, but any article in the category must be of value to a substantial subset of the community.

First, we address “lessons learned”. Research inherently involves failure, but without documentation of negative results, investigators are doomed to repeat prior mistakes. The “Lessons Learned” category will address this gap in the literature in a topic-based way - e.g., “Lessons Learned in Potential of Mean Force Computation.” Each article should be broadly useful, with the potential to aid in the planning of a significant number of studies. Thus, results obtained in multiple systems and different conditions are desirable. Authors will be encouraged to add to an existing topic when possible and appropriate (and receive formal authorship credit, as dicsussed elsewhere).

Reproducibility studies are the second article type in the category. Such a study examines a published method or published application of a method, attempting to reproduce or correct the published material.
Acceptable studies may examine solely the physics or mathematics behind a method, the implementation of that method in a public code, or the application of a method to a particular system.
In the last case, a prior application may have correctly applied a correct method but without sufficient quality controls regarding sampling, for example. Reproducibility studies may include both positive and/or negative results, i.e., conflicting with or confirming prior findings.
If the results confirm prior findings, authors should provide a compelling argument for why the confirmation is of value to the community and worth publishing.
The complexity or novelty of a method or system might provide such a motivation.

Manuscript preparation instructions

Quite simply, a “Lessons Learned” or “Reproducibility” manuscript is a regular scientific article and hence should be prepared with the standard high level of care and documentation. The article should contain the usual sections: Introduction, including background, motivation and a brief discussion of related work; Methods/Theory, with all information required for a reader to reproduce the results, including software version numbers and links to any in-house files or scripts; Results; Discussion, highlighting the lessons learned and tips for avoiding future mistakes; and Conclusions. Data analysis should conform to best practices.

In cases where a reproducibility study conflicts with prior findings, authors are strongly encouraged to work with a LiveCoMS editor to engage authors of the original study to collaborate in clarifying problems or misunderstandings. The editors will make every effort to engage all parties diplomatically to yield a consensus joint paper. If a mutual effort proves impossible, the original authors will be invited to write a rebuttal.

Additional critera considered in the review of lessons learned articles

Reviewers will consider the following factors in reviewing “lessons learned” or “reproducibility” document.

• Is the motivation for the approach clearly described and reasonable given existing literature?
• Are the “lessons” provided likely to be of value to a significant number of researchers, based on demonstrated broad applicability?
• Is the procedure used fully described (and derived mathematically if appropriate) to enable a reader to reproduce the results?
• Are any needed supported files provided?
• Has data analysis been performed in conformance with best practices?
• For a reproducibility document, has the prior work been described fairly given publicly available material?

Revision schedule for lessons learned documents

• Once published, unless subsequently there are recognized flaws with or important updates to an already-published negative-result discussion, such article or article section may not require updating.
• However, for new authors wishing to add to an existing topic-appropriate “Lessons Learned” article, the new authors should work with authors of the existing article, in consultation with editors as needed.