What is a publishable comparison for molecular simulation packages?
Simulation comparison papers describe attempts to perform the same calculation with a range of different simulation programs. Such comparison should be updated periodically with different versions of the same programs (or potentially additional programs).
Additional criteria considered in review of comparisons of molecular simulation programs
- Does the manuscript include developers or advanced users of all programs being compared to ensure that comparisons are fair?
- Are best practices being used in the simulation comparisons?
- Are the methods applied to systems which for which reliable independent data are available? Are the systems considered of sufficient complexity that readers will find the results convincing and generalizable?
- Have the authors addressed whether a naive user could expect similar performance and why or why not?
- Have the authors clearly identified both (a) the version(s) of the molecular simulations programs used in the comparison (e.g., through revision control tags, a standard version numbering system, or equivalent) and (b) any modifications made to the source code to facilitate the comparison?
- Do the authors supply run scripts, other supporting files, and analysis routines so that the comparison study can, in principle, be replicated? Does the article identify the version of the run scripts, etc.?
Revision schedule for comparisons of molecular simulation programs
- Articles should be updated when simulation packages have major revisions, or when the standard benchmarks change.
- Run scripts, supporting files, and analysis routines should be provided via version-controlled repositories, with appropriate updates.