Multiplicity eludes peer-review in COVID-19 research

Abstract

Multiplicity arises when data analysis involves multiple simultaneous tests, increasing the chance of spurious findings. It is a widespread problem in public health and environmental research, but many researchers, referees and editors do not consider it a problem that needs addressing. In this paper, we approach the multiplicity problem in two ways. On the one hand, we analyse recently published COVID-19 research as a case study. We show how a striking, potentially spurious finding may bypass peer-review and quickly spread through important media worldwide. A simple multiplicity analysis could have modulated the tone of certainty of the conclusions reached while maintaining the communication and discussion of the paper’s findings. On the other hand, we perform an exploratory analysis of the Web of Science Core Collection database for COVID-19 studies based on p-values. Of the 100 COVID-19 papers reviewed, 50% included over 34 simultaneous tests, with 10% including over 160 tests. Only one paper explicitly addressed the inflation error induced by multiplicity, suggesting a highly likely inclusion of spurious results that bypass the peer-review process. We argue that authors and reviewers of observational studies involving multiple testing, especially those with a large social impact, should pay special attention to the increased chance of false positives derived from the multiplicity effect. We propose that authors explicitly report the real multiplicity level involved in their studies, either visible or hidden, and clarify the limitations of their conclusions. This good practice should not restrict authors from discussing findings of interest on a per-test basis, regardless of multiplicity adjustments.