Open Peer Review Module (OPRM). Final Report

Abstract

Research productivity is increasing at an unprecedented rate. Technological innovations, a surge in available computing power, and the ease with which digital information is stored and communicated is helping researchers to cross experimentation boundaries, to increase data availability, and to facilitate the transfer of knowledge. As a result, traditional research is being transformed into a dynamic and globally interconnected effort where ideas, tools and results can be made instantly accessible to the entire academic community. Institutional and multidisciplinary open access repositories play a crucial role in this emerging landscape by enabling immediate accessibility to all kinds of research output. One important element still missing from open access repositories, however, is a quantitative assessment of the hosted research items that will facilitate the process of selecting the most relevant and distinguished content. Common currently available metrics, such as number of visits and downloads, do not reflect the quality of a research work, which can only be assessed directly by peers offering their expert opinion together with quantitative ratings based on specific criteria. To address this issue we developed an Open Peer Review Module (OPRM) to be installed on existing open access repositories and offered as an overlay service. Any digital research work hosted in a compliant repository can then be evaluated by an unlimited number of peers who offer not only a qualitative assessment in the form of text, but also quantitative measures that are used to build the reputation of the research work and its authors. Crucially, this evaluation system is open and transparent. By open we mean that the full text of the peer reviews are publicly available along with the original research work. By transparent we mean that the identity of the reviewers is disclosed to the authors and to the public. In our model, openness and transparency are two elemental aspects we consider necessary to address the issue of biased or non-expert opinions, which is inherent in the anonymous peer review model, characterized by the unaccountability of reviewers. Importantly, our open peer review module includes a reviewer reputation system based on the assessment of reviews themselves by other peer reviewers. This allows a sophisticated scaling of the importance of each review on the overall assessment of a research work, based on the reputation of the reviewer. The implementation of a peer review layer on top of institutional repositories could have the potential to transform the current academic publication landscape by introducing new scholarly workflows where a research item can be openly evaluated by the world’s experts right at the institutional repository of its authors, before being submitted to an academic journal. This workflow challenges the current practices of peer review research evaluation. In most cases, journals, acting as brands in a competitive market, foster academic competition for a limited number of publication slots, instead of promoting open scholarship and collaboration. The integration of peer review in repositories will enable direct and transparent academic collaboration between authors and reviewers. In addition, the use of the OPRM will produce novel metrics directly reflecting the perceived quality of a research work by expert peers, contrary to current available altmetrics that only indirectly account for quality through usage statistics.