Fast and reliable? Testing an alternative peer review system in the field

Palan, S. | Dreber, A. | Johannesson, M. | Mehmani, B. | Holzmeister, F.

Status: Writing

We study an alternative system for academic peer review—score-based review—in a preregistered field experiment at the Journal of Behavioral and Experimental Finance (JBEF). Within each manuscript, reviewers are randomly assigned to either standard review or score-based review. In score-based review, reviewers rate the manuscript’s publishability on a 0–10 scale, with optional comments to the authors limited to 1,000 characters, and rate five aspects of the manuscript on the same scale: the credibility of the results, the contribution to the literature, whether conclusions are supported by the data, the appropriateness of the methods, and the clarity of the writing. Data collection continued until our preregistered stopping rule was met, yielding 449 score-based reviews and 157 standard reviews for 146 manuscripts. The estimated inter-reviewer reliability of score-based review is 0.140 (95% CI [0.044, 0.246]), compared with 0.174 (95% CI [0.106, 0.241]) for historical standard reviews in JBEF. Score-based review increases the share of reviewers who accept the invitation and submit their review on time, shortens the time from invitation to submission, and reduces the time reviewers report spending on their review. Our results provide proof-of-concept evidence that score-based review can yield efficiency gains by lowering the burden on reviewers without adversely affecting reviewer reliability.