Reviewer reliability and the extent of randomness in publication decisions

Holzmeister, F. | Johannesson, M. | Palan, S.

Status: Writing

In this project, we study the inter-rater reliability of reviewer recommendations using data from Elsevier’s Peer Review Workbench (PRW). We make three main contributions: (1) estimate the inter-rater reviewer reliability for various journals and fields; (2) estimate the heterogeneity in inter-rater reliability of reviewer recommendations across journals and fields based on random-effects meta-analyses; and (3) based on the estimated inter-rater reviewer reliability, simulate the “false positive rate” and the “false negative rate” in accepted manuscripts and gauge how they vary with the number of reviewers and the journal acceptance rate.