Abstract
In the academic world, peer review is one of the major processes in evaluating a scholars contribution. In this study, we are interested in quantifying the merits of different policies in a peer review process, such as single-blind review, double-blind review, and obtaining authors feedback. Currently, insufficient work has been undertaken to evaluate the benefits of different peer review policies. One of the major reasons for this situation is the inability to conduct any empirical study because data are presently unavailable. In this case, a computer simulation is one of the best ways to conduct a study. We perform a series of simulations to study the effects of different policies on a peer review process. In this study, we focus on the peer review process of a typical computer science conference. Our results point to the crucial role of program chairs in determining the quality and diversity of the articles to be accepted for publication. We demonstrate the importance of discussion among reviewers, suggest circumstances in which the double-blind review policy should be adopted, and question the credibility of the authors feedback mechanism. Finally, we stress that randomness plays an important role in the peer review process, and this role cannot be eliminated. Although our model may not capture every component of a peer review process, it covers some of the most essential elements. Thus, even the simulation results clearly cannot be taken as literal descriptions of an actual peer review process. However, we can at least still use them to identify alternative directions for future study.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3, 63–84.
Arrow, K. J. (1964). Social choice and individual values (2nd ed.). New York: Wiley.
Bachand, R. G., & Sawallis, P. P. (2003). Accuracy in the identification of scholarly and peer-reviewed journals and the peer-review process across disciplines. The Serials Librarian, 45(2), 39–59.
Barry, B. M., & Hardin, R. (1982). Rational man and irrational society?. Newbury Park, CA: Sage.
Billard, L. (1993). Comment. Statistical Science, 8(3), 320–322.
Blank, R. M. (1991). The effects of double-blind versus single-blind reviewing: Experimental evidence from the american economic review. American Economic Review, 81(5), 1041–1067.
Brams, S. J., & Fishburn, P. C. (1983). America’s unfair elections. Sciences, 23(6), 28–34.
Campanario, J. M. (1998). Peer review for journals at it stands today part 1. Science Communication, 19(3), 181–211.
Carland, J. A., Carland, J. W., & Aby, C. D. J. (1992). Proposed codification of ethicacy in the publication process. Journal of Business Ethics, 11, 95–104.
Ceci, S. J., & Peters, D. P. (1984). How blind is blind review? American Psychologist, 29, 1491–1494.
Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., & Waecklerle, J. F. (1998). Masking author identity in peer reviewat factors influence masking success? Journal of the American Medical Association, 280(3), 243–245.
Cox, D., Gleser, L., Perlman, M., Reid, N., & Roeder, K. (1993). Report of the ad hoc committee of double-blind refereeing. Statistical Science, 8(3), 310–317.
Dalton, M. S. (1995). Refereeing of scholarly works for primary publishing. Annual Review of Information Science and Technology, 30, 213–250.
Evans, A. T., McNutt, R. A., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the quality of peer review: A randomized trial. Clinical Research, 38(2), 1497–1525.
Fisher, M., Friedman, S. B., & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association, 272(2), 143–146.
Fletcher, R. H., & Fletcher, S. W. (1997). Evidence for the effectiveness of peer review. Science and Engineering Ethics, 3(1), 35–50.
Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., & Nuutinen, A. (2012). Panel discussion does not improve reliability of peer review for medical research grant applications. Journal of Clinical Epidemiology, 65(1), 47–52.
Franzini, L. R. (1987). Editors are not blind. American Psychologist, 42, 104.
Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawon, E. E. (1994). Effect of institutional prestige on reviewers recommendations and editorial decisions. Journal of the American Medical Association, 272(2), 137–138.
Genest, C. (1993). Comment. Statistical Science, 8(3), 323–327.
Goldberg, P. (1968). Are some women prejudiced against women? Transaction, 5, 28–30.
Harnad, S. (1982). Peer commentary on peer review. Behavioral and Brain Sciences, 5(2), 185–186.
Hill, S., & Provost, F. (2003). The myth of the double-blind review. SIGKDD Explorations, 2(5), 179–184.
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author identity improve peer review quality? Journal of the American Medical Association, 280(3), 240–242.
Korngreen, A. (2005). Peer-review system could gain from author feedback. Nature, 438, 282.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.
Link, A. M. (1998). Us and non-us submission: An analysis of review bias. Journal of the American Medical Association, 280(3), 246–247.
Madden, S., & DeWitt, D. (2006). Impact of double-blind reviewing on SIGMOD publication rates. SIGMOD Record, 35(2), 29–32.
McGiffert, M. (1988). Is justice blind? An inquiry into peer review. Scholarly Publishing, 20(1), 43–48.
Perlman, D. (1982). Reviewer bias: Do peters and ceci protest too much? Behavioral and Brain Sciences, 5(2), 231–232.
Rennie, D., & Flanagin, A. (1994). The second international congress on peer review in biomedical publication. Journal of the American Medical Association, 272(2), 91.
Resnik, D. B., & Elmore, S. A. (2015). Ensuring the quality, fairness, and integrity of journal peer review: A possible role of editors. Science and Engineering Ethics, 1–20. doi:10.1007/s11948-015-9625-5.
Snodgrass, R. T. (2003). Developments at TODS. SIGMOD Record, 32(4), 14–15.
Snodgrass, R. T. (2006). Single- versus double-blind reviewing: An analysis of the literature. SIGMOD Record, 35(3), 8–21.
Spier, R. E., & Poland, G. A. (2013). What is excellent science and how does it relate to what we publish in vaccine? Vaccine, 31(45), 5147–5148.
Tung, A. K. H. (2006). Impact of double blind reviewing on SIGMOD publication: A more detail analysis. SIGMOD Record, 35(3), 6–7.
Acknowledgments
This work was supported by the Youth Teacher Startup Fund of South China Normal University (No. 14KJ18), and was partially supported by the Natural Science Foundation of China (Grant No. 61402313) and the National High Technology Research and Development Program of China (863, No. 2013AA01A212).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhu, J., Fung, G., Wong, W.H. et al. Evaluating the Pros and Cons of Different Peer Review Policies via Simulation. Sci Eng Ethics 22, 1073–1094 (2016). https://doi.org/10.1007/s11948-015-9683-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11948-015-9683-8