On 2015 May 13, Hilda Bastian commented:

This review contributed analysis that has been critical to the debate about editorial peer review. However, the date of the last search for studies was in June 2004. As the eligible literature was sparse, even a few good studies shifts the picture on some questions.

It seems to me likely that the number of relevant studies published in the last decade is now substantive. I include below studies that would be relevant to an update, particularly on the question of blinding/masking of authors' and/or peer reviewers' identities and affiliations, and open publication of peer review reports. A recent systematic on training is also relevant (Galipeau J, 2015).

There are some issues that I believe would be helpful for a new/updated review on editorial peer review to address:

(1) The scope of this review does not include potential sources of editorial/reviewer bias, in particular that related to racial background, gender, country of residence, and institutional prestige. The objective of the review is "to estimate the effect of processes in editorial peer review" and its key focus is the quality of published articles. However, the degree to which these types of biases are minimized in the scientific editorial process has important bearing on the fairness of the processes, as well as the overall quality of literature that may get the most attention in a field.

"Soundness of ethics" is one of the outcome measures of concern, including the avoidance of harm to research subjects. I believe avoidance of harm to authors, who are in a subordinate power relationship in the editorial process, is also a matter of ethics. Publishing mediocre papers from some groups preferentially over higher quality submissions from others, would patently undermine both the fairness and the value of the peer review process at a journal. That may have the power to influence career progress.

Systematic reviews should also point to key areas for further research. The lack of studies into methods to reduce editors’ biases is an important gap to point out, as so much of the literature is concerned primarily with peer reviewers’ bias.

(2) This review did not report on the methods used a priori to systematically assess the risk of bias of included studies, a critical omission in reporting the results of a systematic review (see Oxman AD, 1991, Moher D, 1999, and Liberati A, 2009). A wide variety of study types are eligible for inclusion, raising particular issues specific to them. And studies in this field have a range of specific potential biases. It would be helpful if the experience gained in this review led to an explicit set of criteria for assessing the risk of bias of included studies.

(3) Given the similarities in editorial processes and challenges across scientific disciplines, I believe a systematic review without this restriction would be more valuable, even if the search strategy may have more limitations.

Jefferson T, 2007 included studies with designs that were experimental and other comparative studies that included an attempt to control for confounding. I identified the following additional studies of blinding authors/peer reviewers or publishing peer review reports, that I think need to be considered by reviewers on these questions:

Biomedical science

• Laband DN, 1994
• Nylenna M, 1994 (English abstract, Swedish article Karlsson Y, 1995 - excluded by Jefferson et al)
• Isenberg SJ, 2009
• van Rooyen S, 2010
• Alam M, 2011
• Vinther S, 2012
• Kowalczuk, 2013 (Abstract only)

In addition, Hopewell S, 2014, while addressing another objective in relation to the impact of peer review, was conducted on published pre-publication peer reviews and subsequent manuscript versions.

Non-biomedical sciences

• Ferber MA, 1980
• Blank RM, 1991
• Lloyd ME, 1990
• Laband DN, 1994
• Budden AE, 2008
• Borsuk RM, 2009

I have written more about the evidence base on anonymity and openness in peer review in this blog post.

Finally, a trial of blinding critical appraisers of clinical trials in the context of systematic reviewing was included in this systematic review (Jadad AR, 1996). That is not the context of peer reviewing for publication of those trials. (As it only involves 7 reviewers, including it or not has little effect on overall conclusions on this body of evidence.)

(Disclosure: Part of my job includes working on PubMed Commons, which does not allow anonymous commenting.)

On 2015 May 13, Hilda Bastian commented:

This review contributed analysis that has been critical to the debate about editorial peer review. However, the date of the last search for studies was in June 2004. As the eligible literature was sparse, even a few good studies shifts the picture on some questions.

It seems to me likely that the number of relevant studies published in the last decade is now substantive. I include below studies that would be relevant to an update, particularly on the question of blinding/masking of authors' and/or peer reviewers' identities and affiliations, and open publication of peer review reports. A recent systematic on training is also relevant (Galipeau J, 2015).

There are some issues that I believe would be helpful for a new/updated review on editorial peer review to address:

(1) The scope of this review does not include potential sources of editorial/reviewer bias, in particular that related to racial background, gender, country of residence, and institutional prestige. The objective of the review is "to estimate the effect of processes in editorial peer review" and its key focus is the quality of published articles. However, the degree to which these types of biases are minimized in the scientific editorial process has important bearing on the fairness of the processes, as well as the overall quality of literature that may get the most attention in a field.

"Soundness of ethics" is one of the outcome measures of concern, including the avoidance of harm to research subjects. I believe avoidance of harm to authors, who are in a subordinate power relationship in the editorial process, is also a matter of ethics. Publishing mediocre papers from some groups preferentially over higher quality submissions from others, would patently undermine both the fairness and the value of the peer review process at a journal. That may have the power to influence career progress.

Systematic reviews should also point to key areas for further research. The lack of studies into methods to reduce editors’ biases is an important gap to point out, as so much of the literature is concerned primarily with peer reviewers’ bias.

(2) This review did not report on the methods used a priori to systematically assess the risk of bias of included studies, a critical omission in reporting the results of a systematic review (see Oxman AD, 1991, Moher D, 1999, and Liberati A, 2009). A wide variety of study types are eligible for inclusion, raising particular issues specific to them. And studies in this field have a range of specific potential biases. It would be helpful if the experience gained in this review led to an explicit set of criteria for assessing the risk of bias of included studies.

(3) Given the similarities in editorial processes and challenges across scientific disciplines, I believe a systematic review without this restriction would be more valuable, even if the search strategy may have more limitations.

Jefferson T, 2007 included studies with designs that were experimental and other comparative studies that included an attempt to control for confounding. I identified the following additional studies of blinding authors/peer reviewers or publishing peer review reports, that I think need to be considered by reviewers on these questions:

Biomedical science

• Laband DN, 1994
• Nylenna M, 1994 (English abstract, Swedish article Karlsson Y, 1995 - excluded by Jefferson et al)
• Isenberg SJ, 2009
• van Rooyen S, 2010
• Alam M, 2011
• Vinther S, 2012
• Kowalczuk, 2013 (Abstract only)

In addition, Hopewell S, 2014, while addressing another objective in relation to the impact of peer review, was conducted on published pre-publication peer reviews and subsequent manuscript versions.

Non-biomedical sciences

• Ferber MA, 1980
• Blank RM, 1991
• Lloyd ME, 1990
• Laband DN, 1994
• Budden AE, 2008
• Borsuk RM, 2009

I have written more about the evidence base on anonymity and openness in peer review in this blog post.

Finally, a trial of blinding critical appraisers of clinical trials in the context of systematic reviewing was included in this systematic review (Jadad AR, 1996). That is not the context of peer reviewing for publication of those trials. (As it only involves 7 reviewers, including it or not has little effect on overall conclusions on this body of evidence.)

(Disclosure: Part of my job includes working on PubMed Commons, which does not allow anonymous commenting.)