On 2016 Jul 28, Isabelle Boutron commented:

We would like to thank the Hilda Bastian for her interest in our work. We fully agree that our systematic review has some limitations and we acknowledged most of them in the paper. We also fully agree that the peer review system is a complex system and that we need different approaches to explore this system and that other study designs are also important to tackle this issue. We focused on randomised controlled trials as it provides a high level of evidence and one important result of this systematic review is the appalling lack of randomised controlled trials in this field. Despite huge human and financial investments in the peer review process, its essential role in biomedical research, only 7 RCTs have been published over the last 10 years. Yet, the conduct of randomised controlled trials in this field does not raise any important ethical or methodological concern. These results should be a call for action for editors to facilitate the conduct research in this field and give access to their data.

On 2016 Jun 12, Hilda Bastian commented:

This is a helpful broad brush update on randomized controlled trials (RCTs) of peer review interventions in biomedical journals (see older review Jefferson T, 2007 and my comment on that review). However, while the authors list several limitations, including restricting to RCTs and to biomedical journals, there are other limitations that, in turn, highlight the impact of those limitations.

One of those is the outcomes addressed here. The focus is explicitly on the peer reviews themselves and the process, and not wider outcomes, such as potential benefits and harms to peer reviewers or the impact of policies such as open review on journals (e.g. level of unwillingness to review).

In particular, the issue of harms brings us back to the limitation of looking only at RCTs, and limiting to the biomedical literature and a limited scope for databases searched. The authors provide no rationale for limiting the review to biomedical publications. Given that there are so few eligible studies within the scope this review, moving past this is essential. (In a blog post on anonymity, openness, and blinding of peer review in March 2015, in addition to the 11 RCTs identified in this systematic review, I identified a further 6 comparative studies, as well as other types of studies relevant to the questions around which known concerns exist.)

Peer reviewers are not just a means to an end: biases of peer reviewers can have a major impact on the careers of others, and at a minimum, specifically addressing gender, seniority, and institutional/country/language impact is critical to further work on this topic. A more contextual approach is needed to grapple with the complex ecosystem involved here.

A final point that is less likely to have had an impact, but is worth consideration by others working on this issue. Limiting the search strategy to the single term "peer review" may have an impact on searches, as terms such as open review and post-publication review become more widely used. Terms such as manuscript and editorial review, and peer reviewers could also be considered in constructing a search strategy in this area. (To identify the studies to which I refer above, Google Scholar and a wider range of search terms was necessary.)

(Disclosure: Part of my job includes working on PubMed Commons, which does not allow anonymous commenting.)

On 2016 Jun 12, Hilda Bastian commented:

This is a helpful broad brush update on randomized controlled trials (RCTs) of peer review interventions in biomedical journals (see older review Jefferson T, 2007 and my comment on that review). However, while the authors list several limitations, including restricting to RCTs and to biomedical journals, there are other limitations that, in turn, highlight the impact of those limitations.

One of those is the outcomes addressed here. The focus is explicitly on the peer reviews themselves and the process, and not wider outcomes, such as potential benefits and harms to peer reviewers or the impact of policies such as open review on journals (e.g. level of unwillingness to review).

In particular, the issue of harms brings us back to the limitation of looking only at RCTs, and limiting to the biomedical literature and a limited scope for databases searched. The authors provide no rationale for limiting the review to biomedical publications. Given that there are so few eligible studies within the scope this review, moving past this is essential. (In a blog post on anonymity, openness, and blinding of peer review in March 2015, in addition to the 11 RCTs identified in this systematic review, I identified a further 6 comparative studies, as well as other types of studies relevant to the questions around which known concerns exist.)

Peer reviewers are not just a means to an end: biases of peer reviewers can have a major impact on the careers of others, and at a minimum, specifically addressing gender, seniority, and institutional/country/language impact is critical to further work on this topic. A more contextual approach is needed to grapple with the complex ecosystem involved here.

A final point that is less likely to have had an impact, but is worth consideration by others working on this issue. Limiting the search strategy to the single term "peer review" may have an impact on searches, as terms such as open review and post-publication review become more widely used. Terms such as manuscript and editorial review, and peer reviewers could also be considered in constructing a search strategy in this area. (To identify the studies to which I refer above, Google Scholar and a wider range of search terms was necessary.)

(Disclosure: Part of my job includes working on PubMed Commons, which does not allow anonymous commenting.)

On 2016 Jul 28, Isabelle Boutron commented:

We would like to thank the Hilda Bastian for her interest in our work. We fully agree that our systematic review has some limitations and we acknowledged most of them in the paper. We also fully agree that the peer review system is a complex system and that we need different approaches to explore this system and that other study designs are also important to tackle this issue. We focused on randomised controlled trials as it provides a high level of evidence and one important result of this systematic review is the appalling lack of randomised controlled trials in this field. Despite huge human and financial investments in the peer review process, its essential role in biomedical research, only 7 RCTs have been published over the last 10 years. Yet, the conduct of randomised controlled trials in this field does not raise any important ethical or methodological concern. These results should be a call for action for editors to facilitate the conduct research in this field and give access to their data.