On 2017 Oct 29, Wichor Bramer commented:

Dear Hilda,

Thank you for your comments and your additional clarification via e-mail upon my request.

From your comments and clarification I understand that the nature and differences of the ESM and DAH searches might not be clear to you, and therefore also not to the other readers of the article. Let me start by explaining this in more detail.

The ESM searches were librarian-mediated searches created by me upon requests of researchers at our institute, Erasmus MC in Rotterdam, the Netherlands. The DAH searchers were librarian-mediated searches created by other information specialist upon requests of researchers at other academic hospitals in the Netherlands. The researchers published the outcomes of the searches as systematic reviews between January 2014 and June 2016. The topics of the reviews differ, as can be seen in table 1 in the article. Though we did not check that for all ESM and DAH reviews, the reviews were probably executed and reported according to the PRISMA standards. The review authors were responsible for the inclusion process, and the evaluation of the relevance of the retrieved articles.

This leads to an answer to your two questions:

1) The ESM and DAH reviews differ on topics. Therefore there is no direct comparison of two searches (at different times), which would require the exclusion of references retrieved based on a later search date. The ESM and DAH reviews are published in the same period, therefore we can conclude that the searches had been performed in the same period as well (though we did not analyze this).

2) As the both the ESM and the DAH searches are done at the requests of researchers, it was the responsibility of the review authors, all topic experts, to assess the relevance of the retrieved references. The reviews authors were not involved in the research, neither were they informed of the inclusion of their review in this research. Inclusion was only done after the review was published, therefore to the best of our knowledge, there was no bias in the inclusion process of the ESM and DAH reviews.

We do acknowledge that the weakness of this study is that a direct comparison is impossible, since there are different searches on different topics. We therefore have started with a follow up research in which we create three simultaneous searches on a topic for a systematic reviews by three independent information specialists. All results will then be combined for the researchers to review the retrieved references for relevance.

We will take your comments into account by the execution and writing of that research, as we have done with your earlier comments on a previous research article (a comparison of Medline, Embase and Google Scholar) which have influenced the data analysis and reporting of this research.

Once again thank you for your comments, and looking forward to discussing our research in the future with you.

Kind regards, Wichor

On 2017 Oct 28, Hilda Bastian commented:

It would be useful if the authors could provide detail on two key issues not described in the paper. The first is the method for excluding identified references that were published subsequent to the date of the original searches.

The second is how eligibility for study inclusion was assessed for the ESM group, and by whom. This is a key outcome measure, that is also highly susceptible to bias. A method for reducing this bias, for example, would be assessment by more than one assessor independent of those conducting the searches, blinded to the search strategy by which the study had been identified.

On 2017 Oct 28, Hilda Bastian commented:

It would be useful if the authors could provide detail on two key issues not described in the paper. The first is the method for excluding identified references that were published subsequent to the date of the original searches.

The second is how eligibility for study inclusion was assessed for the ESM group, and by whom. This is a key outcome measure, that is also highly susceptible to bias. A method for reducing this bias, for example, would be assessment by more than one assessor independent of those conducting the searches, blinded to the search strategy by which the study had been identified.

On 2017 Oct 29, Wichor Bramer commented:

Dear Hilda,

Thank you for your comments and your additional clarification via e-mail upon my request.

From your comments and clarification I understand that the nature and differences of the ESM and DAH searches might not be clear to you, and therefore also not to the other readers of the article. Let me start by explaining this in more detail.

The ESM searches were librarian-mediated searches created by me upon requests of researchers at our institute, Erasmus MC in Rotterdam, the Netherlands. The DAH searchers were librarian-mediated searches created by other information specialist upon requests of researchers at other academic hospitals in the Netherlands. The researchers published the outcomes of the searches as systematic reviews between January 2014 and June 2016. The topics of the reviews differ, as can be seen in table 1 in the article. Though we did not check that for all ESM and DAH reviews, the reviews were probably executed and reported according to the PRISMA standards. The review authors were responsible for the inclusion process, and the evaluation of the relevance of the retrieved articles.

This leads to an answer to your two questions:

1) The ESM and DAH reviews differ on topics. Therefore there is no direct comparison of two searches (at different times), which would require the exclusion of references retrieved based on a later search date. The ESM and DAH reviews are published in the same period, therefore we can conclude that the searches had been performed in the same period as well (though we did not analyze this).

2) As the both the ESM and the DAH searches are done at the requests of researchers, it was the responsibility of the review authors, all topic experts, to assess the relevance of the retrieved references. The reviews authors were not involved in the research, neither were they informed of the inclusion of their review in this research. Inclusion was only done after the review was published, therefore to the best of our knowledge, there was no bias in the inclusion process of the ESM and DAH reviews.

We do acknowledge that the weakness of this study is that a direct comparison is impossible, since there are different searches on different topics. We therefore have started with a follow up research in which we create three simultaneous searches on a topic for a systematic reviews by three independent information specialists. All results will then be combined for the researchers to review the retrieved references for relevance.

We will take your comments into account by the execution and writing of that research, as we have done with your earlier comments on a previous research article (a comparison of Medline, Embase and Google Scholar) which have influenced the data analysis and reporting of this research.

Once again thank you for your comments, and looking forward to discussing our research in the future with you.

Kind regards, Wichor