On 2016 Mar 04, Melissa Rethlefsen commented:

We attempted to replicate the authors’ Embase search strategy, as reported in the Appendix (pages 13-14). The authors, according to the PRISMA flow chart also located in the Appendix (page 37), retrieved 1,371 results for their Embase search strategy, prior to deduplication of records. The authors specifically mention that there was no language limit imposed on this search in the "search strategy and selection criteria" panel in the primary manuscript, but there are no other limits mentioned.

In case the phrase “for randomised controlled trials” in the "search strategy and selection criteria" panel indicated a limit to ‘randomized controlled trials’ as a study type (using the EMTREE term), we restricted to this term. There was also no indication of the dates of coverage of the version of Embase (via the Ovid platform) used. There are multiple possibilities of date ranges (1947-1979; 1980-1987; 1988-1995; 1947 to present; 1974 to present; 1980 to present; 1988 to present; 1989 to present; and 1996 to present). We attempted all of the different date ranges, until an entry date of February 1, 2014 to most closely mimic the published search. There was no exact search date mentioned, but the authors did note that the search went "up to January 2014"; we used a February 1, 2014 entry date to allow for a late January 2014 search, though it was not clear if the authors searched in January 2014, or limited to publications published prior to January 2014. Even with the assumed ‘randomized controlled trials’ EMTREE term applied, we could not replicate the search count for any year range available by default in any Embase via Ovid database option.

The search as presented by the authors retrieves over 50,000 records in Embase (up to an entry date of 2/1/14). With the ‘randomized controlled trials’ EMTREE term applied as a limit, it retrieves over 3,100 articles. Mimicking the smallest Embase database coverage at the time of the authors’ search (1996-present, limiting to an entry date before February 1, 2014), we still see nearly 3,000 articles. This is a 1,600 record difference between the authors’ published results and what their published search strategy retrieves, even with these assumptions applied.

This study highlights the need for more accurate and comprehensive reporting needed for search strategies in systematic reviews and other literature search-based research syntheses, and the need for better peer review of search strategies by information specialists/medical librarians. Though the searches in the Appendix are on face value replicable and high quality, on closer inspection, they do not in fact meet the reporting standards as outlined by PRISMA Statement items #7 and #8: “Describe all information sources in the search (e.g., databases with dates of coverage, contact with study authors to identify additional studies) and date last searched” and “Present the full electronic search strategy for at least one major database, including any limits used, such that it could be repeated.”

On 2016 Mar 04, Melissa Rethlefsen commented:

We attempted to replicate the authors’ Embase search strategy, as reported in the Appendix (pages 13-14). The authors, according to the PRISMA flow chart also located in the Appendix (page 37), retrieved 1,371 results for their Embase search strategy, prior to deduplication of records. The authors specifically mention that there was no language limit imposed on this search in the "search strategy and selection criteria" panel in the primary manuscript, but there are no other limits mentioned.

In case the phrase “for randomised controlled trials” in the "search strategy and selection criteria" panel indicated a limit to ‘randomized controlled trials’ as a study type (using the EMTREE term), we restricted to this term. There was also no indication of the dates of coverage of the version of Embase (via the Ovid platform) used. There are multiple possibilities of date ranges (1947-1979; 1980-1987; 1988-1995; 1947 to present; 1974 to present; 1980 to present; 1988 to present; 1989 to present; and 1996 to present). We attempted all of the different date ranges, until an entry date of February 1, 2014 to most closely mimic the published search. There was no exact search date mentioned, but the authors did note that the search went "up to January 2014"; we used a February 1, 2014 entry date to allow for a late January 2014 search, though it was not clear if the authors searched in January 2014, or limited to publications published prior to January 2014. Even with the assumed ‘randomized controlled trials’ EMTREE term applied, we could not replicate the search count for any year range available by default in any Embase via Ovid database option.

The search as presented by the authors retrieves over 50,000 records in Embase (up to an entry date of 2/1/14). With the ‘randomized controlled trials’ EMTREE term applied as a limit, it retrieves over 3,100 articles. Mimicking the smallest Embase database coverage at the time of the authors’ search (1996-present, limiting to an entry date before February 1, 2014), we still see nearly 3,000 articles. This is a 1,600 record difference between the authors’ published results and what their published search strategy retrieves, even with these assumptions applied.

This study highlights the need for more accurate and comprehensive reporting needed for search strategies in systematic reviews and other literature search-based research syntheses, and the need for better peer review of search strategies by information specialists/medical librarians. Though the searches in the Appendix are on face value replicable and high quality, on closer inspection, they do not in fact meet the reporting standards as outlined by PRISMA Statement items #7 and #8: “Describe all information sources in the search (e.g., databases with dates of coverage, contact with study authors to identify additional studies) and date last searched” and “Present the full electronic search strategy for at least one major database, including any limits used, such that it could be repeated.”