On 2015 Nov 19, Cecile Janssens commented:

It is important to distinguish the method from how it was investigated. We applied the method to an unselected sample of meta-analyses rather than limiting to high-quality meta-analyses, because we wanted to show when the method works and when it doesn’t. Our results show that the method does not work that well when the topic of the meta-analysis is heterogeneous, and when there is a need to include grey literature or articles from non-English languages, which is no surprise. The poor performance of the method for these meta-analyses should not be interpreted as a shortcoming of the method, but as guidance for when and when not to use the method.

We did not suggest that the method should be used standalone. We mention in the first paragraph of page 8 that “additional strategies like these can be used to complement our search method–either to find more eligible studies or to increase confidence in the results of the search method when no other studies are found.”

Regarding the specific meta-analyses: The method does not improve the efficiency of meta-analyses that were already efficient. We screened more than 100% of the articles in the studies of Kumar, Stevens and Shan because they screened 573, 400 and 243 articles, compared to 1013, 536 and 289 using our method. More than half of the studies in the meta-analysis of Kumar were case studies, which often have no or a limited number of citations, and several of the other studies were published in non-English language. The topics of both other studies were quite heterogeneous. Shan investigated quality of life, covering quality of life instruments and assessments of functional status, and Stevens summarized six studies on a post-operative drug interaction, which investigated six entirely different surgeries. As said, the method does not work well when topics are heterogeneous.

Finally, this is the first paper on this method, describing two pilot studies. More work is certainly needed. Given the high efficiency of the method and the impressive accuracy, we are most interested in a direct comparison of the method with keyword searching in Pubmed as a first search. The method may well be a more efficient start of literature searching than keyword searching.

On 2015 Nov 10, Wichor Bramer commented:

This article investagates the effectiveness of citation based searching for systematic reviews, because the gold standard keyword-based searching is inefficient. The authors propose a new method of co-citation and bibliographic coupling. Their conclusion is that Citation searching is a reasonably accurate method. But what is reasonably accurate? One should not look at the median or mean or overall recall, but at the minimum recall. A researcher is not performing 20 reviews for which the median recall is accurate enough, researchers decide on which method they want to use for their single review. They want to minimize the chances of missing relevant articles in their N=1 case.

What was the question that the authors tried to answer? Can it be used as a standalone method for indetifying included references in systematic reviews? The answer has te be 'No!'. The most complete method in study 1 (screening all co-citations) retrieves and appropriate median of 94% of all included references, but the minimum of 75% is unacceptable. And that method is only occasionally more efficient than the traditional method. The method that is more efficient than traditional (all co-cited > 1) retrieves sometimes only 50% of all included references. In study 2 the best method (indirect and direct citations) retrieves a minimum of 10% of all included references. In seven out of 42 studies 50% or less of all included references were retrieved, and sometimes not even by reducing the number of hits needed to screen, such as Kumar, where screening 177% of the traditional method resulted in a recall of only 31% of the originally included studies. In their discussion the authors give reasons why their method does not work for the 5 lowest scoring articles, concluding that if these are not taken into account 89% of all references is found. But that still does not explain Stevens [68] in which 138% of hits have to be screened to find 50% of all includes, and Shan [65], where 142% is needed to find 58%.

An interesting question would be whether the described method can add additional references that were not found with the traditional keyword-based approach or traditional citation screening? This cannot be concluded from these results, as the investigators only tried to find the articles that were already included (and thus had been found by the traditional methods).

The conclusion of the authors that: "Researchers … [have] … the advantage of being able to screen only half of the number of articles compared to keyword-base literature search …[which]… is an efficient strategy for finding key articles related to one or more “known” articles" might be correct, but it cannot replace traditional keyword based searching, but only complement it, but that was not investigated.

On 2015 Nov 10, Wichor Bramer commented:

This article investagates the effectiveness of citation based searching for systematic reviews, because the gold standard keyword-based searching is inefficient. The authors propose a new method of co-citation and bibliographic coupling. Their conclusion is that Citation searching is a reasonably accurate method. But what is reasonably accurate? One should not look at the median or mean or overall recall, but at the minimum recall. A researcher is not performing 20 reviews for which the median recall is accurate enough, researchers decide on which method they want to use for their single review. They want to minimize the chances of missing relevant articles in their N=1 case.

What was the question that the authors tried to answer? Can it be used as a standalone method for indetifying included references in systematic reviews? The answer has te be 'No!'. The most complete method in study 1 (screening all co-citations) retrieves and appropriate median of 94% of all included references, but the minimum of 75% is unacceptable. And that method is only occasionally more efficient than the traditional method. The method that is more efficient than traditional (all co-cited > 1) retrieves sometimes only 50% of all included references. In study 2 the best method (indirect and direct citations) retrieves a minimum of 10% of all included references. In seven out of 42 studies 50% or less of all included references were retrieved, and sometimes not even by reducing the number of hits needed to screen, such as Kumar, where screening 177% of the traditional method resulted in a recall of only 31% of the originally included studies. In their discussion the authors give reasons why their method does not work for the 5 lowest scoring articles, concluding that if these are not taken into account 89% of all references is found. But that still does not explain Stevens [68] in which 138% of hits have to be screened to find 50% of all includes, and Shan [65], where 142% is needed to find 58%.

An interesting question would be whether the described method can add additional references that were not found with the traditional keyword-based approach or traditional citation screening? This cannot be concluded from these results, as the investigators only tried to find the articles that were already included (and thus had been found by the traditional methods).

The conclusion of the authors that: "Researchers … [have] … the advantage of being able to screen only half of the number of articles compared to keyword-base literature search …[which]… is an efficient strategy for finding key articles related to one or more “known” articles" might be correct, but it cannot replace traditional keyword based searching, but only complement it, but that was not investigated.