On 2018 Feb 01, Tim Mathes commented:

We thank Maria-Inti Metzendorf for her comments. We wrote that 2 full-texts were unavailable. By unavailable we mean that the full-text was not available anywhere. For sure, we tried to obtain all articles also via a document delivery services or by contacting the study authors. We assumed that it is not necessary to mention all sources that were used for obtaining full-texts. We chose the term not available, instead of not obtainable because one can never be 100% certain that a full-text don’t exist anywhere. The two mentioned cases were abstracts from the Cochrane Colloquium that were very likely never published as full-texts at the time point of the search. The abstracts did not provide sufficient information to allow an adequate data extraction. The studies were therefore excluded. Both authors are experienced in developing search strategies. Moreover, we tested the search strategy by checking whether the relevant publications known to us were identified before conducting the search (see article Methods section). In the comment no studies are mentioned that were missed by our search strategy. Therefore, it is only an assumption that the search strategy is not sensitive. We suppose this idea has come up because we only used “systematic”sb. In our experience this subset strategy is very sensitive. Our experience has also shown that this subset strategy is even more sensitive than some of the validated search filters for systematic reviews (e.g. health-evidence.ca filter: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-12-51). It might be always argued that sensitivity could be further increased. However, scientific manpower is not inexhaustible. So, in our opinion also precision of the search strategy is an important feature. However, we welcome suggestions on how to improve our search strategy, although we are confident that our search strategy is sufficient and did retrieve all relevant studies. We also appreciate comments or suggestions for studies we might have missed.

On 2018 Jan 24, Maria-Inti Metzendorf commented:

I would like to congratulate the authors on preparing a methodological review on the important topic of data extraction methods in systematic reviews (SR). They report that 11 potentially relevant articles were identfied in the screening process. After full-text evaluation 3 articles were excluded because they did not meet eligiblity criteria, and further 2 articles were excluded as the full-text was not available. Therefore, the authors could only base their analysis on 6 studies and conclude that the evidence base to inform the review´s question is sparse. Given this conclusion I wonder why the authors did not acquire the full-text of the above mentioned 2 articles (via document delivery services or by contacting the authors), as these could have further informed their review. In addition, the review would also have benefited from involving an information specialist who could have checked the search and improved its sensitivity.

On 2018 Jan 24, Maria-Inti Metzendorf commented:

I would like to congratulate the authors on preparing a methodological review on the important topic of data extraction methods in systematic reviews (SR). They report that 11 potentially relevant articles were identfied in the screening process. After full-text evaluation 3 articles were excluded because they did not meet eligiblity criteria, and further 2 articles were excluded as the full-text was not available. Therefore, the authors could only base their analysis on 6 studies and conclude that the evidence base to inform the review´s question is sparse. Given this conclusion I wonder why the authors did not acquire the full-text of the above mentioned 2 articles (via document delivery services or by contacting the authors), as these could have further informed their review. In addition, the review would also have benefited from involving an information specialist who could have checked the search and improved its sensitivity.

On 2018 Feb 01, Tim Mathes commented:

We thank Maria-Inti Metzendorf for her comments. We wrote that 2 full-texts were unavailable. By unavailable we mean that the full-text was not available anywhere. For sure, we tried to obtain all articles also via a document delivery services or by contacting the study authors. We assumed that it is not necessary to mention all sources that were used for obtaining full-texts. We chose the term not available, instead of not obtainable because one can never be 100% certain that a full-text don’t exist anywhere. The two mentioned cases were abstracts from the Cochrane Colloquium that were very likely never published as full-texts at the time point of the search. The abstracts did not provide sufficient information to allow an adequate data extraction. The studies were therefore excluded. Both authors are experienced in developing search strategies. Moreover, we tested the search strategy by checking whether the relevant publications known to us were identified before conducting the search (see article Methods section). In the comment no studies are mentioned that were missed by our search strategy. Therefore, it is only an assumption that the search strategy is not sensitive. We suppose this idea has come up because we only used “systematic”sb. In our experience this subset strategy is very sensitive. Our experience has also shown that this subset strategy is even more sensitive than some of the validated search filters for systematic reviews (e.g. health-evidence.ca filter: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-12-51). It might be always argued that sensitivity could be further increased. However, scientific manpower is not inexhaustible. So, in our opinion also precision of the search strategy is an important feature. However, we welcome suggestions on how to improve our search strategy, although we are confident that our search strategy is sufficient and did retrieve all relevant studies. We also appreciate comments or suggestions for studies we might have missed.