On 2017 Oct 15, Erick H Turner commented:

Beyond the question of whether unpublished data were used, how were they used?

This study asks the important question, what proportion of systematic reviews searched for and made use of unpublished data? However, an important follow-up question remains to be addressed: Among those cases in which unpublished data was used, how was it used? Unpublished data can of course address study publication bias, ie. data from unpublished studies can be simply added to data obtained from the published literature. However, unpublished data can also address outcome reporting bias,[1-3] ie. a trial publication conveys that the intervention is safe and/or effective while unpublished data on the same trial tell a different story. For example, in a study of 74 industry-sponsored antidepressants trials,[4] in addition to 23 (31%) unpublished trials, we found 11 (15%) trials with outcome reporting bias. If we had corrected for the former while ignoring the latter, we would have obtained an effect size estimate that was still inflated. Returning to the current study,[5] an informative follow-up would be to look within the cohort of systematic reviews that made use of unpublished data and determine how many used it to verify the published results.

References:

1 Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365.

2 Chan A-W, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 2005;330:753. doi:10.1136/bmj.38356.424606.8F

3 Chan A-W, Hróbjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457–65. doi:10.1001/jama.291.20.2457

4 Turner EH, Matthews AM, Linardatos E, et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252–60. doi:10.1056/NEJMsa065779

5 Ziai H, Zhang R, Chan A-W, et al. Search for unpublished data by systematic reviewers: an audit. BMJ Open 2017;7:e017737. doi:10.1136/bmjopen-2017-017737

On 2017 Oct 15, Erick H Turner commented:

Beyond the question of whether unpublished data were used, how were they used?

This study asks the important question, what proportion of systematic reviews searched for and made use of unpublished data? However, an important follow-up question remains to be addressed: Among those cases in which unpublished data was used, how was it used? Unpublished data can of course address study publication bias, ie. data from unpublished studies can be simply added to data obtained from the published literature. However, unpublished data can also address outcome reporting bias,[1-3] ie. a trial publication conveys that the intervention is safe and/or effective while unpublished data on the same trial tell a different story. For example, in a study of 74 industry-sponsored antidepressants trials,[4] in addition to 23 (31%) unpublished trials, we found 11 (15%) trials with outcome reporting bias. If we had corrected for the former while ignoring the latter, we would have obtained an effect size estimate that was still inflated. Returning to the current study,[5] an informative follow-up would be to look within the cohort of systematic reviews that made use of unpublished data and determine how many used it to verify the published results.

References:

1 Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365.

2 Chan A-W, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 2005;330:753. doi:10.1136/bmj.38356.424606.8F

3 Chan A-W, Hróbjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457–65. doi:10.1001/jama.291.20.2457

4 Turner EH, Matthews AM, Linardatos E, et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252–60. doi:10.1056/NEJMsa065779

5 Ziai H, Zhang R, Chan A-W, et al. Search for unpublished data by systematic reviewers: an audit. BMJ Open 2017;7:e017737. doi:10.1136/bmjopen-2017-017737