On 2016 May 17, Annika Hoyer commented:

With great interest we noticed this paper by Nikoloulopoulos. The author proposes an approach for the meta-analysis of diagnostic accuracy studies modelling random effects while using copulas. In his work, he compares his model to the copula approach presented by Kuss et al. [1], referred to henceforth as KHS model. We appreciate a lot Nikoloulopoulos referring to our work, but we feel there are some open questions.

The author shows in the appendix that the association parameter from the copula is estimated with large biases from the KHS model, and this is what we also saw in our simulation study. However, the association parameter is not the parameter of main interest which are the overall sensitivities and specificities. They were estimated well in the KHS model, and we considered the copula parameter more as a nuisance parameter. This was also pointed out by Nikoloulopoulos in his paper. As a consequence, we are thus surprised that the bad performance in terms of the association parameter led the author to the verdict that the KHS method is 'inefficient' and 'flawed' and should no longer be used. We do not agree here, because our simulation as well as your theoretical results do clearly show that the KHS estimates the parameters of actual interest very well. Just aside, we saw compromised results for the association parameter also for the GLMM model in our simulation.

Nikoloulopoulos also wrote that the KHS approximation can only be used if the 'number of observations in the respective study group of healthy and diseased probands is the same for each study'. This claim is done at least 3 times in the article. But, unfortunately, there is no proof or reference or at least an example which supports this statement. Without a mathematical proof, we think there could be a misunderstanding in the model. In our model, we assume beta-binomial distributions for the true positives and the true negatives of the i-th study. They were linked using a copula. This happens on the individual study level because we wanted to account for different study sizes. For estimating the meta-analytic parameters of interest we assume that the shape and scale parameters of the beta-binomial distributions as well as the copula parameter are the same across studies, so that the expectation values of the marginal distributions can be treated as the meta-analytic sensitivities and specificities. Of course, it is true that we used equal sample sizes in our simulation [1], however, we see no theoretical reason why different sample sizes should not work. In a recently accepted follow up paper on trivariate copulas [2] we used differing sample sizes in the simulation and we also saw a superior performance of the KHS model as compared to the GLMM. In a follow-up paper of Nikoloulopoulos [3], he repeats this issue with equal group sizes, but, unfortunately, did not answer our question [4,5] with respect to that point.

As the main advantage of the KHS over the GLMM model we see its robustness. Our SAS NLMIXED code for the copula models converged better than PQL estimation (SAS PROC GLIMMIX) and much better that Gauss-Hermite-Quadrature estimation for the GLMM model (SAS PROC NLMIXED). This was true for the original bivariate KHS model, but also for the recent trivariate update. This is certainly to be expected because fitting the KHS model reduces essentially to the fit of a bivariate distribution, but without the complicated computations or approximations for the random effects as it is required for the GLMM and the model of Nikoloulopoulos given here. Numerical problems are also frequently observed if one uses the already existing methods for copula models with non-normal random effects from Liu and Yu [6]. It would be thus very interesting to learn how the authors’ model performs in terms of robustness.

Annika Hoyer, Oliver Kuss

References

[1] Kuss O, Hoyer A, Solms A. Meta-analysis for diagnostic accuracy studies: A new statistical model using beta-binomial distributions and bivariate copulas. Statistics in Medicine 2014; 33(1):17-30. DOI: 10.1002/sim.5909

[2] Hoyer A, Kuss O. Statistical methods for meta-analysis of diagnostic tests accounting for prevalence - A new model using trivariate copulas. Statistics in Medicine 2015; 34(11):1912-24. DOI: 10.1002/sim.6463

[3] Nikoloulopoulos AK. A vine copula mixed effect model for trivariate meta-analysis of diagnostic test accuracy studies accounting for disease prevalence. Statistical Methods in Medical Research 2015 11 Aug; Epub ahead of print

[4] Hoyer A, Kuss O. Comment on 'A vine copula mixed effect model for trivariate meta-analysis of diagnostic test accuracy studies accounting for disease prevalence' by Aristidis K Nikoloulopoulos. Statistical Methods in Medical Research 2016; 25(2):985-7. DOI: 10.1177/0962280216640628

[5] Nikoloulopoulos AK. Comment on 'A vine copula mixed effect model for trivariate meta-analysis of diagnostic test accuracy studies accounting for disease prevalence'. Statistical Methods in Medical Research 2016; 25(2):988-91. DOI: 10.1177/0962280216630190

[6] Liu L, Yu Z. A likelihood reformulation method in non-normal random effects models. Statistics in Medicine 2008; 27(16):3105-3124.

On 2016 May 17, Annika Hoyer commented:

With great interest we noticed this paper by Nikoloulopoulos. The author proposes an approach for the meta-analysis of diagnostic accuracy studies modelling random effects while using copulas. In his work, he compares his model to the copula approach presented by Kuss et al. [1], referred to henceforth as KHS model. We appreciate a lot Nikoloulopoulos referring to our work, but we feel there are some open questions.

The author shows in the appendix that the association parameter from the copula is estimated with large biases from the KHS model, and this is what we also saw in our simulation study. However, the association parameter is not the parameter of main interest which are the overall sensitivities and specificities. They were estimated well in the KHS model, and we considered the copula parameter more as a nuisance parameter. This was also pointed out by Nikoloulopoulos in his paper. As a consequence, we are thus surprised that the bad performance in terms of the association parameter led the author to the verdict that the KHS method is 'inefficient' and 'flawed' and should no longer be used. We do not agree here, because our simulation as well as your theoretical results do clearly show that the KHS estimates the parameters of actual interest very well. Just aside, we saw compromised results for the association parameter also for the GLMM model in our simulation.

Nikoloulopoulos also wrote that the KHS approximation can only be used if the 'number of observations in the respective study group of healthy and diseased probands is the same for each study'. This claim is done at least 3 times in the article. But, unfortunately, there is no proof or reference or at least an example which supports this statement. Without a mathematical proof, we think there could be a misunderstanding in the model. In our model, we assume beta-binomial distributions for the true positives and the true negatives of the i-th study. They were linked using a copula. This happens on the individual study level because we wanted to account for different study sizes. For estimating the meta-analytic parameters of interest we assume that the shape and scale parameters of the beta-binomial distributions as well as the copula parameter are the same across studies, so that the expectation values of the marginal distributions can be treated as the meta-analytic sensitivities and specificities. Of course, it is true that we used equal sample sizes in our simulation [1], however, we see no theoretical reason why different sample sizes should not work. In a recently accepted follow up paper on trivariate copulas [2] we used differing sample sizes in the simulation and we also saw a superior performance of the KHS model as compared to the GLMM. In a follow-up paper of Nikoloulopoulos [3], he repeats this issue with equal group sizes, but, unfortunately, did not answer our question [4,5] with respect to that point.

As the main advantage of the KHS over the GLMM model we see its robustness. Our SAS NLMIXED code for the copula models converged better than PQL estimation (SAS PROC GLIMMIX) and much better that Gauss-Hermite-Quadrature estimation for the GLMM model (SAS PROC NLMIXED). This was true for the original bivariate KHS model, but also for the recent trivariate update. This is certainly to be expected because fitting the KHS model reduces essentially to the fit of a bivariate distribution, but without the complicated computations or approximations for the random effects as it is required for the GLMM and the model of Nikoloulopoulos given here. Numerical problems are also frequently observed if one uses the already existing methods for copula models with non-normal random effects from Liu and Yu [6]. It would be thus very interesting to learn how the authors’ model performs in terms of robustness.

Annika Hoyer, Oliver Kuss

References

[1] Kuss O, Hoyer A, Solms A. Meta-analysis for diagnostic accuracy studies: A new statistical model using beta-binomial distributions and bivariate copulas. Statistics in Medicine 2014; 33(1):17-30. DOI: 10.1002/sim.5909

[2] Hoyer A, Kuss O. Statistical methods for meta-analysis of diagnostic tests accounting for prevalence - A new model using trivariate copulas. Statistics in Medicine 2015; 34(11):1912-24. DOI: 10.1002/sim.6463

[3] Nikoloulopoulos AK. A vine copula mixed effect model for trivariate meta-analysis of diagnostic test accuracy studies accounting for disease prevalence. Statistical Methods in Medical Research 2015 11 Aug; Epub ahead of print

[4] Hoyer A, Kuss O. Comment on 'A vine copula mixed effect model for trivariate meta-analysis of diagnostic test accuracy studies accounting for disease prevalence' by Aristidis K Nikoloulopoulos. Statistical Methods in Medical Research 2016; 25(2):985-7. DOI: 10.1177/0962280216640628

[5] Nikoloulopoulos AK. Comment on 'A vine copula mixed effect model for trivariate meta-analysis of diagnostic test accuracy studies accounting for disease prevalence'. Statistical Methods in Medical Research 2016; 25(2):988-91. DOI: 10.1177/0962280216630190

[6] Liu L, Yu Z. A likelihood reformulation method in non-normal random effects models. Statistics in Medicine 2008; 27(16):3105-3124.