On 2015 Apr 24, CHARLES KING commented:

I feel there is a serious methodological flaw in the recently published Cochrane Review “Circulating antigen tests and urine reagent strips for diagnosis of active schistosomiasis in endemic areas” by Ochodo, et el., 2015. The inaccurate analysis used in the HSROC estimation skews the reported summary results on the diagnostic performance of both antigen tests and dipsticks, and a correct analysis would actually invalidate several of the conclusions and recommendations found in the current version of the review: My objection is to the use of egg count diagnostics as a reference standard for the diagnosis of Schistosoma infection. The stool egg count for S. mansoni and S. japonicum and the urine filtration egg count for S. haematobium have long been known to be poorly sensitive for low intensity infections. When subjects are repeatedly tested for 7-15 days in a row, single day egg count testing has a sensitivity of 40-60%. Although these imperfect tests remain in widespread use in field studies and control campaigns because they accurately detect persons with heavy infections, they cannot be relied upon to establish infection in all subjects. The failings of stool counting for S. mansoni were documented in the work of de Vlas and colleagues cited below [1,2]. The limitations of stool counting for S. japonicum have been established by Carabin, et al.,[3] and Hubbard, et al.,[4] and the limitations of egg counting for S. haematobium can be found in Savioli et al.,[5] and Warren, et al.[6]

In all cases, egg counting cannot be considered a ‘gold standard’ diagnostic. Such a test, with ~50% sensitivity, is so inaccurate to be useless on a per-individual diagnostic basis. I feel that the appropriate comparison for the Ochodo, et al., review would have been Latent Class Analysis, in which two imperfect tests are compared in their attempt to classify an unmeasured ‘true’ infection status. Using egg count results as a reference standard in the HSROC was a serious error--reporting a new test’s performance benchmarked against an already flawed test is meaningless, and in fact, the reported comparisons are misleading to the practitioner or public health officer.

Secondly, I feel that the exclusion of results from populations or areas without significant Schistosoma risk was also a tactical error. If we are concerned about the specificity of new tests, there is great value in measuring results among persons who have a very low prior probability of infection.

I would strongly recommend that the Cochrane review be revised and re-issued after the authors revisit the data using the approach of Dendukuri, et al., 2012 [7] for situations where there is no gold standard. Their SAS code is available online, and the reanalysis could be done in a matter of a day. The Bayesian LCA should be informed by prior observations on the estimated specificity of single (or treble stool) examinations, and the known specificity estimates of dipsticks and antigen testing among non-endemic populations.

The concern is that while the authors discuss and reiterate the lack of a gold standard and the insensitivity of the egg count procedures they go right ahead and use them as their comparator and they present strong conclusions that are biased by their approach. This will only add to policymakers’ confusion about the utility of these alternative test approaches.

By way of disclosure, I have no relation to the manufacturers of these tests, and I have no conflict of interest in this matter.

1. de Vlas SJ, Engels D, Rabello AL, Oostburg BF, Van Lieshout L, Polderman AM, Van Oortmarssen GJ, Habbema JD, Gryseels B, 1997. Validation of a chart to estimate true Schistosoma mansoni prevalences from simple egg counts. Parasitology 114 ( Pt 2): 113-21.
2. de Vlas SJ, Gryseels B, 1992. Underestimation of Schistosoma mansoni prevalences. Parasitol Today 8: 274-277.
3. Carabin H, Marshall CM, Joseph L, Riley S, Olveda R, McGarvey ST, 2005. Estimating the intensity of infection with Schistosoma japonicum in villagers of Leyte, Philippines. Part I: A Bayesian cumulative logit model. The Schistosomiasis Transmission & Ecology Project (STEP). Am J Trop Med Hyg 72: 745-753.
4. Hubbard A, Liang S, Maszle D, Qiu D, Gu X, Spear RC, 2002. Estimating the distribution of worm burden and egg excretion of Schistosoma japonicum by risk group in Sichuan Province, China. Parasitology 125: 221-31.
5. Savioli L, Hatz C, Dixon H, Kisumku UM, Mott KE, 1990. Control of morbidity due to Schistosoma haematobium on Pemba Island: egg excretion and hematuria as indicators of infection. Am J Trop Med Hyg 43: 289-295.
6. Warren KS, Arap Siongok TK, Hauser HB, Ouma JH, Peters PAS, 1978. Quantification of infection with Schistosoma haematobium in relation to epidemiology and selective population chemotherapy. I. Minimal number of daily egg counts in urine necessary to establish intensity of infection. Journal of Infectious Diseases 138: 849-55.
7. Dendukuri N, Schiller I, Joseph L, Pai M, 2012. Bayesian meta-analysis of the accuracy of a test for tuberculous pleuritis in the absence of a gold standard reference. Biometrics 68: 1285-1293.

On 2015 Apr 24, CHARLES KING commented:

I feel there is a serious methodological flaw in the recently published Cochrane Review “Circulating antigen tests and urine reagent strips for diagnosis of active schistosomiasis in endemic areas” by Ochodo, et el., 2015. The inaccurate analysis used in the HSROC estimation skews the reported summary results on the diagnostic performance of both antigen tests and dipsticks, and a correct analysis would actually invalidate several of the conclusions and recommendations found in the current version of the review: My objection is to the use of egg count diagnostics as a reference standard for the diagnosis of Schistosoma infection. The stool egg count for S. mansoni and S. japonicum and the urine filtration egg count for S. haematobium have long been known to be poorly sensitive for low intensity infections. When subjects are repeatedly tested for 7-15 days in a row, single day egg count testing has a sensitivity of 40-60%. Although these imperfect tests remain in widespread use in field studies and control campaigns because they accurately detect persons with heavy infections, they cannot be relied upon to establish infection in all subjects. The failings of stool counting for S. mansoni were documented in the work of de Vlas and colleagues cited below [1,2]. The limitations of stool counting for S. japonicum have been established by Carabin, et al.,[3] and Hubbard, et al.,[4] and the limitations of egg counting for S. haematobium can be found in Savioli et al.,[5] and Warren, et al.[6]

In all cases, egg counting cannot be considered a ‘gold standard’ diagnostic. Such a test, with ~50% sensitivity, is so inaccurate to be useless on a per-individual diagnostic basis. I feel that the appropriate comparison for the Ochodo, et al., review would have been Latent Class Analysis, in which two imperfect tests are compared in their attempt to classify an unmeasured ‘true’ infection status. Using egg count results as a reference standard in the HSROC was a serious error--reporting a new test’s performance benchmarked against an already flawed test is meaningless, and in fact, the reported comparisons are misleading to the practitioner or public health officer.

Secondly, I feel that the exclusion of results from populations or areas without significant Schistosoma risk was also a tactical error. If we are concerned about the specificity of new tests, there is great value in measuring results among persons who have a very low prior probability of infection.

I would strongly recommend that the Cochrane review be revised and re-issued after the authors revisit the data using the approach of Dendukuri, et al., 2012 [7] for situations where there is no gold standard. Their SAS code is available online, and the reanalysis could be done in a matter of a day. The Bayesian LCA should be informed by prior observations on the estimated specificity of single (or treble stool) examinations, and the known specificity estimates of dipsticks and antigen testing among non-endemic populations.

The concern is that while the authors discuss and reiterate the lack of a gold standard and the insensitivity of the egg count procedures they go right ahead and use them as their comparator and they present strong conclusions that are biased by their approach. This will only add to policymakers’ confusion about the utility of these alternative test approaches.

By way of disclosure, I have no relation to the manufacturers of these tests, and I have no conflict of interest in this matter.

1. de Vlas SJ, Engels D, Rabello AL, Oostburg BF, Van Lieshout L, Polderman AM, Van Oortmarssen GJ, Habbema JD, Gryseels B, 1997. Validation of a chart to estimate true Schistosoma mansoni prevalences from simple egg counts. Parasitology 114 ( Pt 2): 113-21.
2. de Vlas SJ, Gryseels B, 1992. Underestimation of Schistosoma mansoni prevalences. Parasitol Today 8: 274-277.
3. Carabin H, Marshall CM, Joseph L, Riley S, Olveda R, McGarvey ST, 2005. Estimating the intensity of infection with Schistosoma japonicum in villagers of Leyte, Philippines. Part I: A Bayesian cumulative logit model. The Schistosomiasis Transmission & Ecology Project (STEP). Am J Trop Med Hyg 72: 745-753.
4. Hubbard A, Liang S, Maszle D, Qiu D, Gu X, Spear RC, 2002. Estimating the distribution of worm burden and egg excretion of Schistosoma japonicum by risk group in Sichuan Province, China. Parasitology 125: 221-31.
5. Savioli L, Hatz C, Dixon H, Kisumku UM, Mott KE, 1990. Control of morbidity due to Schistosoma haematobium on Pemba Island: egg excretion and hematuria as indicators of infection. Am J Trop Med Hyg 43: 289-295.
6. Warren KS, Arap Siongok TK, Hauser HB, Ouma JH, Peters PAS, 1978. Quantification of infection with Schistosoma haematobium in relation to epidemiology and selective population chemotherapy. I. Minimal number of daily egg counts in urine necessary to establish intensity of infection. Journal of Infectious Diseases 138: 849-55.
7. Dendukuri N, Schiller I, Joseph L, Pai M, 2012. Bayesian meta-analysis of the accuracy of a test for tuberculous pleuritis in the absence of a gold standard reference. Biometrics 68: 1285-1293.