I would like to thank the authors for the thorough evaluation of the Cell-/SO-Check device. I believe that this work is very important as it will help other researchers assess the suitability of the device for their own work. However, I believe that not all of the conclusions drawn by the authors are supported by the data shown.
The authors state that no linear relationship between spectrophotometer zinc and serum zinc was found (r = 0.03, p-value = 0.800) and that the spectrophotometer method failed to perform better than chance at identifying individuals with zinc deficiency (ROC-AUC = 0.547). Nevertheless, the authors come to the conclusion that "The Cell-/SO-Check device may be used to rank children in population-based studies in SSA according to their zinc status, [...].". This conclusion seems to be based on two arguments: The first argument is that, according to the authors, the difference between spectrophotometer zinc and serum zinc appears to be consistent, so that it may be possible to correct for the bias. The second argument is a high specificity of 90.91% of the spectrophotometer method when distinguishing between individuals with and without zinc deficiency.
With regard to the first argument, the authors explain:
"Nevertheless, the P-value for the paired samples t-test of both zinc and ferritin was significant (>0.05), suggesting that the bias albeit present, was consistent. This does not necessarily stipulate that the spectrometer and laboratory methods are incomparable. It could be a matter of calibration discrepancies. In fact, according to Bland & Altman, consistent bias can be adjusted for by subtracting the mean difference of the ferritin or zinc measurements by the spectrometer and the laboratory method from the measurement by the spectrometer method [23]."
It is not clear to me what is meant by "paired samples t-test" as I did not find such a test described in the methods section. I'm assuming that this paragraph refers to the simple linear regression of the paired differences on the mean of the paired measurements, so that a significant p-value (< 0.05) in this context likely indicates that a trend is present in the data shown in Fig. 3a. However, to my understanding, the trend seen in Fig. 3a is not at all an indication of consistency of the bias, but merely a result of the fact that the variance of the serum zinc data is much higher than the variance of the data measured using spectrophotometry. The higher variance of the serum zinc data causes the mean of the paired measurements to be mostly controlled by serum zinc, meaning that a low mean of the paired measurements is likely associated with low serum zinc. If serum zinc is low, there is a larger chance that the corresponding spectrophotometer zinc concentration will exceed this value, even if the spectrophotometer data was simply random numbers drawn from a normal distribution with constant mean and standard deviation. The pattern observed would therefore be expected even for random data.
In order for the discrepancy between spectrophotometer zinc and serum zinc to be correctable, there would still need to be a correlation between spectrophotometer zinc and serum zinc. A significant linear correlation was not found as shown by the Pearson correlation coefficient of 0.03 and the respective p-value of 0.8. Visual inspection of the data shown in the scatter plot in Fig. 2a also does not indicate that any other type of relationship between serum zinc and spectrophotometer zinc is present. The data therefore does not seem to support the authors' argument that correction of a presumed consistent bias could make this method suitable for determining zinc status of children in a population-based study.
The second argument presented by the authors in support of the conclusion is that a high specificity of 90.91% with a corresponding sensitivity of 33.3% was found when distinguishing between individuals with and without zinc deficiency. I might be missing something but to me it seems that the given specificity and sensitivity are in contradiction to the numbers shown in Tab. 2. According to Tab. 2, of the 72 study participants, 6 individuals were identified as zinc-deficient by a serum zinc concentration < 7.7 micromol/L (the gold standard the spectrophotometer method is tested against) whereas 17 individuals were identified as zinc-deficient by spectrophotometry. If the sensitivity was 33.3%, that means that 2 of the 17 individuals flagged as zinc-deficient by spectrophotometry were "correctly" classified as zinc-deficient. This means that 15 individuals were "incorrectly" classified as zinc deficient and only 51 participants of the 66 individuals without zinc deficiency were "correctly" classified as not zinc deficient. This would result in a specificity of 77% which is very close to what would be expected by chance and consistent with the ROC-AUC very close to 0.5.
Did the authors use a different cutoff value to calculate sensitivity and specificity than the one used in Tab. 2, possibly the "optimal" cutoff value determined from the ROC-plot? Even if a specific cutoff value can be found that would yield these results, it seems that this is very likely due to chance considering the ROC-AUC close to 0.5 and the same cutoff value is therefore unlikely to perform as well in future studies. Could the authors please clarify what cutoff values were used for detecting zinc deficiency using spectrophotometry for the data in Tab. 2 and for the stated specificity of 90.91% and sensitivity of 33.3%?
In summary, even though I understand the author's point that a measuring device used in a population-based study does not have to fulfill the same requirements as a device intended for clinical use, the data shown in this article seems to be consistent with the interpretation that the spectrophotometer data does not contain any information about the zinc status of the study participants and is therefore not suitable for measuring zinc status in the population investigated, not even for the purpose of a population-based study.