On 2017 Nov 19, Kenneth Witwer commented:

This study suggests that approximately 20 microRNAs (miRNAs) in cerebrospinal fluid (CSF) may distinguish between different responses to exercise or between sedentary controls and individuals living with chronic fatigue syndrome (CFS, also known as myalgic encephalomyelitis, or ME) and Gulf War Illness (GWI). Funded by recent NIH efforts to support research into several related, poorly-understood and debilitating conditions, this report represents a large and commendable undertaking in terms of patient recruitment (182 individuals) and sample collection. Unfortunately, experimental and analytical issues limit interpretation of the results. I would like to highlight several of these here in the hopes of stimulating rigorous follow-up as appropriate.

1) The choice of a threshold of 35 for a clearly noisy qPCR array. A Cq threshold (to distinguish noise from real signal) was assigned to Cq=35, higher than might have been supported by the negative controls. 82 "positive" negative control features were found in this dataset, with an average Cq of 35.6. 34 of these data points were below Cq 35, averaging 33.1, and several amplified before 25.9. This is a high false positive signal, raising concerns about specificity of the array and suggesting that a lower threshold might have been appropriate.

2) The noise threshold was effectively ignored by adjusting all values above 35 (PCR noise) to 35. This replaced noise with "data" points and introduced a block of artificially invariable values. In a re-analysis after discarding Cq>35, at least half of the reported differences fall away or are smaller. Specifically, results for the following miRNAs appear to be affected by the Cq=35 adjustment: miR-99b-5p, miR-423-5p, miR-204-5p, miR-30d-5p; let-7i-5p, miR-200a-5p, and miR-93-3p (these three miRNAs also do not reach the 2/3 detection threshold in the exercise groups); miR-19b-3p, miR-505-3p, miR-532-5p, and miR-186-3p (for the latter, only one and two Cq<35 data points were gathered for the SC and CFS groups, respectively); and miR-22-3p and miR-9-3p (detected in only one group). miR-22-3p may remain significant, but the real difference between groups is about half the reported value.

3) A miRNA was considered "expressed" in any one of the seven groups only if it was found in 2/3 of all samples. Unfortunately, this threshold was then ignored in analysis, when even miRNAs that were "expressed" in only one group were compared between groups. In some cases, miRNAs with only one or two amplifications in a group were reported (like miR-186-3p).

4) No validation or measures of technical variability. First, there is no assessment of technical variability. The qPCR array gives one reading per miRNA per sample, with the exception of a negative control, read twice. Reading the same sample on three or four arrays would show how precise the readings are. Some of the miRNAs have up to 15-cycle detection ranges (even with Cq=35 as the cutoff) within groups. This represents a >32,000-fold range if it is an accurate measure of abundance. As a result, the purported differences between groups are often overshadowed by large standard deviations. It is unclear how much of this variability is technical, thus it is difficult to assign any differences to biology. Second, no findings are validated by individual assays. Instead of, or in addition to, technical array repeats, individual qPCR assays are routinely used in RNA biomarker studies to confirm profiling findings. These experiments are relatively cheap and permit multiple technical replicates for each sample. Third, no spike-ins are reported (RNA extraction efficiency) nor is there quality control of RNA prior to array measurements. While I can understand that the authors's enthusiasm might have spurred them to proceed without some standard procedures, it is perplexing that reviewers (and editors) would not ask for these crucial experiments.

Despite these critiques, there are several findings that appear to be supported by a reanalysis (excluding Cq>35). These include the finding of 40-fold decreased abundance of miR-328 in CSF of exercised versus non-exercised individuals (although this is not restricted to any particular disease group), and, to a lesser extent (8-fold), miR-608. While validation is needed, and ideally a before-and-after study (in animals if this is too demanding in humans), at least these miRNAs may justify follow-up. Several other miRNAs are also significantly different between the two groups, but with large variability, diminishing their likely utility as biomarkers. However, with most of the purported CFS or GWI differences confounded by the thresholding strategies, and with no validation, additional evidence will be needed before other conclusions can be drawn. Notably, there was no discernible pattern of neuronal, oligodendrocyte (providing myelin sheath), or inflammation-associated miRNAs, which one might expect with damage or inflammation associated with CFS/ME or the acetylcholinesterase-linked mechanisms for GWI that is mentioned, so I am doubtful that this avenue of investigation will be fruitful.

Patients of CFS/ME not only have to live with this condition, they have been through a lot in recent years with irreproducible science. I would thus encourage the authors both to moderate speculation about CFS or GWI-associated miRNAs for now and to proceed with and report the results of blinded qPCR quantification. I would recommend using the standard stem-loop RT/TaqMan assays for this. Indeed, I would be glad to examine samples along with you in a blinded fashion if it would be helpful.

On 2017 Nov 19, Kenneth Witwer commented:

This study suggests that approximately 20 microRNAs (miRNAs) in cerebrospinal fluid (CSF) may distinguish between different responses to exercise or between sedentary controls and individuals living with chronic fatigue syndrome (CFS, also known as myalgic encephalomyelitis, or ME) and Gulf War Illness (GWI). Funded by recent NIH efforts to support research into several related, poorly-understood and debilitating conditions, this report represents a large and commendable undertaking in terms of patient recruitment (182 individuals) and sample collection. Unfortunately, experimental and analytical issues limit interpretation of the results. I would like to highlight several of these here in the hopes of stimulating rigorous follow-up as appropriate.

1) The choice of a threshold of 35 for a clearly noisy qPCR array. A Cq threshold (to distinguish noise from real signal) was assigned to Cq=35, higher than might have been supported by the negative controls. 82 "positive" negative control features were found in this dataset, with an average Cq of 35.6. 34 of these data points were below Cq 35, averaging 33.1, and several amplified before 25.9. This is a high false positive signal, raising concerns about specificity of the array and suggesting that a lower threshold might have been appropriate.

2) The noise threshold was effectively ignored by adjusting all values above 35 (PCR noise) to 35. This replaced noise with "data" points and introduced a block of artificially invariable values. In a re-analysis after discarding Cq>35, at least half of the reported differences fall away or are smaller. Specifically, results for the following miRNAs appear to be affected by the Cq=35 adjustment: miR-99b-5p, miR-423-5p, miR-204-5p, miR-30d-5p; let-7i-5p, miR-200a-5p, and miR-93-3p (these three miRNAs also do not reach the 2/3 detection threshold in the exercise groups); miR-19b-3p, miR-505-3p, miR-532-5p, and miR-186-3p (for the latter, only one and two Cq<35 data points were gathered for the SC and CFS groups, respectively); and miR-22-3p and miR-9-3p (detected in only one group). miR-22-3p may remain significant, but the real difference between groups is about half the reported value.

3) A miRNA was considered "expressed" in any one of the seven groups only if it was found in 2/3 of all samples. Unfortunately, this threshold was then ignored in analysis, when even miRNAs that were "expressed" in only one group were compared between groups. In some cases, miRNAs with only one or two amplifications in a group were reported (like miR-186-3p).

4) No validation or measures of technical variability. First, there is no assessment of technical variability. The qPCR array gives one reading per miRNA per sample, with the exception of a negative control, read twice. Reading the same sample on three or four arrays would show how precise the readings are. Some of the miRNAs have up to 15-cycle detection ranges (even with Cq=35 as the cutoff) within groups. This represents a >32,000-fold range if it is an accurate measure of abundance. As a result, the purported differences between groups are often overshadowed by large standard deviations. It is unclear how much of this variability is technical, thus it is difficult to assign any differences to biology. Second, no findings are validated by individual assays. Instead of, or in addition to, technical array repeats, individual qPCR assays are routinely used in RNA biomarker studies to confirm profiling findings. These experiments are relatively cheap and permit multiple technical replicates for each sample. Third, no spike-ins are reported (RNA extraction efficiency) nor is there quality control of RNA prior to array measurements. While I can understand that the authors's enthusiasm might have spurred them to proceed without some standard procedures, it is perplexing that reviewers (and editors) would not ask for these crucial experiments.

Despite these critiques, there are several findings that appear to be supported by a reanalysis (excluding Cq>35). These include the finding of 40-fold decreased abundance of miR-328 in CSF of exercised versus non-exercised individuals (although this is not restricted to any particular disease group), and, to a lesser extent (8-fold), miR-608. While validation is needed, and ideally a before-and-after study (in animals if this is too demanding in humans), at least these miRNAs may justify follow-up. Several other miRNAs are also significantly different between the two groups, but with large variability, diminishing their likely utility as biomarkers. However, with most of the purported CFS or GWI differences confounded by the thresholding strategies, and with no validation, additional evidence will be needed before other conclusions can be drawn. Notably, there was no discernible pattern of neuronal, oligodendrocyte (providing myelin sheath), or inflammation-associated miRNAs, which one might expect with damage or inflammation associated with CFS/ME or the acetylcholinesterase-linked mechanisms for GWI that is mentioned, so I am doubtful that this avenue of investigation will be fruitful.

Patients of CFS/ME not only have to live with this condition, they have been through a lot in recent years with irreproducible science. I would thus encourage the authors both to moderate speculation about CFS or GWI-associated miRNAs for now and to proceed with and report the results of blinded qPCR quantification. I would recommend using the standard stem-loop RT/TaqMan assays for this. Indeed, I would be glad to examine samples along with you in a blinded fashion if it would be helpful.