- Jul 2018
-
europepmc.org europepmc.org
-
On 2013 Dec 09, Gregory Francis commented:
I submitted the following text to J. Neuroscience a few months ago. The editor evaluated the critique but decided against publishing it.
Zhou, Zhang, Chen, Wang, and Chen (2012) reported experimental findings that perfectly matched their theory that olfaction influences early visual perception. Calculations of experimental power suggest that such perfection is contrary to the uncertainty that exists from random sampling.
Three results showed significant nostril-specific effects of odor on vision: experiments 1 (two results) and 2 had powers of 0.501, 0.513, and 0.529, respectively. Experiment 3 showed a significant effect of odor on visual perception (power 0.547) but, consistent with the theory, no nostril-specific effect.
Seventeen null results also provided evidence for the theory by ruling out alternative explanations. If the theory is valid, the true effects are similar to what was reported, and the null findings are accurate, then the probability of an entire set of findings being perfectly aligned with the theory is the product of the power values times the probability of the null findings: 0.075 x (0.95)<sup>17</sup> = 0.031.
One hundred thousand simulated replications, using the observed significant statistics as population parameters, checked the possibility of a chance pattern. Applying the same power analysis, only two of the simulated experiment sets produced a probability below the observed 0.075. On average, a simulated experiment set produced 2.1 significant findings out of four experiments.
The too perfect data undermine the theoretical interpretation. Perhaps experiments were run improperly, the analysis was done improperly, or publication bias filtered out inconsistent experimental findings. Seemingly trivial choices in experimental design or analysis can bias findings that depend on complex patterns of experimental outcomes.
This comment, imported by Hypothesis from PubMed Commons, is licensed under CC BY.
-
- Feb 2018
-
europepmc.org europepmc.org
-
On 2013 Dec 09, Gregory Francis commented:
I submitted the following text to J. Neuroscience a few months ago. The editor evaluated the critique but decided against publishing it.
Zhou, Zhang, Chen, Wang, and Chen (2012) reported experimental findings that perfectly matched their theory that olfaction influences early visual perception. Calculations of experimental power suggest that such perfection is contrary to the uncertainty that exists from random sampling.
Three results showed significant nostril-specific effects of odor on vision: experiments 1 (two results) and 2 had powers of 0.501, 0.513, and 0.529, respectively. Experiment 3 showed a significant effect of odor on visual perception (power 0.547) but, consistent with the theory, no nostril-specific effect.
Seventeen null results also provided evidence for the theory by ruling out alternative explanations. If the theory is valid, the true effects are similar to what was reported, and the null findings are accurate, then the probability of an entire set of findings being perfectly aligned with the theory is the product of the power values times the probability of the null findings: 0.075 x (0.95)<sup>17</sup> = 0.031.
One hundred thousand simulated replications, using the observed significant statistics as population parameters, checked the possibility of a chance pattern. Applying the same power analysis, only two of the simulated experiment sets produced a probability below the observed 0.075. On average, a simulated experiment set produced 2.1 significant findings out of four experiments.
The too perfect data undermine the theoretical interpretation. Perhaps experiments were run improperly, the analysis was done improperly, or publication bias filtered out inconsistent experimental findings. Seemingly trivial choices in experimental design or analysis can bias findings that depend on complex patterns of experimental outcomes.
This comment, imported by Hypothesis from PubMed Commons, is licensed under CC BY.
-