On 2013 Dec 16, Gregory Francis commented:

A slightly longer version of the following text was submitted as a comment to the journal Psychological Research. The editor declined to publish the comment even though two reviewers agreed that the analysis was accurate.

Doerrfeld, Sebanz and Shiffrar (2012) tested whether anticipated effort alters perceptual experience by having individuals judge the weight of a filled basket when they intended to lift the basket either alone or with another person. Four experiments consistently rejected the null hypothesis that the weight judgments were the same across the two intention conditions. The experimental results were interpreted as strong evidence for the validity of the result and confirmation of a theoretical hypothesis that perception is shaped by what can be accomplished with other people.

However, experiments should only reject the null hypothesis at a rate that reflects the power of the experiments. When experiments have low or moderate experimental power, random sampling means that some experiments should not reject the null hypothesis even if the reported effect is true. The absence of expected null findings indicates publication bias, which makes it impossible to judge whether the reported experiments are valid. As shown below, such is the case for the findings in Doerrfeld et al. (2012).

The four experiments used very similar methods and measures, so it is appropriate to pool the effect sizes across the experiments. This produces a Hedges' g value of 0.865, which is the best estimate of the effect size for this phenomenon. Using that pooled effect size gives power values of 0.791, 0.364, 0.449 and 0.637 for the four experiments. The probability that four out of four experiments like these would reject the null hypothesis is the product of the power values, which is 0.082. If they were run properly and reported fully, the experimental outcomes in Doerrfeld et al. (2012) are quite unusual for the reported effect sizes and sample sizes. The low probability of the experiment set is below a 0.1 criterion commonly used to establish publication bias.

This observation is not intended as an accusation against Doerrfeld et al. (2012), who I suspect operated with the best of intentions in designing and reporting their studies. Seemingly minor decisions throughout a research project can lead to a biased experiment set. Ultimately, it does not much matter exactly how bias was introduced; the main observation is that the set of experimental results reported by Doerrfeld et al. (2012) as evidence for their thesis would be quite rare if they were generated without some form of bias. Since there is no way to know the extent of the bias, readers should be skeptical about the findings and conclusions of the study.

The full submitted text and an Excel file detailing the power analyses can be downloaded from:

http://www1.psych.purdue.edu/~gfrancis/Publications/DoerrfeldSebanzShiffrar2012.zip

On 2013 Dec 16, Gregory Francis commented:

A slightly longer version of the following text was submitted as a comment to the journal Psychological Research. The editor declined to publish the comment even though two reviewers agreed that the analysis was accurate.

Doerrfeld, Sebanz and Shiffrar (2012) tested whether anticipated effort alters perceptual experience by having individuals judge the weight of a filled basket when they intended to lift the basket either alone or with another person. Four experiments consistently rejected the null hypothesis that the weight judgments were the same across the two intention conditions. The experimental results were interpreted as strong evidence for the validity of the result and confirmation of a theoretical hypothesis that perception is shaped by what can be accomplished with other people.

However, experiments should only reject the null hypothesis at a rate that reflects the power of the experiments. When experiments have low or moderate experimental power, random sampling means that some experiments should not reject the null hypothesis even if the reported effect is true. The absence of expected null findings indicates publication bias, which makes it impossible to judge whether the reported experiments are valid. As shown below, such is the case for the findings in Doerrfeld et al. (2012).

The four experiments used very similar methods and measures, so it is appropriate to pool the effect sizes across the experiments. This produces a Hedges' g value of 0.865, which is the best estimate of the effect size for this phenomenon. Using that pooled effect size gives power values of 0.791, 0.364, 0.449 and 0.637 for the four experiments. The probability that four out of four experiments like these would reject the null hypothesis is the product of the power values, which is 0.082. If they were run properly and reported fully, the experimental outcomes in Doerrfeld et al. (2012) are quite unusual for the reported effect sizes and sample sizes. The low probability of the experiment set is below a 0.1 criterion commonly used to establish publication bias.

This observation is not intended as an accusation against Doerrfeld et al. (2012), who I suspect operated with the best of intentions in designing and reporting their studies. Seemingly minor decisions throughout a research project can lead to a biased experiment set. Ultimately, it does not much matter exactly how bias was introduced; the main observation is that the set of experimental results reported by Doerrfeld et al. (2012) as evidence for their thesis would be quite rare if they were generated without some form of bias. Since there is no way to know the extent of the bias, readers should be skeptical about the findings and conclusions of the study.

The full submitted text and an Excel file detailing the power analyses can be downloaded from:

http://www1.psych.purdue.edu/~gfrancis/Publications/DoerrfeldSebanzShiffrar2012.zip