On 2016 Jun 21, Lydia Maniatis commented:

The trick in science is to ask good questions, but that's more difficult than asking nonsense questions. Both types of questions can generate data, but only the former will, in some cases at least, produce data that is intelligible. The question and data here clearly fall into the latter category.

The question being asked is how confident observers will be if we ask them to judge some aspect of a stimulus presented for only 200 milliseconds, and how will this judgment of confidence change when we vary the frequency distribution of some aspect of the display. (The logic of varying frequencies in this way is related to a never-really-explained and huge and falsifiable assumption that a. signal detection theory is appropriate for modelling perceptual processes and b. that varying the distribution of some experimental display is equivalent to the noise term in this SDT paradigm.)

Why 200 milliseconds? Why were observers encouraged to respond quickly? Why do we care how observers feel (or at least respond when forced to express a feeling) about their performance in such extremely pinched and unnatural circumstances?

As it happens, the authors found no consistency between individuals (apparently each person has some preference for responding, but so what?) and no consistency between this study and other studies with the same focus.

The authors provide no justification for why they chose the particular task and the particular experimental parameters they did. What if they had used a different task? What if they had let observers see the stimulus for 500 milliseconds, or even an entire second? Would there be reason to expect different results? Why the forced choice? If you're really interested in confidence, then forcing observers to respond even if they have no real preference will just produce a false impression of confidence, a response based on some rule of thumb that doesn't necessarily measure confidence. You lose precisely the information that you're supposed to be interested in.

The number of possible studies along these lines, using arbitrary tasks and arbitrary parameters, is infinite. Each one will certainly produce some result, and the results will just as certainly be widely variable and often mutually inconsistent. "Visual confidence" is a data mill in need of a rationale adequate to guide experimental control of variables, so that the results of different experiments can be compared with more confidence than the hopelessly vague speculation on display here:

"In any case, the reasons why these inter-individual differences occurred in our study, but not in other studies (e.g. [9, 13]), remain to be clarified, but the stimuli or the procedure (e.g. the use of confidence ratings vs. confidence comparison) might have played a role in this issue. One could speculate that these inter-individual differences relate to the prior beliefs about the noise variance, or to the sensory sensitivity to motion variability or motion deviations. It is also possible that they relate to more cognitive phenomena like misperception of statistical variability ([20]), distortions in subjective probability weighting ([21]), or attitudes towards risk. In any case, past research has demonstrated that such individual differences do provide an interesting leverage when investigating choice behavior and metacognitive abilities, both from a behavioral and a neuroscientific perspective (e.g. [22㲴])."

Could be anything; the pitfalls of theory-lite data collection.

Throwing out three out of the 18 original participants based on "poor performance" or "extreme bias" in the confidence task is questionable methodologically and seems like a way to paper over the lack of robustness and consistency (and intelligibility) in the results of a task whose difficulty is of the nature of testing, e.g., figure-ground perception in a myope without their glasses. Adding theoretically trivial difficulty of this type makes the data fuzzy and allows flexibility and perpetual readjustment in their interpretation, and perpetual pretexts for more data collection.

With respect to the idea that individual diffs might "relate to the prior beliefs about the noise variance":

Not only is variance treated as "noise" in the signal detection scheme, but now we layer on top of this tenuous assumption another one, that the perceptual (or cognitive?) system might have prior beliefs about this variance/noise. Note that the experimenter-determined variance is never expressed in the stimulus as such. The stimuli experienced could be from any distribution that simply contains each sample experienced. How would we determine which distribution conforms to the supposed prior belief? If there is no way to determine this, then the possibility is not testable, and thus not scientific.

On 2016 Jun 21, Lydia Maniatis commented:

The trick in science is to ask good questions, but that's more difficult than asking nonsense questions. Both types of questions can generate data, but only the former will, in some cases at least, produce data that is intelligible. The question and data here clearly fall into the latter category.

The question being asked is how confident observers will be if we ask them to judge some aspect of a stimulus presented for only 200 milliseconds, and how will this judgment of confidence change when we vary the frequency distribution of some aspect of the display. (The logic of varying frequencies in this way is related to a never-really-explained and huge and falsifiable assumption that a. signal detection theory is appropriate for modelling perceptual processes and b. that varying the distribution of some experimental display is equivalent to the noise term in this SDT paradigm.)

Why 200 milliseconds? Why were observers encouraged to respond quickly? Why do we care how observers feel (or at least respond when forced to express a feeling) about their performance in such extremely pinched and unnatural circumstances?

As it happens, the authors found no consistency between individuals (apparently each person has some preference for responding, but so what?) and no consistency between this study and other studies with the same focus.

The authors provide no justification for why they chose the particular task and the particular experimental parameters they did. What if they had used a different task? What if they had let observers see the stimulus for 500 milliseconds, or even an entire second? Would there be reason to expect different results? Why the forced choice? If you're really interested in confidence, then forcing observers to respond even if they have no real preference will just produce a false impression of confidence, a response based on some rule of thumb that doesn't necessarily measure confidence. You lose precisely the information that you're supposed to be interested in.

The number of possible studies along these lines, using arbitrary tasks and arbitrary parameters, is infinite. Each one will certainly produce some result, and the results will just as certainly be widely variable and often mutually inconsistent. "Visual confidence" is a data mill in need of a rationale adequate to guide experimental control of variables, so that the results of different experiments can be compared with more confidence than the hopelessly vague speculation on display here:

"In any case, the reasons why these inter-individual differences occurred in our study, but not in other studies (e.g. [9, 13]), remain to be clarified, but the stimuli or the procedure (e.g. the use of confidence ratings vs. confidence comparison) might have played a role in this issue. One could speculate that these inter-individual differences relate to the prior beliefs about the noise variance, or to the sensory sensitivity to motion variability or motion deviations. It is also possible that they relate to more cognitive phenomena like misperception of statistical variability ([20]), distortions in subjective probability weighting ([21]), or attitudes towards risk. In any case, past research has demonstrated that such individual differences do provide an interesting leverage when investigating choice behavior and metacognitive abilities, both from a behavioral and a neuroscientific perspective (e.g. [22㲴])."

Could be anything; the pitfalls of theory-lite data collection.

Throwing out three out of the 18 original participants based on "poor performance" or "extreme bias" in the confidence task is questionable methodologically and seems like a way to paper over the lack of robustness and consistency (and intelligibility) in the results of a task whose difficulty is of the nature of testing, e.g., figure-ground perception in a myope without their glasses. Adding theoretically trivial difficulty of this type makes the data fuzzy and allows flexibility and perpetual readjustment in their interpretation, and perpetual pretexts for more data collection.

With respect to the idea that individual diffs might "relate to the prior beliefs about the noise variance":

Not only is variance treated as "noise" in the signal detection scheme, but now we layer on top of this tenuous assumption another one, that the perceptual (or cognitive?) system might have prior beliefs about this variance/noise. Note that the experimenter-determined variance is never expressed in the stimulus as such. The stimuli experienced could be from any distribution that simply contains each sample experienced. How would we determine which distribution conforms to the supposed prior belief? If there is no way to determine this, then the possibility is not testable, and thus not scientific.