On 2015 Nov 13, Lydia Maniatis commented:

Brainard and colleagues have apparently been using the practical-definition-free term "naturalistic" for at least 18 years, as indicated by the article titled: "Color constancy in the nearly natural image. 1. Asymmetric matches. (Brainard, D. H., Brunt, W. A, & Speigle, J. M. (1997), Journal of the Optical Society of America. A, Optics, Image Science, and l'ision, 14, 2091-2110.)

Without stimulus/variable control, questions cannot be framed in a meaningful, clear way. To say that we are controlling for "degree of naturalism" is clearly not a viable option. Not to mention that, as noted above, proving that color constancy is pretty good in "natural" conditions is not interesting - if it weren't, we wouldn't even be asking the question. The question for scientists is, how is it achieved? Refining the answers to this question requires that we posit the role of specific factors, and this means clarifying what features of our stimulus are being manipulated. Again, "degree of naturalness" is a non-starter, and if investigators want to use it, they need to be clear about what they mean by it, and what features are under their control, and what features are NOT under their control, and might affect their results.

In the present study, not even the factor supposedly being manipulated - illumination - was under the investigators' control (see above). They are taking a graphics package's assumptions about what will be perceived as the color of the illumination at face value.

On 2015 Nov 03, Lydia Maniatis commented:

What is the purpose of the data collected in this study? Assuming there is a purpose, are its methods consistent with achieving it?

The title, “Color constancy in a naturalistic, goal-directed task,” is no guide. The ability of humans to achieve color constancy evolved in the natural environment, on the basis of interaction with that environment. Only goal-directed action could lead to selection for a functional perceptual process. “Natural” conditions are where the process enabling color constancy is expected to be most reliable. A plausible question, then, might be “how accurate can color constancy be at its best (assuming we were interested and could measure this.)” However, this is not what the authors are asking.

The abstract suggests two possible motives. First, we are told that “whether and to what degree constancy supports accurate cross-illumination object selection is not well understood.” It is not clear why we would not expect reliable surface perception under variable lighting to “support object selection.” At any rate, this framing of the question seems to assume good color constancy as the independent variable and “object selection” as the dependent variable. A few sentences later, however, the question seems to change; the authors tell us that they “were able to infer and quantify the degree of color constancy that mediated subjects performance.” Now, what seems to have been under investigation is the degree of subjects' color constancy.

However, the question “How good is a subject's color constancy” is meaningless unless we specify conditions. Under some conditions, it is very good, under others, it fails completely. Much is known about these conditions. Artists, for example, are expert at causing radical failures of color constancy via pictorial impressions of inhomogeneous illumination. Perception scientists sometimes have an inkling of this as well. So what specific conditions did the authors aim to test, and why?

The term used is “naturalistic.” What does this mean? The authors apparently can't say. We are told that they manipulated “complexity” by increasing (relative to a previous experiment) the number of distinct surfaces, but that they “do not have at present a precise technical definition of complexity nor precise ways of relating the concepts of scene complexity and scene naturalness” (p. 19). Evidently, “naturalistic” is being defined as “containing more surfaces than stimuli used in a previous experiment.”

At any rate, the stimuli used are clearly not “naturalistic” in any intuitive sense of the word. The authors themselves describe stimuli in one condition as having “a visually jarring appearance – somewhat reminiscent of an image of an early 1970's discotheque...” (p. 17).

So are the authors at least testing color constancy in a randomly selected stimulus? They are not doing even that. This is because their definition of a color constant selection is not based on subjects selecting an e.g. actually red surface under different illuminations, but on subjects matching a red surface with a surface that, in many cases, is not red (i.e. does not have the same chromaticity as the sample.) The stimuli are images on a computer screen, and the changes of illumination are manipulations that the authors hope, or expect, will correspond to the impression of a change in illumination, e.g. the impression of orange lighting. The theoretical assumptions underlying this expectation are part of the graphics package and are never made explicit. They might need reassessment, as subjects did not always make the predicted “color constant” selections. There was, for example, a deviation from the authors' definition/prediction of veridical choices, which based on deviation from their assumptions, they call a “blue bias.” In other words, subjects sometimes matched a sample surface with one that projected more in the blue part of the spectrum than what the authors defined as a “color constant” match. They “do not have an explanation for this bias” (p. 18). However, whether we are dealing with a bias on the part of subjects' perceptual system (unlikely) or a failed prediction of the authors' prepackaged, implicit color constancy/illumination perception model is impossible to say in this hopelessly confounded study.

I would note, finally, that defining the effect of simultaneous contrast as a “failure of color constancy” constitutes a failure to distinguish between figure/ground contrast and adjustments of color contingent on perceived illumination variation.

On 2015 Nov 03, Lydia Maniatis commented:

What is the purpose of the data collected in this study? Assuming there is a purpose, are its methods consistent with achieving it?

The title, “Color constancy in a naturalistic, goal-directed task,” is no guide. The ability of humans to achieve color constancy evolved in the natural environment, on the basis of interaction with that environment. Only goal-directed action could lead to selection for a functional perceptual process. “Natural” conditions are where the process enabling color constancy is expected to be most reliable. A plausible question, then, might be “how accurate can color constancy be at its best (assuming we were interested and could measure this.)” However, this is not what the authors are asking.

The abstract suggests two possible motives. First, we are told that “whether and to what degree constancy supports accurate cross-illumination object selection is not well understood.” It is not clear why we would not expect reliable surface perception under variable lighting to “support object selection.” At any rate, this framing of the question seems to assume good color constancy as the independent variable and “object selection” as the dependent variable. A few sentences later, however, the question seems to change; the authors tell us that they “were able to infer and quantify the degree of color constancy that mediated subjects performance.” Now, what seems to have been under investigation is the degree of subjects' color constancy.

However, the question “How good is a subject's color constancy” is meaningless unless we specify conditions. Under some conditions, it is very good, under others, it fails completely. Much is known about these conditions. Artists, for example, are expert at causing radical failures of color constancy via pictorial impressions of inhomogeneous illumination. Perception scientists sometimes have an inkling of this as well. So what specific conditions did the authors aim to test, and why?

The term used is “naturalistic.” What does this mean? The authors apparently can't say. We are told that they manipulated “complexity” by increasing (relative to a previous experiment) the number of distinct surfaces, but that they “do not have at present a precise technical definition of complexity nor precise ways of relating the concepts of scene complexity and scene naturalness” (p. 19). Evidently, “naturalistic” is being defined as “containing more surfaces than stimuli used in a previous experiment.”

At any rate, the stimuli used are clearly not “naturalistic” in any intuitive sense of the word. The authors themselves describe stimuli in one condition as having “a visually jarring appearance – somewhat reminiscent of an image of an early 1970's discotheque...” (p. 17).

So are the authors at least testing color constancy in a randomly selected stimulus? They are not doing even that. This is because their definition of a color constant selection is not based on subjects selecting an e.g. actually red surface under different illuminations, but on subjects matching a red surface with a surface that, in many cases, is not red (i.e. does not have the same chromaticity as the sample.) The stimuli are images on a computer screen, and the changes of illumination are manipulations that the authors hope, or expect, will correspond to the impression of a change in illumination, e.g. the impression of orange lighting. The theoretical assumptions underlying this expectation are part of the graphics package and are never made explicit. They might need reassessment, as subjects did not always make the predicted “color constant” selections. There was, for example, a deviation from the authors' definition/prediction of veridical choices, which based on deviation from their assumptions, they call a “blue bias.” In other words, subjects sometimes matched a sample surface with one that projected more in the blue part of the spectrum than what the authors defined as a “color constant” match. They “do not have an explanation for this bias” (p. 18). However, whether we are dealing with a bias on the part of subjects' perceptual system (unlikely) or a failed prediction of the authors' prepackaged, implicit color constancy/illumination perception model is impossible to say in this hopelessly confounded study.

I would note, finally, that defining the effect of simultaneous contrast as a “failure of color constancy” constitutes a failure to distinguish between figure/ground contrast and adjustments of color contingent on perceived illumination variation.

On 2015 Nov 13, Lydia Maniatis commented:

Brainard and colleagues have apparently been using the practical-definition-free term "naturalistic" for at least 18 years, as indicated by the article titled: "Color constancy in the nearly natural image. 1. Asymmetric matches. (Brainard, D. H., Brunt, W. A, & Speigle, J. M. (1997), Journal of the Optical Society of America. A, Optics, Image Science, and l'ision, 14, 2091-2110.)

Without stimulus/variable control, questions cannot be framed in a meaningful, clear way. To say that we are controlling for "degree of naturalism" is clearly not a viable option. Not to mention that, as noted above, proving that color constancy is pretty good in "natural" conditions is not interesting - if it weren't, we wouldn't even be asking the question. The question for scientists is, how is it achieved? Refining the answers to this question requires that we posit the role of specific factors, and this means clarifying what features of our stimulus are being manipulated. Again, "degree of naturalness" is a non-starter, and if investigators want to use it, they need to be clear about what they mean by it, and what features are under their control, and what features are NOT under their control, and might affect their results.

In the present study, not even the factor supposedly being manipulated - illumination - was under the investigators' control (see above). They are taking a graphics package's assumptions about what will be perceived as the color of the illumination at face value.