On 2015 Sep 30, Lydia Maniatis commented:

This study has very serious problems in terms of both its purpose and its methods.

Purpose: The authors' guiding question is “how color constancy supports object selection,” an issue they claim is “not understood.” What, exactly, is not understood? If color constancy is achieved, if the percept is clear and the task unambiguous, then what do we hope to gain by having subjects choose a matching surface (out of a limited number of pre-selected options) rather thanfreely adjusting a patch, and then comparing it to a target in order to judge the correspondence between the two? Unless the means of gauging the subjects' perceptual experience is expected to affect the experience itself, then why not choose the most direct, precise, and easy-to-analyze means, rather than a cruder, ambiguous, difficult-to-analyze method deemed to be more “naturalistic”? (The authors themselves acknowledge the crudeness of their method – basically a forced-choice method - in relation to the free-choice “asymmetric matching” method).

Methods: 1) The authors' use the dimensions “naturalistic” vs “simple” as their independent variables, without explaining what they mean by the terms. The study purports to compare color comparison/selection in a “naturalistic” stimulus vs a “simple” stimulus. The problem with this may be appreciated if we describe the “naturalistic” stimulus, which was actually a series of “simulated naturalistic scenes:” “The target objects were embedded in a multifaceted cube suspended in midair in a room in which the illumination, coming from multiple light sources, varied spatially.” Each facet of the cube contained 49 differently-colored, semi-randomly assigned checks. The illumination on one side was blue or yellow, on the other “standard.” Targets were placed on top of these colored checkerboards. The “simple” stimuli were “flat patches embedded in a textured color background across which simulated illumination varied.” It is not clear why the backgrounds in the “simple” stimulus were not checkerboards matched to those of the “natural” stimuli, or, alternatively, a solid background. Even though the authors state, at one point, that they are controlling for “low-level” features of the stimuli, methodological choices are never clarified beyond the “simple/natural” dichotomy.

The authors themselves acknowledge the theoretical gap at the heart of this dichotomy: “A systematic characterization of how color constancy varies with the degree of stimulus naturalness is challenging, because a definition of naturalness remains elusive.” So, “Rather than attempting to define dimensions along which naturalness varies, we chose to study two configurations that we judged differed considerably in how natural they appeared.” Despite this vagueness in their chosen dimensions, they qualify the undefined value, “stimulus naturalness,” as a “key factor” in their results (at another point they refer to their stimulus as “nearly naturalistic” and the task as “fairly natural”). The problems here are beyond methodological, they are epistemological – failure to specify conditions in an experimental study means it does not meet the basic demands of the scientific method. Scientists go from nature to the lab in order to control and test potential causal variables; these authors are arguing that going "natural" is a merit - is somehow informative - even if they can't define what they mean by the term.

2) The authors conclude that “a reasonable degree of color constancy operates effectively in support of object selection.” (Again, why wouldn't it?). The assertion turns the scientific method on its head. This is because the surfaces and illumination were “simulated.” Thus, judgment of whether observers' choices are 'color-constant' or not hinged on assumptions about what the true reflectance of a patch should be judged to be, based on the “simulated” illumination: “To find the reflectance match for each target under a test illuminant, we [derived] a surface reflectance function...using a three-dimensional linear model for surface reflectance...derived from analysis of the spectra of Munsell papers, using the tabulated spectral data reported by Nickerson (1957)...”

Here's the problem: Let's say I take a black surface, place it on a larger grey surface, and brightly illuminate the smaller surface such that its boundary and the boundary of the illumination precisely coincide. Then, based on my knowledge of the illumination, I declare that a well-functioning lightness constancy mechanism should label the smaller surface “black.” Of course, my subjects would all fail this test. But it will not have been fair test, because observers are required to use the luminance structure of the stimulus to guess at both the reflectance and the illumination of the surfaces; they don't have my inside information about the illumination. If the stimulus itself doesn't support the inference of differential illumination, then the best inference is that illumination is uniform. Similarly, observers (their visual systems) of Radonjic et al's stimuli are called on to make guesses about both the reflectance and the illumination of target surfaces based on stimulus structure (geometric/chromatic). In order for the authors' presumed “correct” answers to constitute a valid test of observers' choices, they would have to provide a specific theory of why these answers constitute the best-guess based on stimulus structure. They would need to explicitly state their assumptions, rather than “cheating” by simply stripping away the “simulated illuminant.” And if they had made and shared any such theoretical assumptions, it would probably be more appropriate to consider observer choices a test of the theory, rather than making the predictions of the theory a test of observer choices. Because if we're studying perception, then the typical observer is never wrong.

On 2015 Sep 30, Lydia Maniatis commented:

This study has very serious problems in terms of both its purpose and its methods.

Purpose: The authors' guiding question is “how color constancy supports object selection,” an issue they claim is “not understood.” What, exactly, is not understood? If color constancy is achieved, if the percept is clear and the task unambiguous, then what do we hope to gain by having subjects choose a matching surface (out of a limited number of pre-selected options) rather thanfreely adjusting a patch, and then comparing it to a target in order to judge the correspondence between the two? Unless the means of gauging the subjects' perceptual experience is expected to affect the experience itself, then why not choose the most direct, precise, and easy-to-analyze means, rather than a cruder, ambiguous, difficult-to-analyze method deemed to be more “naturalistic”? (The authors themselves acknowledge the crudeness of their method – basically a forced-choice method - in relation to the free-choice “asymmetric matching” method).

Methods: 1) The authors' use the dimensions “naturalistic” vs “simple” as their independent variables, without explaining what they mean by the terms. The study purports to compare color comparison/selection in a “naturalistic” stimulus vs a “simple” stimulus. The problem with this may be appreciated if we describe the “naturalistic” stimulus, which was actually a series of “simulated naturalistic scenes:” “The target objects were embedded in a multifaceted cube suspended in midair in a room in which the illumination, coming from multiple light sources, varied spatially.” Each facet of the cube contained 49 differently-colored, semi-randomly assigned checks. The illumination on one side was blue or yellow, on the other “standard.” Targets were placed on top of these colored checkerboards. The “simple” stimuli were “flat patches embedded in a textured color background across which simulated illumination varied.” It is not clear why the backgrounds in the “simple” stimulus were not checkerboards matched to those of the “natural” stimuli, or, alternatively, a solid background. Even though the authors state, at one point, that they are controlling for “low-level” features of the stimuli, methodological choices are never clarified beyond the “simple/natural” dichotomy.

The authors themselves acknowledge the theoretical gap at the heart of this dichotomy: “A systematic characterization of how color constancy varies with the degree of stimulus naturalness is challenging, because a definition of naturalness remains elusive.” So, “Rather than attempting to define dimensions along which naturalness varies, we chose to study two configurations that we judged differed considerably in how natural they appeared.” Despite this vagueness in their chosen dimensions, they qualify the undefined value, “stimulus naturalness,” as a “key factor” in their results (at another point they refer to their stimulus as “nearly naturalistic” and the task as “fairly natural”). The problems here are beyond methodological, they are epistemological – failure to specify conditions in an experimental study means it does not meet the basic demands of the scientific method. Scientists go from nature to the lab in order to control and test potential causal variables; these authors are arguing that going "natural" is a merit - is somehow informative - even if they can't define what they mean by the term.

2) The authors conclude that “a reasonable degree of color constancy operates effectively in support of object selection.” (Again, why wouldn't it?). The assertion turns the scientific method on its head. This is because the surfaces and illumination were “simulated.” Thus, judgment of whether observers' choices are 'color-constant' or not hinged on assumptions about what the true reflectance of a patch should be judged to be, based on the “simulated” illumination: “To find the reflectance match for each target under a test illuminant, we [derived] a surface reflectance function...using a three-dimensional linear model for surface reflectance...derived from analysis of the spectra of Munsell papers, using the tabulated spectral data reported by Nickerson (1957)...”

Here's the problem: Let's say I take a black surface, place it on a larger grey surface, and brightly illuminate the smaller surface such that its boundary and the boundary of the illumination precisely coincide. Then, based on my knowledge of the illumination, I declare that a well-functioning lightness constancy mechanism should label the smaller surface “black.” Of course, my subjects would all fail this test. But it will not have been fair test, because observers are required to use the luminance structure of the stimulus to guess at both the reflectance and the illumination of the surfaces; they don't have my inside information about the illumination. If the stimulus itself doesn't support the inference of differential illumination, then the best inference is that illumination is uniform. Similarly, observers (their visual systems) of Radonjic et al's stimuli are called on to make guesses about both the reflectance and the illumination of target surfaces based on stimulus structure (geometric/chromatic). In order for the authors' presumed “correct” answers to constitute a valid test of observers' choices, they would have to provide a specific theory of why these answers constitute the best-guess based on stimulus structure. They would need to explicitly state their assumptions, rather than “cheating” by simply stripping away the “simulated illuminant.” And if they had made and shared any such theoretical assumptions, it would probably be more appropriate to consider observer choices a test of the theory, rather than making the predictions of the theory a test of observer choices. Because if we're studying perception, then the typical observer is never wrong.