On 2016 May 22, Lydia Maniatis commented:

(Fifth comment). Some more thoughts: Ariely's main concepts - sets, similarity - are vague. He could have explained what, for the purpose of his claims, he intends them to mean. But he doesn't - they're just placeholders.

What, for Ariely, is a set? He refers to "a collection of items." He challenges the idea that "representations of complex scenes still consist of many individual representations" suggesting instead that in many cases where "proximal items are somewhat similar the representation of a set may contain information about...the average size, color, orientation, aspect ratio, and shape of the items in the set and essentially no information about individual items."

So it would seem that Ariely's working definition of a "set" is a collection of proximal, somewhat similar items (which may differ in pretty much any respect, including shape). So vague - the description "somewhat similar" is absolutely without content - as to be practically useless, and as noted earlier, many or most of the implications are falsifiable. As noted above, even the author's own attempt to explain the alternative Ebbinghaus groups involves a contradiction of that definition.

In my discussion of the arbitrary methodological choices, I didn't mention the main one, which is the decision to use black circles and to focus on size. The reason he gives is that "such sets have the advantage that the members do not fall into distinct categories, as they could if they varied in color, shape, or orientation." So they would be "sets," but variations other than strictly size would be "disadvantageous." In what sense would they be disadvantageous? There's no discussion about this; it seems like an evasion.

Even limiting our discussion to size, we could introduce intractable complications that even Ariely's "novel paradigms" couldn't make fit. What if each circle consisted of a set of concentric circles, variously spaced? Would we construct an average of the envelope, an average of each subset of circles? What if the large group of circles, due to a configurational accident, visually grouped into two or three subsets (as may indeed have happened)? Would we have three different averages, and would these be averaged in turn? Etc.

On 2016 May 22, Lydia Maniatis commented:

(Fourth comment) Ariely's claims that his size results (such as they are) can be extended to shape, color, aspect ratio, orientation, are falsifiable. In the simplest case, a set of rectangles that includes one item (not corresponding to the mean) that has an aspect ratio of one will be registered and remembered with precision, biasing the results in favour of the individual items. The mean of the set, on the other hand, would most likely be less precisely registered. Familiar faces in a group will be remembered precisely, their mean, huh? A set of red, white and green objects would be registered, at least categorically with precision, their "mean" probably at all. The shapes in a set of triangles, squares, circles would be remembered precisely; their "mean," not so much.

These may seem like trivial examples, but there is nothing to exclude them from Ariely's broad and vague claims. That this vagueness is used as license to perform an odd and arbitrary experiment whose hugely variable and limited datasets may be favourably (if obscurely) interpreted doesn't let him off the hook.

On 2016 May 22, Lydia Maniatis commented:

(Third comment) Arbitrary methodological choices, data presentation

The title of the paper is “Seeing sets,” and question of interest is said to be “to determine what observers know about the members of a set and what they know about the statistical properties of the set (mean and distribution)” and that “sets of objects could be represented in a qualitatively different way than single items.”

These are quite broad objectives, and don't imply any particular methodology.

The author uses a very specific methodology that places demands on subjects that go beyond straightforward perception, and imposes conditions that compromise straightforward perception.

He uses brief presentation times, 500ms for the first figure and 500s for the comparison figure, with no blank space in between.

This means that there is time for about two saccades and fixations (barely), and involves potential masking effects. How did he choose these parameters? Do they matter? Would we expect the same results with longer presentation times? With a blank interval? Also, the circles increase in size geometrically, not linearly and the mean that we are talking about is the geometric, not the arithmetic mean. Why not a linear progression and an arithmetic mean?

It's been understood for a long time that by adjusting experimental conditions we can get pretty much any result we want, which is why such conditions should come with a theoretical rationale attached. Because the author's choices here seemingly make the task much more difficult than it should be given the goal, they need to be explained, not just flatly asserted.

Another blank check is to be found in a footnote referring to a “small study” which was the basis for selecting/describing one of the parameters in the reported study (“members differed in size from non-members by at least 18%, about three times the size-discrimination threshold for sets of same-size spots [as determined in the “small study”].” In a personal communication, Ariely told me that this study used about ten subjects, and was mostly about studying “within-subject variation.” Such a study could also have given an indication of between-subject variation. Given the link between the two studies, and the oddity of using only two observers (why only two?), some more specific info on results/methods of this "small study" would seems called for.

The presentation of the data is also rather weird. For the mean-discrimination experiment, we aren't given information on the proportions of correct answers the two observers achieved. Maybe this information is hidden in the opaque measure that is provided (and which assumes normality in the data without any justification - isn't that a problem?). This measure is the "mean-discrimination threshold" derived using "a standard profit analysis (Finney, 1971) ...to determine the mean-discrimination threshold (the standard deviation of the best-fitting cumulative normal distribution." Is best-fitting the same thing as "well-fitting"?

Given the method, and assuming that it really is the case that subjects were better at guessing the mean rather than the size of individual members, I could speculate on why that might be. First of all, observers never had to play the "yes-no" game with means (why not?). All they had to do was say "larger-smaller." This seems like an inherently easier task. Further, given the time constraint, there wasn't enough time to inspect each of the four different circle sizes individually. Observers only had time for one or two. So for half of each set, at least, they were out of luck on each trial in the individual member task. For the mean estimation, if they could learn to target the middle two on each trial - not the largest, not the smallest - then they could ballpark the mean. Who knows? The situation is as Runeson (1995) has described for a different type of vision study:

"For perception and cognition research in general, it is noteworthy that the cue-based style of theorising exhibits a certain lack of inherent correctives for ineffectual experiments because of its emphasis on weak and irregular performance. Thus, an experiment that is unsuitable in design or procedure could easily provide supportive-looking data at least as long as cues, salience limits, and trade-off functions are not specified in detail....data that represent suboptimal observer performance seem to have provided spurious support for untenable theoretical commitments,... "

On 2016 May 22, Lydia Maniatis commented:

(Second comment) Comment on section titled “The Economy of Set Representations"

Ariely begins this section by saying that: “Although the exquisite sensitivity of the human visual system ...is a source of wonder and delight, a computer can be equally sensitive. Much harder to solve, and currently far beyond the abilities of computer vision systems, is the problem of knowing what information to throw away.” He considers it problematic, therefore, that “in research in spatial vision, the focus has been more on reproducing the image exactly, on compressing the representation without losing information (e.g. Watson, 1987)” and suggests that “”If our models do not lose any of the original data, then no decisions have been made, no information has been extracted.”

These statements are strikingly naive with respect to basic theoretical and empirical facts of perception.

By “the original data” Ariely can only be referring to the photons hitting the retina. If the visual system accurately encoded the intensity of each and every one of these, then the only information it would possess would be the intensity of each and every one of these. If it recorded the relative intensities of each and every one of these, then that would the only information it would possess. Something is missing, and that is the understanding that to be informative, the “original data” must be massively leveraged, organised via complex principles which group, inflate, complete, emphasize or de-emphasize - in general interpret these points to create 3D impressions of shape, light, movement, space. The fundamental problem is not compression but “going beyond the information given.” This is a very old misunderstanding for which there's no longer any excuse, and which should not be recycled, decade after decade. To correct Ariely's statement above, “If our models do not interpret the original data, no information has been created.” The processes Ariely is positing are actually post-organisation, his "original data" are the visual system's perceptual constructs, that are then to be averaged, or whatever. This is not less work, it is more.

Finally, “The reduction of a set of similar items to a mean (or prototypical value), a range, and a few other important statistical properties may preserve just the information needed to navigate in the real world, to form a global percept, and to identify candidate locations of interest.”

Leaving aside the vagueness of the term “set,” the idea that sets of “similar” (how similar?) objects – a set of chairs, a group of kids – is reduced to a group mean and a range defies both experience and logic. The most important units in our environment are objects, not loose collections thereof. We see items, groups of items, but I have never seen a mean of a group of items. The idea that our perception of such groups turns into a random hall of mirrors of sizes, colors, shapes when they are simply near each other...where does that come from?

On 2016 May 18, Lydia Maniatis commented:

This highly-cited article is celebrated as seminal, pioneering, influential. (See for example Bauer, 2015). In contrast, I find the arguments facile, incomplete, and inadequate to support a bold but very vague and seemingly implausible proposal. I'll deal with separate issues in separate comments here, beginning with the one that is easiest for me to make. It deals with the second to last section of the paper, titled "Visual Contrast Illusions and Set Representations." The style of argument of this section is a good illustration of the casualness of the authors arguments in general.

Here's my comment:

Ariely argues that the Ebbinghaus illusion is amenable to explanation in terms of “set representation.” Specifically, he says that the circle surrounded by large circles appears smaller than the one surrounded by small circles because each is “judged relative to the set properties of the circles surrounding it. If the basic representation of a set contains the relationship of an object to the mean of its set, it follows that the [former] will be judged smaller than the [latter].”

Assuming that what applies to the central circle applies to every circle, Ariely is saying that the “basic representation of a set” includes, in its composition, the relationship of each object to the mean of the set. That means five sub-representations in addition to the representation of the mean size of the set and of each individual member (adjusted for set mean). And since he also suggests that mean representations are created for other features as well, e.g. color, shape, we would have to add these to the hypothetical representations. (If efficiency of representation is the goal, this does not seem to fit the bill.)

Ariely even presents a kind of test of his claim by citing a study that showed that when the central circles are surrounded by triangles, the effect is attenuated. This is no where near good enough. Many logical and empirical falsifications of his thesis are close to hand.

Ariely overlooks the fundamental fact that the Ebbinghaus illusion is contingent on configuration. It should be pretty clear that it would be a trivial project create sets of circles (putting them in a row, for example, and adjusting which is adjacent to which) for which his prediction would fail.

In addition, while he only focusses on size, Ariely does not limit his claims to size. But it is also trivial to show that it doesn't work for, e.g. lightness, color, or shape even to the extent that it might be can be made to appear to work for size. There are countless demonstrations, involving sets of discrete figures, of so-called assimilation effects in which this explanation, involving a “contrast” effect, would fail. And it obviously fails for shape.

So we are dealing with a facile, ad hoc explanation that should at least have been challenged during review, and certainly not have been admitted as a serious argument after the fact.

Finally, it's worth pointing out that the Ebbinghaus effect is evident and robust under leisurely inspection. If the effects Ariely is proposing are this salient, why does he choose to present his multi-item stimuli using only brief presentation times (thus allowing time for only a couple of saccades) and without a blank period before (briefly) presenting the comparison figures (thus incurring possible masking effects)? Why use sub-optimal conditions rather than make his case using longer presentation times and clear and robust perceptual effects such as the Ebbinghaus? Nowhere does he explain his methodological choices.

But the priority is to explain the easy falsifications, otherwise the there is no viable hypothesis.

p.s. More conceptual inconsistencies: In his account of the Ebbinghaus, Ariely is saying that the triangles are not perceived as part of a set including the central circle. Yet a. they do group with it visually, and b. earlier in the article he says that "orientation, aspect ration, mean hue, and shape (however represented" may all be set properties." So why don't we see the circle changing shape, and why does shape difference preclude set affiliation?

On 2016 May 18, Lydia Maniatis commented:

This highly-cited article is celebrated as seminal, pioneering, influential. (See for example Bauer, 2015). In contrast, I find the arguments facile, incomplete, and inadequate to support a bold but very vague and seemingly implausible proposal. I'll deal with separate issues in separate comments here, beginning with the one that is easiest for me to make. It deals with the second to last section of the paper, titled "Visual Contrast Illusions and Set Representations." The style of argument of this section is a good illustration of the casualness of the authors arguments in general.

Here's my comment:

Ariely argues that the Ebbinghaus illusion is amenable to explanation in terms of “set representation.” Specifically, he says that the circle surrounded by large circles appears smaller than the one surrounded by small circles because each is “judged relative to the set properties of the circles surrounding it. If the basic representation of a set contains the relationship of an object to the mean of its set, it follows that the [former] will be judged smaller than the [latter].”

Assuming that what applies to the central circle applies to every circle, Ariely is saying that the “basic representation of a set” includes, in its composition, the relationship of each object to the mean of the set. That means five sub-representations in addition to the representation of the mean size of the set and of each individual member (adjusted for set mean). And since he also suggests that mean representations are created for other features as well, e.g. color, shape, we would have to add these to the hypothetical representations. (If efficiency of representation is the goal, this does not seem to fit the bill.)

Ariely even presents a kind of test of his claim by citing a study that showed that when the central circles are surrounded by triangles, the effect is attenuated. This is no where near good enough. Many logical and empirical falsifications of his thesis are close to hand.

Ariely overlooks the fundamental fact that the Ebbinghaus illusion is contingent on configuration. It should be pretty clear that it would be a trivial project create sets of circles (putting them in a row, for example, and adjusting which is adjacent to which) for which his prediction would fail.

In addition, while he only focusses on size, Ariely does not limit his claims to size. But it is also trivial to show that it doesn't work for, e.g. lightness, color, or shape even to the extent that it might be can be made to appear to work for size. There are countless demonstrations, involving sets of discrete figures, of so-called assimilation effects in which this explanation, involving a “contrast” effect, would fail. And it obviously fails for shape.

So we are dealing with a facile, ad hoc explanation that should at least have been challenged during review, and certainly not have been admitted as a serious argument after the fact.

Finally, it's worth pointing out that the Ebbinghaus effect is evident and robust under leisurely inspection. If the effects Ariely is proposing are this salient, why does he choose to present his multi-item stimuli using only brief presentation times (thus allowing time for only a couple of saccades) and without a blank period before (briefly) presenting the comparison figures (thus incurring possible masking effects)? Why use sub-optimal conditions rather than make his case using longer presentation times and clear and robust perceptual effects such as the Ebbinghaus? Nowhere does he explain his methodological choices.

But the priority is to explain the easy falsifications, otherwise the there is no viable hypothesis.

p.s. More conceptual inconsistencies: In his account of the Ebbinghaus, Ariely is saying that the triangles are not perceived as part of a set including the central circle. Yet a. they do group with it visually, and b. earlier in the article he says that "orientation, aspect ration, mean hue, and shape (however represented" may all be set properties." So why don't we see the circle changing shape, and why does shape difference preclude set affiliation?

On 2016 May 22, Lydia Maniatis commented:

(Second comment) Comment on section titled “The Economy of Set Representations"

Ariely begins this section by saying that: “Although the exquisite sensitivity of the human visual system ...is a source of wonder and delight, a computer can be equally sensitive. Much harder to solve, and currently far beyond the abilities of computer vision systems, is the problem of knowing what information to throw away.” He considers it problematic, therefore, that “in research in spatial vision, the focus has been more on reproducing the image exactly, on compressing the representation without losing information (e.g. Watson, 1987)” and suggests that “”If our models do not lose any of the original data, then no decisions have been made, no information has been extracted.”

These statements are strikingly naive with respect to basic theoretical and empirical facts of perception.

By “the original data” Ariely can only be referring to the photons hitting the retina. If the visual system accurately encoded the intensity of each and every one of these, then the only information it would possess would be the intensity of each and every one of these. If it recorded the relative intensities of each and every one of these, then that would the only information it would possess. Something is missing, and that is the understanding that to be informative, the “original data” must be massively leveraged, organised via complex principles which group, inflate, complete, emphasize or de-emphasize - in general interpret these points to create 3D impressions of shape, light, movement, space. The fundamental problem is not compression but “going beyond the information given.” This is a very old misunderstanding for which there's no longer any excuse, and which should not be recycled, decade after decade. To correct Ariely's statement above, “If our models do not interpret the original data, no information has been created.” The processes Ariely is positing are actually post-organisation, his "original data" are the visual system's perceptual constructs, that are then to be averaged, or whatever. This is not less work, it is more.

Finally, “The reduction of a set of similar items to a mean (or prototypical value), a range, and a few other important statistical properties may preserve just the information needed to navigate in the real world, to form a global percept, and to identify candidate locations of interest.”

Leaving aside the vagueness of the term “set,” the idea that sets of “similar” (how similar?) objects – a set of chairs, a group of kids – is reduced to a group mean and a range defies both experience and logic. The most important units in our environment are objects, not loose collections thereof. We see items, groups of items, but I have never seen a mean of a group of items. The idea that our perception of such groups turns into a random hall of mirrors of sizes, colors, shapes when they are simply near each other...where does that come from?

On 2016 May 22, Lydia Maniatis commented:

(Third comment) Arbitrary methodological choices, data presentation

The title of the paper is “Seeing sets,” and question of interest is said to be “to determine what observers know about the members of a set and what they know about the statistical properties of the set (mean and distribution)” and that “sets of objects could be represented in a qualitatively different way than single items.”

These are quite broad objectives, and don't imply any particular methodology.

The author uses a very specific methodology that places demands on subjects that go beyond straightforward perception, and imposes conditions that compromise straightforward perception.

He uses brief presentation times, 500ms for the first figure and 500s for the comparison figure, with no blank space in between.

This means that there is time for about two saccades and fixations (barely), and involves potential masking effects. How did he choose these parameters? Do they matter? Would we expect the same results with longer presentation times? With a blank interval? Also, the circles increase in size geometrically, not linearly and the mean that we are talking about is the geometric, not the arithmetic mean. Why not a linear progression and an arithmetic mean?

It's been understood for a long time that by adjusting experimental conditions we can get pretty much any result we want, which is why such conditions should come with a theoretical rationale attached. Because the author's choices here seemingly make the task much more difficult than it should be given the goal, they need to be explained, not just flatly asserted.

Another blank check is to be found in a footnote referring to a “small study” which was the basis for selecting/describing one of the parameters in the reported study (“members differed in size from non-members by at least 18%, about three times the size-discrimination threshold for sets of same-size spots [as determined in the “small study”].” In a personal communication, Ariely told me that this study used about ten subjects, and was mostly about studying “within-subject variation.” Such a study could also have given an indication of between-subject variation. Given the link between the two studies, and the oddity of using only two observers (why only two?), some more specific info on results/methods of this "small study" would seems called for.

The presentation of the data is also rather weird. For the mean-discrimination experiment, we aren't given information on the proportions of correct answers the two observers achieved. Maybe this information is hidden in the opaque measure that is provided (and which assumes normality in the data without any justification - isn't that a problem?). This measure is the "mean-discrimination threshold" derived using "a standard profit analysis (Finney, 1971) ...to determine the mean-discrimination threshold (the standard deviation of the best-fitting cumulative normal distribution." Is best-fitting the same thing as "well-fitting"?

Given the method, and assuming that it really is the case that subjects were better at guessing the mean rather than the size of individual members, I could speculate on why that might be. First of all, observers never had to play the "yes-no" game with means (why not?). All they had to do was say "larger-smaller." This seems like an inherently easier task. Further, given the time constraint, there wasn't enough time to inspect each of the four different circle sizes individually. Observers only had time for one or two. So for half of each set, at least, they were out of luck on each trial in the individual member task. For the mean estimation, if they could learn to target the middle two on each trial - not the largest, not the smallest - then they could ballpark the mean. Who knows? The situation is as Runeson (1995) has described for a different type of vision study:

"For perception and cognition research in general, it is noteworthy that the cue-based style of theorising exhibits a certain lack of inherent correctives for ineffectual experiments because of its emphasis on weak and irregular performance. Thus, an experiment that is unsuitable in design or procedure could easily provide supportive-looking data at least as long as cues, salience limits, and trade-off functions are not specified in detail....data that represent suboptimal observer performance seem to have provided spurious support for untenable theoretical commitments,... "

On 2016 May 22, Lydia Maniatis commented:

(Fourth comment) Ariely's claims that his size results (such as they are) can be extended to shape, color, aspect ratio, orientation, are falsifiable. In the simplest case, a set of rectangles that includes one item (not corresponding to the mean) that has an aspect ratio of one will be registered and remembered with precision, biasing the results in favour of the individual items. The mean of the set, on the other hand, would most likely be less precisely registered. Familiar faces in a group will be remembered precisely, their mean, huh? A set of red, white and green objects would be registered, at least categorically with precision, their "mean" probably at all. The shapes in a set of triangles, squares, circles would be remembered precisely; their "mean," not so much.

These may seem like trivial examples, but there is nothing to exclude them from Ariely's broad and vague claims. That this vagueness is used as license to perform an odd and arbitrary experiment whose hugely variable and limited datasets may be favourably (if obscurely) interpreted doesn't let him off the hook.

On 2016 May 22, Lydia Maniatis commented:

(Fifth comment). Some more thoughts: Ariely's main concepts - sets, similarity - are vague. He could have explained what, for the purpose of his claims, he intends them to mean. But he doesn't - they're just placeholders.

What, for Ariely, is a set? He refers to "a collection of items." He challenges the idea that "representations of complex scenes still consist of many individual representations" suggesting instead that in many cases where "proximal items are somewhat similar the representation of a set may contain information about...the average size, color, orientation, aspect ratio, and shape of the items in the set and essentially no information about individual items."

So it would seem that Ariely's working definition of a "set" is a collection of proximal, somewhat similar items (which may differ in pretty much any respect, including shape). So vague - the description "somewhat similar" is absolutely without content - as to be practically useless, and as noted earlier, many or most of the implications are falsifiable. As noted above, even the author's own attempt to explain the alternative Ebbinghaus groups involves a contradiction of that definition.

In my discussion of the arbitrary methodological choices, I didn't mention the main one, which is the decision to use black circles and to focus on size. The reason he gives is that "such sets have the advantage that the members do not fall into distinct categories, as they could if they varied in color, shape, or orientation." So they would be "sets," but variations other than strictly size would be "disadvantageous." In what sense would they be disadvantageous? There's no discussion about this; it seems like an evasion.

Even limiting our discussion to size, we could introduce intractable complications that even Ariely's "novel paradigms" couldn't make fit. What if each circle consisted of a set of concentric circles, variously spaced? Would we construct an average of the envelope, an average of each subset of circles? What if the large group of circles, due to a configurational accident, visually grouped into two or three subsets (as may indeed have happened)? Would we have three different averages, and would these be averaged in turn? Etc.