On 2016 Nov 30, Lydia Maniatis commented:

Do you know what "p-hacking" means? I think that's what you're labelling as "data-driven." Are you aware that there have been six news reports amplifying the uncorroborated claims in your title? Sorry, but your title should have been, "We collected a lot of data."

On 2016 Nov 30, Antoine Coutrot commented:

Dear Lydia,

thank you for highlighting our limitation section, it is indeed quite important. You are absolutely right, our method is too confounded to allow us to draw any general conclusion. As are all experiments in cognitive science. They are all limited by the sample size, by the participant profile, by the task... But we try and do our best. For instance we collected 400+ participants from 58 nationalities, more than any eye-tracking experiment ever published. The main points of the paper are 1- gaze contains a wealth of information about the observer 2- with a big and diverse eye database it is possible to capture in a data-driven fashion what demographics explain different gaze patterns. Here, it happens to be the gender, hence the title.

On 2016 Nov 29, Lydia Maniatis commented:

Anyone familiar with the vision science literature should know by now that the best place to start reading a published paper is the section at the tail end titled “Limitations of the study.” This is where we can see whether the titular claims have any connection to what the study is actually entitled to report in terms of findings.

Compare, for example, the title of this study with its “limitations” section, quoted below in full (caps mine):

“The authors would like to make clear that THIS STUDY DOES NOT DEMONSTRATE THAT GENDER IS THE VARIABLE THAT MOST INFLUENCES GAZE PATTERNS DURING FACE EXPLORATION in general. [i.e. our method is too confounded to allow us to draw any such conclusion in principle].Many aspects of the experimental design might have influenced the results presented in this paper. The actors we used were all Caucasian between 20 and 40 years old with a neutral expression and did not speak—all factors that could have influenced observers' strategies (Coutrot & Guyader, 2014; Schurgin et al., 2014; Wheeler et al., 2011). Even the initial gaze position has been shown to have a significant impact on the following scanpaths (Arizpe et al., 2012; Arizpe et al., 2015). In particular, the task given to the participants—rating the level of comfort they felt with the actor's duration of direct gaze—would certainly bias participants' attention toward actors' eyes. One of the first eye-tracking experiments in history suggested that gaze patterns are strongly modulated by different task demands (Yarbus, 1965). This result has since been replicated and extended: More recent studies showed that the task at hand can even be inferred using gaze-based classifiers (Boisvert & Bruce, 2016; Borji & Itti, 2014; Haji-Abolhassani & Clark, 2014; Kanan et al., 2015). Here, gender appears to be the variable that produces the strongest differences between participants. But one could legitimately hypothesize that if the task had been to determine the emotion displayed by the actors' face, the culture of the observer could have played a more important role as it has been shown that the way we perceive facial expression is not universal (Jack, Blais, Scheepers, Schyns, & Caldara, 2009). Considering the above, the key message of this paper is that our method allows capturing systematic differences between groups of observers in a data-driven fashion.”

In other words, this is not a research paper, but a preliminary application of a method that might be useful for a research study. As far as the reported results go, it is an exercise in p-hacking. The title is purely cosmetic. As such it seems to have been rather effective, insofar as the article has already been the subject of six news stories, including reports in the Daily Mail and Le Monde.

On 2016 Nov 29, Lydia Maniatis commented:

Anyone familiar with the vision science literature should know by now that the best place to start reading a published paper is the section at the tail end titled “Limitations of the study.” This is where we can see whether the titular claims have any connection to what the study is actually entitled to report in terms of findings.

Compare, for example, the title of this study with its “limitations” section, quoted below in full (caps mine):

“The authors would like to make clear that THIS STUDY DOES NOT DEMONSTRATE THAT GENDER IS THE VARIABLE THAT MOST INFLUENCES GAZE PATTERNS DURING FACE EXPLORATION in general. [i.e. our method is too confounded to allow us to draw any such conclusion in principle].Many aspects of the experimental design might have influenced the results presented in this paper. The actors we used were all Caucasian between 20 and 40 years old with a neutral expression and did not speak—all factors that could have influenced observers' strategies (Coutrot & Guyader, 2014; Schurgin et al., 2014; Wheeler et al., 2011). Even the initial gaze position has been shown to have a significant impact on the following scanpaths (Arizpe et al., 2012; Arizpe et al., 2015). In particular, the task given to the participants—rating the level of comfort they felt with the actor's duration of direct gaze—would certainly bias participants' attention toward actors' eyes. One of the first eye-tracking experiments in history suggested that gaze patterns are strongly modulated by different task demands (Yarbus, 1965). This result has since been replicated and extended: More recent studies showed that the task at hand can even be inferred using gaze-based classifiers (Boisvert & Bruce, 2016; Borji & Itti, 2014; Haji-Abolhassani & Clark, 2014; Kanan et al., 2015). Here, gender appears to be the variable that produces the strongest differences between participants. But one could legitimately hypothesize that if the task had been to determine the emotion displayed by the actors' face, the culture of the observer could have played a more important role as it has been shown that the way we perceive facial expression is not universal (Jack, Blais, Scheepers, Schyns, & Caldara, 2009). Considering the above, the key message of this paper is that our method allows capturing systematic differences between groups of observers in a data-driven fashion.”

In other words, this is not a research paper, but a preliminary application of a method that might be useful for a research study. As far as the reported results go, it is an exercise in p-hacking. The title is purely cosmetic. As such it seems to have been rather effective, insofar as the article has already been the subject of six news stories, including reports in the Daily Mail and Le Monde.