On 2013 Dec 09, Rhodri Cusack commented:

Neuroimaging can be used to probe cognitive processes like language development at an age when behavioral responses are not yet available. Mueller JL, 2012's report describes the use of electroencephalography (EEG) to investigate individual differences in grammatical rule learning in 3-month old infants.

In their auditory oddball paradigm, triplets of syllables were presented in a sequence. Approximately 10% of triplets contained a final syllable that deviated in pitch and ~10% of triplets deviated in the rule with which the syllables were chosen. Mismatch Responses (MMRs) to pitch deviants have been shown in young infants before (He C, 2007). Early in development (2-3 months), deviants evoke a slowly changing positive MMR and this typically matures by four months to a more rapid, negative MMR, as seen in adults.

The report presents data from 65 infants of age three months. It concludes that pitch deviants elicit an immature positive MMR in 49% of the infants and a negative MMR in the remainder; and that rule deviants elicit a response only in infants with a mature negative pitch MMR. Together, these are taken to indicate that abstract rule learning depends on basic auditory processing abilities. However, the statistics reported do not justify these conclusions.

As the infants were split into groups according to the sign of their pitch MMR, it is circular to then test for a non-zero pitch MMR within each of these groups. The tests have a high risk of Type 2 error, and significant results would be obtained under the null hypothesis that the data comprised noise alone (Kriegeskorte N, 2009). This makes it difficult to interpret the blue waveforms and bars in Fig 2 A,B,C&D, the PxMMR interaction, or the Neg MMR measures in Table 1. Thus, evidence was not provided for the presence of a pitch MMR, although anecdotally, the shapes of the positive and negative curves are similar to those observed previously (He C, 2007). The interactions involving pitch MMR and region will also be biased in the likely circumstance that noise level varies by region. To assess the statistical significance of the pitch MMR (or higher-level interactions), cross validation could be performed by splitting each participant’s data (e.g., assign groups using half of the data, run statistics on the other half)

Statistics were also not presented to show that the positive and negative pitch-MMR groups had a different MMR response to rule deviants. The rule MMR reached significance in the negative group and not in the positive group, but tests were not reported to establish these differed significantly (i.e., Boys: MMRxR; Girls: MMRxR) (Nieuwenhuis S, 2011). The higher-level interaction reported (MMRxRxSex) might be driven by the opposite signs of the boys and girls with negative MMR. Further, the positive rule MMR appears to have different morphology from that previously reported (He C, 2007).

In summary, the conclusions stated are not justified given the reported statistics. Rigorous statistical practices are critical for guiding the interpretation of neuroimaging data, particularly as prominent researchers have suggested only half of all studies will replicate.

Thanks to Bobby Stojanoski, Michelle C. Tran, Annika C. Linke & Conor J. Wild from the Cusack lab for their help in drafting this comment.

On 2013 Dec 09, Rhodri Cusack commented:

Neuroimaging can be used to probe cognitive processes like language development at an age when behavioral responses are not yet available. Mueller JL, 2012's report describes the use of electroencephalography (EEG) to investigate individual differences in grammatical rule learning in 3-month old infants.

In their auditory oddball paradigm, triplets of syllables were presented in a sequence. Approximately 10% of triplets contained a final syllable that deviated in pitch and ~10% of triplets deviated in the rule with which the syllables were chosen. Mismatch Responses (MMRs) to pitch deviants have been shown in young infants before (He C, 2007). Early in development (2-3 months), deviants evoke a slowly changing positive MMR and this typically matures by four months to a more rapid, negative MMR, as seen in adults.

The report presents data from 65 infants of age three months. It concludes that pitch deviants elicit an immature positive MMR in 49% of the infants and a negative MMR in the remainder; and that rule deviants elicit a response only in infants with a mature negative pitch MMR. Together, these are taken to indicate that abstract rule learning depends on basic auditory processing abilities. However, the statistics reported do not justify these conclusions.

As the infants were split into groups according to the sign of their pitch MMR, it is circular to then test for a non-zero pitch MMR within each of these groups. The tests have a high risk of Type 2 error, and significant results would be obtained under the null hypothesis that the data comprised noise alone (Kriegeskorte N, 2009). This makes it difficult to interpret the blue waveforms and bars in Fig 2 A,B,C&D, the PxMMR interaction, or the Neg MMR measures in Table 1. Thus, evidence was not provided for the presence of a pitch MMR, although anecdotally, the shapes of the positive and negative curves are similar to those observed previously (He C, 2007). The interactions involving pitch MMR and region will also be biased in the likely circumstance that noise level varies by region. To assess the statistical significance of the pitch MMR (or higher-level interactions), cross validation could be performed by splitting each participant’s data (e.g., assign groups using half of the data, run statistics on the other half)

Statistics were also not presented to show that the positive and negative pitch-MMR groups had a different MMR response to rule deviants. The rule MMR reached significance in the negative group and not in the positive group, but tests were not reported to establish these differed significantly (i.e., Boys: MMRxR; Girls: MMRxR) (Nieuwenhuis S, 2011). The higher-level interaction reported (MMRxRxSex) might be driven by the opposite signs of the boys and girls with negative MMR. Further, the positive rule MMR appears to have different morphology from that previously reported (He C, 2007).

In summary, the conclusions stated are not justified given the reported statistics. Rigorous statistical practices are critical for guiding the interpretation of neuroimaging data, particularly as prominent researchers have suggested only half of all studies will replicate.

Thanks to Bobby Stojanoski, Michelle C. Tran, Annika C. Linke & Conor J. Wild from the Cusack lab for their help in drafting this comment.