On 2015 Sep 01, E Schuman commented:

Critical Flaws in Ainsley et al., “Functionally diverse dendritic mRNAs rapidly associate with ribosomes following a novel experience”, DOI: 10.1038/ncomms5510

Submitted by: Georgi Tushev and Erin Schuman (Max Planck Institute for Brain Research, Frankfurt am Main, Germany) and Wei Chen (Max Delbruck Center, Berlin, Germany).

http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf

In this paper the authors conduct a fear conditioning experiment (“novel-experience”), isolate ribosome-bound mRNA from the dendrites of mouse hippocampal neurons, conduct deep RNA sequencing and then use a machine-learning classification scheme to identify the mRNAs present in dendrites following learning. Upon close reading and re-analysis of the published data we found substantive flaws including the absence of an analyzed control group, low sequencing depth, high variability between replicas, and poor classification of genes. We discuss these issues in greater detail below.

Absence of a completely analyzed control group. In biological experiments, the assessment of statistically significant differences between groups (e.g. control and experimental groups, drawn from a normal distribution) uses information about the variability in each group in order to estimate the likelihood that between-group differences occurred by chance. Ainsley et al. attempted to compare ribosome-bound RNA from control ("home-cage") and "novel- experience" animals, ignoring their own observations (see Ainsley et al Fig. S2) that the variation among the biological replicates within each group is as large as the variation between the two conditions. Moreover, the authors themselves indicate that their machine learning classifier was unable to classify their home-cage data and as such, do not analyze the “control” group. Thus, without further statistical assessment, it is not appropriate to make general statements about the translational regulation induced upon “novel experience” (fear-conditioning) because there is no control group data analysis to compare it to.

Low sequencing depth compromises reproducibility and leads to high replicate variability. In the analysis of deep-sequencing datasets, one maps the sequenced reads to a reference genome to identify the gene from which the transcript originated. With current RNA extraction, library preparation and sequencing techniques, the generally accepted lower-bound of “mappable” reads for an analyzable data set is around 75-80% (Sims et al.). In Ainsley et al. the average fraction of reads which yield useful information in all samples is low, (47.4%) leading to low sequencing depth; the analysis reported in Ainsley et al and our own re-analysis of the data agree on this (see our Figure 1A and B). Indeed, the fraction of mapped reads in the immunoprecipitated (IP) fraction (“ribosome-bound”) (which the authors focus their analysis on) is even lower than all other groups (< 40%) (our Figure 1A and B, http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf). For comparison, a very similar recent study (using the same technique with a brain sample) showed a much higher fraction of mapped reads (Hupe et al., 2014). Although the authors report the use of a ribosomal RNA-depleted sample, (the Ovation RNA-Seq kit used for RNA amplification is not supposed to bind ribosomal RNA), a substantial fraction of their reads actually map to ribosomal RNA (Figure 1B). As a consequence, the genes detected in the sequencing replicates show very low gene overlap and the estimated RNA abundance is also poorly correlated (~ range 0.32 -0.72), again particularly in the IP fraction which is the focus of Ainsley et al.’s analysis (range 0.32-0.42) (our Figure 1C and D). For comparison, Hupe et al reported high replicability for their brain translatome data with all correlation values in excess of 0.98 (Hupe et al., 2014). Incorrect gene feature representation: Inclusion of non-coding RNA reads as mRNA reads. In the analysis of their data, Ainsley et al. map the sequencing reads to different domains of a gene e.g. reads aligned to the 5’UTR (untranslated region), coding sequence (CDS) and 3’UTR. In Figure 2a they report, surprisingly, that in different groups, between 22-80% of the reads map to the 3’UTR. Their annotation is in stark contrast to other published studies where, using similar techniques, an even distribution of reads throughout the gene body is typically observed (e.g. Hupe et al., 2014). In our Figure 2A (http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf), using the raw published data, Ainsley et al.’s reads are independently mapped and the observed fractional annotation fails to confirm Ainsley’s report (our Figure 2A; compare to original Figure 2A in Ainsley et al). Indeed, the only way one can replicate their skewed (huge contribution of the 3’UTR reads) distribution is to include reads that map to non-coding RNA loci and erroneously assign them to the 3’UTR (our Figure 2B). The correct annotation of read position is important because Ainsley et al. go on to use these read positions in genes to train a classifier to identify mRNAs as pyramidal (+pyr) or non-pyramidal (-pyr).

Flawed input, flawed design, and flawed output of a gene classifier. Ainsley et al used a training gene set to train a classifier to recognize transcripts as those transcripts arising from dendrites of pyramidal neurons (+pyr) or other cells in the neuropil (-pyr). The training set comprised 74 (+pyr) and 124 (-pyr) genes that the authors compiled by their own visual inspection of the Allen Brain Atlas. Unfortunately a significant fraction of these genes are completely absent from the author’s own analyzed dataset (40/74 +pyr genes and 38/123 -pyr genes missing, Supplementary Data 2 in Ainsley et al.). Furthermore, the authors propose, without any data to substantiate it, that reads from +pyr genes should be CDS-enriched and 3’UTR- de-enriched in their IP (“ribosome-bound”) fraction, relative to the supernatant fraction. Recreating the clustering procedure of Ainsley et al with adequate accuracy one observes that the genes are separated only by expression dominance in the IP fraction, independent of CDS or 3’UTR enrichment (Figure 2C). This graph reveals absolutely no clustering of genes, as would be expected if the author’s original formulation/idea were correct. Furthermore, we attempted to recreate the clustering using different pairs of biological replicates provided in the Ainsley et al study. The majority of replicates samples fail to successfully cluster the data with the same precision and accuracy as expected (clustering measures <0.5, Figure 2D). The above result is not surprising given the overall poor data quality, depth and expression correlation. This offers one explanation for why the authors were unable to classify their home-cage “control” dataset (Ainsley et al. page 4). In addition, we compiled a list of 139 in situ hybridization-verified dendritic mRNAs from several sources (Lein et al., Cajigas et al and references therein) and compared it to the “+pyr“ and “–pyr“ classified set suggested by Ainsley et al. (Figure 2E and 2F, Table 1 http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf). We found that less than 35% of the dendritically-verified transcripts (from the above lists) were included in the Ainsley et al., (+pyr) list, the majority were in fact listed on the -pyr list. Even 16 genes that the Allen Brain Atlas team indicated as “dendritic” (e.g. Lein et al.,) were mis-classified by Ainsley et al. as (–pyr) (Table 1). In summary, the study of Ainsley et al., is flawed at many levels, including experimental design, poor data quality and incorrect analyses, invalidating many, if not all, of the conclusions of the authors.

On 2015 Sep 01, E Schuman commented:

Critical Flaws in Ainsley et al., “Functionally diverse dendritic mRNAs rapidly associate with ribosomes following a novel experience”, DOI: 10.1038/ncomms5510

Submitted by: Georgi Tushev and Erin Schuman (Max Planck Institute for Brain Research, Frankfurt am Main, Germany) and Wei Chen (Max Delbruck Center, Berlin, Germany).

http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf

In this paper the authors conduct a fear conditioning experiment (“novel-experience”), isolate ribosome-bound mRNA from the dendrites of mouse hippocampal neurons, conduct deep RNA sequencing and then use a machine-learning classification scheme to identify the mRNAs present in dendrites following learning. Upon close reading and re-analysis of the published data we found substantive flaws including the absence of an analyzed control group, low sequencing depth, high variability between replicas, and poor classification of genes. We discuss these issues in greater detail below.

Absence of a completely analyzed control group. In biological experiments, the assessment of statistically significant differences between groups (e.g. control and experimental groups, drawn from a normal distribution) uses information about the variability in each group in order to estimate the likelihood that between-group differences occurred by chance. Ainsley et al. attempted to compare ribosome-bound RNA from control ("home-cage") and "novel- experience" animals, ignoring their own observations (see Ainsley et al Fig. S2) that the variation among the biological replicates within each group is as large as the variation between the two conditions. Moreover, the authors themselves indicate that their machine learning classifier was unable to classify their home-cage data and as such, do not analyze the “control” group. Thus, without further statistical assessment, it is not appropriate to make general statements about the translational regulation induced upon “novel experience” (fear-conditioning) because there is no control group data analysis to compare it to.

Low sequencing depth compromises reproducibility and leads to high replicate variability. In the analysis of deep-sequencing datasets, one maps the sequenced reads to a reference genome to identify the gene from which the transcript originated. With current RNA extraction, library preparation and sequencing techniques, the generally accepted lower-bound of “mappable” reads for an analyzable data set is around 75-80% (Sims et al.). In Ainsley et al. the average fraction of reads which yield useful information in all samples is low, (47.4%) leading to low sequencing depth; the analysis reported in Ainsley et al and our own re-analysis of the data agree on this (see our Figure 1A and B). Indeed, the fraction of mapped reads in the immunoprecipitated (IP) fraction (“ribosome-bound”) (which the authors focus their analysis on) is even lower than all other groups (< 40%) (our Figure 1A and B, http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf). For comparison, a very similar recent study (using the same technique with a brain sample) showed a much higher fraction of mapped reads (Hupe et al., 2014). Although the authors report the use of a ribosomal RNA-depleted sample, (the Ovation RNA-Seq kit used for RNA amplification is not supposed to bind ribosomal RNA), a substantial fraction of their reads actually map to ribosomal RNA (Figure 1B). As a consequence, the genes detected in the sequencing replicates show very low gene overlap and the estimated RNA abundance is also poorly correlated (~ range 0.32 -0.72), again particularly in the IP fraction which is the focus of Ainsley et al.’s analysis (range 0.32-0.42) (our Figure 1C and D). For comparison, Hupe et al reported high replicability for their brain translatome data with all correlation values in excess of 0.98 (Hupe et al., 2014). Incorrect gene feature representation: Inclusion of non-coding RNA reads as mRNA reads. In the analysis of their data, Ainsley et al. map the sequencing reads to different domains of a gene e.g. reads aligned to the 5’UTR (untranslated region), coding sequence (CDS) and 3’UTR. In Figure 2a they report, surprisingly, that in different groups, between 22-80% of the reads map to the 3’UTR. Their annotation is in stark contrast to other published studies where, using similar techniques, an even distribution of reads throughout the gene body is typically observed (e.g. Hupe et al., 2014). In our Figure 2A (http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf), using the raw published data, Ainsley et al.’s reads are independently mapped and the observed fractional annotation fails to confirm Ainsley’s report (our Figure 2A; compare to original Figure 2A in Ainsley et al). Indeed, the only way one can replicate their skewed (huge contribution of the 3’UTR reads) distribution is to include reads that map to non-coding RNA loci and erroneously assign them to the 3’UTR (our Figure 2B). The correct annotation of read position is important because Ainsley et al. go on to use these read positions in genes to train a classifier to identify mRNAs as pyramidal (+pyr) or non-pyramidal (-pyr).

Flawed input, flawed design, and flawed output of a gene classifier. Ainsley et al used a training gene set to train a classifier to recognize transcripts as those transcripts arising from dendrites of pyramidal neurons (+pyr) or other cells in the neuropil (-pyr). The training set comprised 74 (+pyr) and 124 (-pyr) genes that the authors compiled by their own visual inspection of the Allen Brain Atlas. Unfortunately a significant fraction of these genes are completely absent from the author’s own analyzed dataset (40/74 +pyr genes and 38/123 -pyr genes missing, Supplementary Data 2 in Ainsley et al.). Furthermore, the authors propose, without any data to substantiate it, that reads from +pyr genes should be CDS-enriched and 3’UTR- de-enriched in their IP (“ribosome-bound”) fraction, relative to the supernatant fraction. Recreating the clustering procedure of Ainsley et al with adequate accuracy one observes that the genes are separated only by expression dominance in the IP fraction, independent of CDS or 3’UTR enrichment (Figure 2C). This graph reveals absolutely no clustering of genes, as would be expected if the author’s original formulation/idea were correct. Furthermore, we attempted to recreate the clustering using different pairs of biological replicates provided in the Ainsley et al study. The majority of replicates samples fail to successfully cluster the data with the same precision and accuracy as expected (clustering measures <0.5, Figure 2D). The above result is not surprising given the overall poor data quality, depth and expression correlation. This offers one explanation for why the authors were unable to classify their home-cage “control” dataset (Ainsley et al. page 4). In addition, we compiled a list of 139 in situ hybridization-verified dendritic mRNAs from several sources (Lein et al., Cajigas et al and references therein) and compared it to the “+pyr“ and “–pyr“ classified set suggested by Ainsley et al. (Figure 2E and 2F, Table 1 http://brain.mpg.de/uploads/media/Ainsley_Rebuttal_01Sep2015.pdf). We found that less than 35% of the dendritically-verified transcripts (from the above lists) were included in the Ainsley et al., (+pyr) list, the majority were in fact listed on the -pyr list. Even 16 genes that the Allen Brain Atlas team indicated as “dendritic” (e.g. Lein et al.,) were mis-classified by Ainsley et al. as (–pyr) (Table 1). In summary, the study of Ainsley et al., is flawed at many levels, including experimental design, poor data quality and incorrect analyses, invalidating many, if not all, of the conclusions of the authors.