On 2017 Jan 23, Andy Collings commented:

(Original comment found at: https://elifesciences.org/content/6/e21634#disqus_thread)

Response to: “Replication Study: Melanoma genome sequencing reveals frequent PREX2 mutations"

Lynda Chin and Levi Garraway

We applaud the Reproducibility Project and support its goal to reproduce published scientific results. We also thank Horrigan et al for a carefully executed study, for which we provided reagents and extensive consultation throughout. Their work illustrates the inherent challenges in attempting to reproduce scientific results.

We summarize below the results of Horrigan et al., first in lay terms and then in more scientific detail.

Description for Lay Readers

Briefly, our 2012 paper reported that human melanoma patients often carry mutations in the PREX2 gene. To study the effect of mutations in PREX2, we made modified versions of a commonly used immortalized human melanocyte cell line (called p’mels) and injected them into mice. When mice were injected with cells carrying an irrelevant gene or a normal copy of PREX2 (“control mice”), tumors started to form in about 9 weeks. When mice were injected with cells carried the mutated PREX2 genes (“experimental mice”), tumors began to form after around 4-5 weeks—indicating that mutated PREX2 accelerated tumor formation.

When Horrigan et al. tried to reproduce our experiment, they found that tumors began to form in their control mice after about 1 week—not 9-10 weeks. Because their control developed tumors so rapidly, Horrigan et al. recognized that they could not meaningfully test our finding that mutant PREX2 accelerated the tumor formation.

Why did the human melanocyte cells grow tumors in the control mice so much faster in Horrigan et al.’s experiment? The likely explanation is that human cells engineered in this way are known to undergo dramatic changes when they are grown for extended periods in culture. Therefore, Horrigan et al.’s study underscores how important it is to have appropriate control cells, before attempting to reproduce experimental findings.

Finally, we emphasize that Horrigan et al.’s results do not call into question our results about PREX2 because their experiment was not informative. Moreover, we have recently validated the findings about PREX2 in an independent way—by creating genetically engineered mice that carry mutated PREX2 in their own genomes. These PREX2 mutant mice showed accelerated tumor growth compared to controls.

Description for Scientific Readers

The authors repeated a xenograft experiment (Figure 3b) in our 2012 report. In our experiment, we overexpressed GFP (negative control), wild type PREX2 (normal control) and two PREX2 mutants (G844D and Q1430*) (experimental arm) in a TERT-immortalized human melanocyte line engineered with RB and p53 inactivation (p’mel). To further sensitize these melanocytes for tumorigenicity, they were also engineered to overexpress oncogenic NRASG12D. We showed that the mutant PREX2 expression in p’mel cells significantly accelerated tumor formation in vivo. However, Horrigan et al found that the control and PREX WT or mutant expressing p’mels all behaved identically, forming tumors rapidly in vivo (within 1 week of implantation). This finding differed from our study, in which the control cells (both GFP and PREX2) did not form tumors until >10 weeks after implantation.

The fact that Horrigan et al observed rapid tumor formation in all settings means that their findings are uninformative with regard to the reproducibility of a central conclusion of our 2012 report, namely that mutant PREX2 can accelerate tumor formation in vivo. Testing this hypothesis requires a control arm in which tumor formation is sufficiently latent so that a discernible effect on the rate of tumorigenesis by the mutants can be observed. In the Horrigan et al study, tumorigenesis in the control arms was so rapid that it essentially became impossible to detect any additional effect of mutant PREX2.

Why were the controls so much more tumorigenic in the hands of Horrigan et al.? We note that although the investigators were provided with clones from the same p’mels used in the 2012 study, by the time Horrigan et al received the cells, more than two years had passed since the original p’mel cells were engineered. This is a crucial point, because as with many other cell lines, these “primed” human primary melanocytes are known to readily undergo adaptation during extended cultivation in vitro. In particular, these p’mels can spontaneously acquire a more transformed phenotype over time (we have seen this happen on multiple occasions). Thus, although a clone from the same engineered cells were provided to Horrigan et al, the fact that that clone of p’mel cells exhibited very different phenotype suggests that the additional passages, a major geographic relocation, and subsequent freeze-thaw manipulations have rendered them unsuitable as an experimental frame of reference.

When we notice such “drifts” in engineered cell culture models, we often have to re-derive the relevant lines starting from even earlier stages in order to have controls with suitable tumorigenic latency. For example, in this case, we would have re-introduced NRASG12D into a clone of non-transformed melanocytes harboring TERT immortalization and RB/P53 inactivation to re-engineer a p’mel cell line. Had Horrigan et al used less tumorigenic controls, they would have a much better chance to reproduce an accelerating effect of mutant PREX2.

To validate our initial observations regarding the oncogenic role of mutant PREX2, we have since taken an orthogonal approach: we created a genetically engineered mouse (GEM) model targeting both a truncating PREX2 mutation (E824*) and oncogenic NRASG12D expression to melanocytes under a tet-regulated promoter. In this GEM model, we observed significantly increased penetrance and decreased latency of melanoma formation (Lissanu Deribe et al PNAS, 2016, E1296-305; see Figure 3b in Lissanu Deribe et al PNAS, 2016), thus confirming the xenograft findings of our 2012 report showing that mutant PREX2 is oncogenic.

In summary, we support rigorous assessments of reproducibility such as this. Equally, we consider it crucial to recognize and account for salient underlying properties of the model systems and experimental controls in order to minimize the risk of misleading conclusions regarding the reproducibility of any given experiment. Indeed, Horrigan et al. nicely articulated the importance of these considerations when discussing their results.

On 2017 Jan 23, Andy Collings commented:

(Original comment found at: https://elifesciences.org/content/6/e21634#disqus_thread)

Response to: “Replication Study: Melanoma genome sequencing reveals frequent PREX2 mutations"

Lynda Chin and Levi Garraway

We applaud the Reproducibility Project and support its goal to reproduce published scientific results. We also thank Horrigan et al for a carefully executed study, for which we provided reagents and extensive consultation throughout. Their work illustrates the inherent challenges in attempting to reproduce scientific results.

We summarize below the results of Horrigan et al., first in lay terms and then in more scientific detail.

Description for Lay Readers

Briefly, our 2012 paper reported that human melanoma patients often carry mutations in the PREX2 gene. To study the effect of mutations in PREX2, we made modified versions of a commonly used immortalized human melanocyte cell line (called p’mels) and injected them into mice. When mice were injected with cells carrying an irrelevant gene or a normal copy of PREX2 (“control mice”), tumors started to form in about 9 weeks. When mice were injected with cells carried the mutated PREX2 genes (“experimental mice”), tumors began to form after around 4-5 weeks—indicating that mutated PREX2 accelerated tumor formation.

When Horrigan et al. tried to reproduce our experiment, they found that tumors began to form in their control mice after about 1 week—not 9-10 weeks. Because their control developed tumors so rapidly, Horrigan et al. recognized that they could not meaningfully test our finding that mutant PREX2 accelerated the tumor formation.

Why did the human melanocyte cells grow tumors in the control mice so much faster in Horrigan et al.’s experiment? The likely explanation is that human cells engineered in this way are known to undergo dramatic changes when they are grown for extended periods in culture. Therefore, Horrigan et al.’s study underscores how important it is to have appropriate control cells, before attempting to reproduce experimental findings.

Finally, we emphasize that Horrigan et al.’s results do not call into question our results about PREX2 because their experiment was not informative. Moreover, we have recently validated the findings about PREX2 in an independent way—by creating genetically engineered mice that carry mutated PREX2 in their own genomes. These PREX2 mutant mice showed accelerated tumor growth compared to controls.

Description for Scientific Readers

The authors repeated a xenograft experiment (Figure 3b) in our 2012 report. In our experiment, we overexpressed GFP (negative control), wild type PREX2 (normal control) and two PREX2 mutants (G844D and Q1430*) (experimental arm) in a TERT-immortalized human melanocyte line engineered with RB and p53 inactivation (p’mel). To further sensitize these melanocytes for tumorigenicity, they were also engineered to overexpress oncogenic NRASG12D. We showed that the mutant PREX2 expression in p’mel cells significantly accelerated tumor formation in vivo. However, Horrigan et al found that the control and PREX WT or mutant expressing p’mels all behaved identically, forming tumors rapidly in vivo (within 1 week of implantation). This finding differed from our study, in which the control cells (both GFP and PREX2) did not form tumors until >10 weeks after implantation.

The fact that Horrigan et al observed rapid tumor formation in all settings means that their findings are uninformative with regard to the reproducibility of a central conclusion of our 2012 report, namely that mutant PREX2 can accelerate tumor formation in vivo. Testing this hypothesis requires a control arm in which tumor formation is sufficiently latent so that a discernible effect on the rate of tumorigenesis by the mutants can be observed. In the Horrigan et al study, tumorigenesis in the control arms was so rapid that it essentially became impossible to detect any additional effect of mutant PREX2.

Why were the controls so much more tumorigenic in the hands of Horrigan et al.? We note that although the investigators were provided with clones from the same p’mels used in the 2012 study, by the time Horrigan et al received the cells, more than two years had passed since the original p’mel cells were engineered. This is a crucial point, because as with many other cell lines, these “primed” human primary melanocytes are known to readily undergo adaptation during extended cultivation in vitro. In particular, these p’mels can spontaneously acquire a more transformed phenotype over time (we have seen this happen on multiple occasions). Thus, although a clone from the same engineered cells were provided to Horrigan et al, the fact that that clone of p’mel cells exhibited very different phenotype suggests that the additional passages, a major geographic relocation, and subsequent freeze-thaw manipulations have rendered them unsuitable as an experimental frame of reference.

When we notice such “drifts” in engineered cell culture models, we often have to re-derive the relevant lines starting from even earlier stages in order to have controls with suitable tumorigenic latency. For example, in this case, we would have re-introduced NRASG12D into a clone of non-transformed melanocytes harboring TERT immortalization and RB/P53 inactivation to re-engineer a p’mel cell line. Had Horrigan et al used less tumorigenic controls, they would have a much better chance to reproduce an accelerating effect of mutant PREX2.

To validate our initial observations regarding the oncogenic role of mutant PREX2, we have since taken an orthogonal approach: we created a genetically engineered mouse (GEM) model targeting both a truncating PREX2 mutation (E824*) and oncogenic NRASG12D expression to melanocytes under a tet-regulated promoter. In this GEM model, we observed significantly increased penetrance and decreased latency of melanoma formation (Lissanu Deribe et al PNAS, 2016, E1296-305; see Figure 3b in Lissanu Deribe et al PNAS, 2016), thus confirming the xenograft findings of our 2012 report showing that mutant PREX2 is oncogenic.

In summary, we support rigorous assessments of reproducibility such as this. Equally, we consider it crucial to recognize and account for salient underlying properties of the model systems and experimental controls in order to minimize the risk of misleading conclusions regarding the reproducibility of any given experiment. Indeed, Horrigan et al. nicely articulated the importance of these considerations when discussing their results.