On 2013 Jul 09, Joshua L Cherry commented:

This work develops metabolic models for six E. coli strains, performs relevant experiments with these strains, and compares model predictions to observations. The authors state that they have demonstrated that "quantitative models of different strains of E. coli can accurately predict strain-specific phenotypes." The results, however, do not support this conclusion. In fact, they point to the opposite conclusion: the models had little or no value for predicting differences between strains.

The authors predicted and experimentally determined the abilities of the strains to use various carbon sources under aerobic and anaerobic conditions. Most individual predictions agree with observations, but this fact alone means little; all strains were able to utilize the vast majority of substrates, so that simply "predicting" all positives would yield 95% agreement. For the data in Table 5, results are statistically significant under aerobic conditions for only one strain (Fisher's exact test). Under anaerobic conditions the predictions fare better: 5 of 6 cases are statistically significant. However, none of this tells us whether the predictions capture differences between strains. To address this question requires more information than counts of the sort found in Table 5.

Additional file 7 of the article provides the necessary kind of data: predicted and observed utilization capabilities for individual substrates. For reasons not clear to me, it contains predictions for only 68 substrates, rather than the 76 reported in Table 5. I have searched this data for correct predictions of differences between strains. These would be cases where one strain utilizes a substrate, another fails to do so under the same conditions, and this is correctly predicted. Across all 68 substrates and all pairings of the 6 strains, there is just one such case. This involves aerobic utilization of phenylethylamine. Even this cannot be considered a success because predictions for the other four strains are all incorrect, and all four possible combinations of predictions and observations occur. All the other differences between strains are missed, and there are several predicted differences that are shown to be false by experiment.

The authors also predict and measure growth rate and yield for growth on glucose (actually, the flux balance analysis itself predicts only yield; conversion to rate relies on measured rates of glucose uptake). When the aerobic and anaerobic data are combined, correlations between prediction and experiment are strong and significant. However, these high correlations have nothing to do with differences between strains. Rather, they reflect the difference between aerobic growth and anaerobic growth. Simply assuming that the growth rate (or yield) is higher under aerobic conditions than anaerobic conditions, and that there are no strain differences, results in similar high correlations. So does randomly permuting the predictions among strains. As the authors note, correlations disappear when strains are considered under a single condition, so there is no evidence for correct prediction of differences among strains.

Thus, it appears that the predictions contain little or no information about phenotypic differences between strains. Reconciling the models and experimental results may aid our understanding of metabolism and help improve future predictions.

On 2013 Jul 09, Joshua L Cherry commented:

This work develops metabolic models for six E. coli strains, performs relevant experiments with these strains, and compares model predictions to observations. The authors state that they have demonstrated that "quantitative models of different strains of E. coli can accurately predict strain-specific phenotypes." The results, however, do not support this conclusion. In fact, they point to the opposite conclusion: the models had little or no value for predicting differences between strains.

The authors predicted and experimentally determined the abilities of the strains to use various carbon sources under aerobic and anaerobic conditions. Most individual predictions agree with observations, but this fact alone means little; all strains were able to utilize the vast majority of substrates, so that simply "predicting" all positives would yield 95% agreement. For the data in Table 5, results are statistically significant under aerobic conditions for only one strain (Fisher's exact test). Under anaerobic conditions the predictions fare better: 5 of 6 cases are statistically significant. However, none of this tells us whether the predictions capture differences between strains. To address this question requires more information than counts of the sort found in Table 5.

Additional file 7 of the article provides the necessary kind of data: predicted and observed utilization capabilities for individual substrates. For reasons not clear to me, it contains predictions for only 68 substrates, rather than the 76 reported in Table 5. I have searched this data for correct predictions of differences between strains. These would be cases where one strain utilizes a substrate, another fails to do so under the same conditions, and this is correctly predicted. Across all 68 substrates and all pairings of the 6 strains, there is just one such case. This involves aerobic utilization of phenylethylamine. Even this cannot be considered a success because predictions for the other four strains are all incorrect, and all four possible combinations of predictions and observations occur. All the other differences between strains are missed, and there are several predicted differences that are shown to be false by experiment.

The authors also predict and measure growth rate and yield for growth on glucose (actually, the flux balance analysis itself predicts only yield; conversion to rate relies on measured rates of glucose uptake). When the aerobic and anaerobic data are combined, correlations between prediction and experiment are strong and significant. However, these high correlations have nothing to do with differences between strains. Rather, they reflect the difference between aerobic growth and anaerobic growth. Simply assuming that the growth rate (or yield) is higher under aerobic conditions than anaerobic conditions, and that there are no strain differences, results in similar high correlations. So does randomly permuting the predictions among strains. As the authors note, correlations disappear when strains are considered under a single condition, so there is no evidence for correct prediction of differences among strains.

Thus, it appears that the predictions contain little or no information about phenotypic differences between strains. Reconciling the models and experimental results may aid our understanding of metabolism and help improve future predictions.