On 2017 Jan 15, John Tucker commented:

Addendum:

Performing a Monte Carlo calculation using actual PFS and OS data from P3 NSCLC trials in first-line, Stage IIIb /IV NSCLC patients and standard errors of measurement of 0.5 and 0.2 months for OS and PFS respectively, one can calculate that the maximum attainable measured correlation would be about 0.93. This number falls about halfway between my estimate of 0.85 and the 1.0 assumed in the paper.

If one assumes that this maximum attainable value is representative of other cancers, and recategorizes the PFS-OS correlations accordingly, the fraction of correlations in the paper categorized as high or medium rises from 44% to 72%.

So it appears that, as is so often the case, that the truth lies somewhere in the middle of the two positions argued here.

On 2017 Jan 15, John Tucker commented:

Dr. Prasad is correct if one accepts that referencing a source which used the same proposed cutoffs, but which also failed to provide any justification for them, is sufficient validation.

Looking at the math, however, it clearly doesn't make sense to expect a correlation coefficient of >0.85 when the OS values included in the analysis span a range of 7 months, and the individual OS values are measured with 95% confidence intervals that span 2 months.

In the absence of a convincing rationale for the selection of the cutoff values, the conclusions of the paper represent little more than a tautology. The correlations are low because Prasad et al. have defined them as such.

On 2016 Dec 22, Vinay Prasad commented:

Sadly, the reader is incorrect. We state clearly in the manuscript that the proposed cutoffs arise from ones proposed by the Institute of Quality and Efficiency in Health Care.

On 2016 Dec 07, John Tucker commented:

Prasad and coworkers provide a review of previous meta analyses undertaken to determine the correlation of various surrogate endpoints with overall survival in cancer trials. They tabulate the various studies according to tumor type and provide their assessment in each case of whether the correlation found was high, medium or low. They conclude that most such studies found only low correlation with survival and that such surrogates should thus have no place in the approval of cancer therapeutics.

Unfortunately, the authors provide no explanation for their chosen cutoffs of what represents a "high" (r > 0.85), "medium" (r = 0.7 to 0.85) or "low" correlation. This is a critical shortcoming, as the choice of these definitions dictates the paper's conclusions. Given the modest accuracy with which median PFS and OS values are determined in typically sized Phase 3 trials, these cutoffs are not mathematically justified.

Looking at typical phase 3 trials in 1st line metastatic NSCLC, for example, overall survival typically ranges from 9 to 16 months with an accuracy of measurement (standard error) of about 0.5 months. Progression free survival is measured with a standard error of about 0.2 months. This suggests that even if the true correlation of PFS with OS in this indication were 1.00, the maximum observed correlation would be on the order of 0.85, which according to Prasad's criteria would only be a "medium" correlation. Because of the limited precision with which the values of PFS and OS are determined in clinical trials, the observed correlation will always be less than the true correlation, and Prasad's analysis fails to take this into account.

In light of the unrealistic and mathematically unjustified categorization criteria used in this paper, it is not surprising to find that Prasad's characterization of the observed correlations often differs from that of the authors of the underlying studies. For example, Tang et al. characterize the relationship between PFS and OS in first line mCRC (r = 0.74) as "strong", but Prasad et al demote this surprisingly high correlation to "medium". Similarly, Petrelli et al characterize the correlation between OS and PFS in metastatic breast cancer as strong, but Prasad et al recharacterize the correlation (0.7) as "low".

On 2016 Dec 07, John Tucker commented:

Prasad and coworkers provide a review of previous meta analyses undertaken to determine the correlation of various surrogate endpoints with overall survival in cancer trials. They tabulate the various studies according to tumor type and provide their assessment in each case of whether the correlation found was high, medium or low. They conclude that most such studies found only low correlation with survival and that such surrogates should thus have no place in the approval of cancer therapeutics.

Unfortunately, the authors provide no explanation for their chosen cutoffs of what represents a "high" (r > 0.85), "medium" (r = 0.7 to 0.85) or "low" correlation. This is a critical shortcoming, as the choice of these definitions dictates the paper's conclusions. Given the modest accuracy with which median PFS and OS values are determined in typically sized Phase 3 trials, these cutoffs are not mathematically justified.

Looking at typical phase 3 trials in 1st line metastatic NSCLC, for example, overall survival typically ranges from 9 to 16 months with an accuracy of measurement (standard error) of about 0.5 months. Progression free survival is measured with a standard error of about 0.2 months. This suggests that even if the true correlation of PFS with OS in this indication were 1.00, the maximum observed correlation would be on the order of 0.85, which according to Prasad's criteria would only be a "medium" correlation. Because of the limited precision with which the values of PFS and OS are determined in clinical trials, the observed correlation will always be less than the true correlation, and Prasad's analysis fails to take this into account.

In light of the unrealistic and mathematically unjustified categorization criteria used in this paper, it is not surprising to find that Prasad's characterization of the observed correlations often differs from that of the authors of the underlying studies. For example, Tang et al. characterize the relationship between PFS and OS in first line mCRC (r = 0.74) as "strong", but Prasad et al demote this surprisingly high correlation to "medium". Similarly, Petrelli et al characterize the correlation between OS and PFS in metastatic breast cancer as strong, but Prasad et al recharacterize the correlation (0.7) as "low".

On 2016 Dec 22, Vinay Prasad commented:

Sadly, the reader is incorrect. We state clearly in the manuscript that the proposed cutoffs arise from ones proposed by the Institute of Quality and Efficiency in Health Care.