move to Methods

be more specific - a MS-CPM that allows all the transitions of interest to be modelled holistically etc.

Should say what the concrete implications of this are: e.g. risk probabilties may be mis-interpreted when the model is used etc.

@Mark would be good to have something on clinical implication here too

you mean treated? A little hard to follow here.

think now agreed to just report one (with the others in a supplement)

need to say why such predictions are clinically useful - @Mark.

this is commonly background + objectives in an abstract. So should also include the objective (to develop the MS-CPM)

this seems an unusual thing to do to me. Have you seen this before? Wouldn't it mean that standard errors, coverage etc are underestimated? I'd suggest instead using all data to calculate things like standard error, but show 5-95% intervals on the plots?

Looking at the Results, and at your code, I think this removal of the top and bottom 5% could have quite unpredictable implications on bias, SE and coverage.

Please can we see the results without the extremes dropped?

in terms of estimating our target parameter theta that measures the calibration in the large.

recall again here - these come from the 'true' models that match the DGMs. Also important to mention that this use of 'truth' doesn't (in my opinion) bias results in our favour since all methods are able to use this 'true' expected event proportion.

However, you could argue that the 'true' IPCW is favouring our approach - you could mention this as a limitation in the Discussion: that for finite samples one of the simpler approaches might outperform IPCW because it doesn't need to estimate the IPCW

we really need to get guidance from Paul Lambert on this: have you managed to get hold of him?

Presumably this is no problem at all as long as there is a weighting option (as there is in glm())

worth noting that this is a very bad idea and nobody (hopefully!) would do this in practice.

suggest cutting this (it's not an accurate description of what the KM estimate is, so I suggest just say we use the KM estimate as the observed proportion of events)

make it very clear that you are assuming these models are known: not attempting to estimate them - and explain why this is the right thing to do (i.e. this part of the process is not of interest)

as here we are interested in bias rather than variability of the estimates

in a factorial design

Suggest cutting: giving the hazard functions is enough (corresponding survival functions are obvious to derive).

just write down the equation all together - feels a bit strange to separate out like this

Observations

suggest moving a bit later - since I'm immediately wondering what the models are - so I'd present the models first then say these will be the parameters that we vary.

• other ways that censoring handled - e.g. pohar-perme approach etc.

with

model for censoring?

to be precise, exchangeable conditional on the covariates in the model used to construct the IPCW, assuming that this model is correctly specified.

the inverse of the

also assumes

stick with either mean calibration or calibration in the large, for consistency.

addressed? (overcome a bit strong given what we go on to say...)

yes: qrisk only provide a point estimate of the baseline hazard at 10y - and even for that you have to go digging

how and whether

Replace to 'Other'

add comma

repetition of prev sentence

This title needs changing: doesn't capture the contents at all. How about 'Using inverse probability of censoring weights to estimate calibration-in-the-large for time-to-event models'

and replaces with the assumption that they are identical/exchangeable conditional on the measured covariates.

Looks good. For bias I wonder if absolute bias might be clearer for eyeballing?

change to time to event model

i.e. Weibull(?)

in this case censoring is just random - so wouldn't we expect all methods to do well here?

0.2?

express as a proportional hazards model instead. We can either estimate as a proportional baseline Weibull hazard or using Cox, I assume it wouldn't matter which

I know I wrote this(!), but this para is quite hard to follow - suggest moving the points that address the first two of the three ways to where they are introduced - then introduce the third way (censoring) and say that is our focus.

move above to where we introduce the challenge of sharing baseline hazard.

what is 'perfect'? Coverage should be 95%. Too high and too low are both bad

but why? I would expect the coverage to be ok

i think this section can be shortened.Only need a brief 'dismissal' of the existing models - no need to labour the point.

there seems to be a lot of information missing from the Methods section:

what model was used missing data handling model selection

etc

work through TRIPOD guidelines

the models with different number of states are modelling different outcomes, and probably answering slightly different clinical questions. So I think clinical considerations should primarily inform which of 2, 3, 4 state model is used. Then a question about whether all three need to be presented in this paper

not sure this quite captures. Multiple outcomes to me suggests multivariate prediction models (like Glen's MRC grant and https://arxiv.org/abs/2001.07624 ). I think here we still have a single outcome but complex, i.e. changes over time as captured by a multi-state model.

but shouldn't sample size calculations be done to inform how many predictors to include in the models? https://www.bmj.com/content/368/bmj.m441.abstract

paper is quite acronym heavy. Suggest cutting them down.

I think it is necessary to at least check the PH assumption. For example, there might be a strong non-proportionality across gender: then it would be entirely reasonable to fit separate models by gender (e.g. as QRISK does)

Paper needs to explain more how you can indeed apply at any point in the journey. Presume that you mean only before any state transition has occurred?

is that what is recommended in the Lambert paper / or the Pohar-Perme one?