Appendix A
We comment on items in the GRIPS and REMARK guidelines that do not appear in the RiGoR guidelines. Our purpose here is explain the rationale of the RiGoR guidelines and our comments should not be interpreted as criticisms of these important, previous efforts.
GRIPS
3. “Specify the study objectives and state the specific model(s) that is/are investigated. State if the study concerned the development of the model(s), a validation effort, or both.”
The RiGoR guidelines apply to papers documenting the development of risk models.
13. “Describe all subgroups, interactions, and exploratory analyses that were examined.”
20. “Present results of any subgroup, interaction, or exploratory analyses, whenever pertinent.”
Item 13 is very broad, and item 20 is vague as to what constitutes “pertinent” results. In the RiGoR guidelines, we focus on sources of bias (e.g. model-selection bias and resubstitution bias) that are common in risk model development and provide specific guidelines to identify them.
17. “Report distributions of predicted risks and/or risk scores.”
Such distributions are one way to evaluate model performance. The RiGoR guidelines provide flexibility on metrics by which to evaluate risk models.
24. “State whether databases for the analyzed data, risk models, and/or protocols are or will become publicly available and if so, how they can be accessed.”
We did not include a similar item in the RiGoR guidelines because it is not crucial for assessing the study that was done.
25. “Give the source of funding and the role of the funders for the present study. State whether there are any conflicts of interest.”
We did not include an item like this in the RiGoR guidelines because it is not specific to risk model research.
REMARK
3. “Describe treatments received and how chosen (e.g. randomized or rule-based).”
This item is specific to prognosis in cancer.
9. “Give rationale for sample size; if the study was designed to detect a specified effect size, give the target power and effect size.”
This guideline is more appropriate for studies of association.
12. “Describe the flow of patients through the study, including the number of patients included in each stage of the analysis (a diagram may be helpful) and reasons for dropout. Specifically, both overall and for each subgroup extensively examined report the numbers of patients and the number of events.”
This item is not widely applicable to risk model development.
14. “Show the relation of the marker to standard prognostic variables.”
15. “Present univariate analyses showing the relation between the marker and outcome, with the estimated effect (e.g. hazard ratio and survival probability.) Preferably provide similar analyses for all other variables being analyzed. For the effect of a tumor marker on a time-to-event outcome, a Kaplan-Meier plot is recommended.”
16. “For key multivariable analyses, report estimated effects (e.g. hazard ratio) with confidence intervals for the marker and, at least for the final model, all other variables in the model.”
17. “Among reported results, provide estimated effects with confidence intervals from an analysis in which the marker and standard prognostic variable are included, regardless of their statistical significance.”
Such information is tangential to the validity and utility of a risk model. Furthermore, in the context of extensive variable or model selection, estimated effects may be biased.
18. “If done, report results of further investigations, such as checking assumptions, sensitivity analyses, and internal validation.”
This guideline is somewhat broad and open-ended, so we chose not to include a similar item in the RiGoR guidelines.