Methods and standards
Recognised standards, applied the same way every time
Decision-makers and reviewers should not have to take our work on trust. These are the published standards we follow and the processes we use to apply them.
Reporting: CHEERS 2022
Economic evaluation reports follow the Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022), a 28-item checklist covering the study question, methods, results and the involvement of patients and others affected by the study. We complete the checklist for each report and supply it with the report, so a reviewer can find each item quickly.
- Completed CHEERS 2022 checklist supplied with every economic evaluation report
- Health economics analysis plan agreed before analysis where the study design allows
Good practice guidance: ISPOR and reference cases
Model design, budget impact analysis and preference studies follow the relevant good practice reports of ISPOR (the Professional Society for Health Economics and Outcomes Research) and the Society for Medical Decision Making. For economic evaluations in low- and middle-income settings we use the International Decision Support Initiative (iDSI) reference case as a guide to methods choices, and for costing studies the Global Health Cost Consortium reference case. Where a country has its own methods guidelines, those take precedence.
- Model structure, transparency and validation: ISPOR-SMDM Modeling Good Research Practices
- Budget impact analysis: ISPOR principles of good practice
- Discrete choice experiments: ISPOR conjoint analysis good practice reports
- Methods choices stated explicitly in each report, with the reason for each
Reproducible analysis pipelines
Analyses and models are written in R, or in documented Excel where the client needs it, and kept under version control. Each result is produced from a recorded set of inputs, so any figure can be regenerated and every change between versions is logged. Parameter tables record each input's value, distribution, source and the reason it was chosen.
- One repository per engagement: code, parameter tables and outputs together
- Tagged versions for every report that leaves the firm
- Every input traced to a source, and every proxy stated and tested in sensitivity analysis
Independent quality control
Before results are released, a second analyst who did not build the model checks it against a written checklist covering inputs, logic, code and outputs. Issues are logged and resolved, and work cannot move on until a senior expert signs off the scope and structure, the inputs, and the results. The quality control log is available to clients on request.
- Input checks against the parameter table and sources
- Logic, extreme-value and consistency tests
- Report tables checked against model outputs
- Expert sign-off at three gates
For the full picture — our pipeline, sign-off gates and how we validate models — see Approach and quality.
References
- Husereau D, et al. Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement. BMJ 2022;376:e067975. doi.org/10.1136/bmj-2021-067975 (opens in a new tab)
- Caro JJ, et al. Modeling good research practices — overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1. Value in Health 2012;15:796–803. doi.org/10.1016/j.jval.2012.06.012 (opens in a new tab)
- Eddy DM, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health 2012;15:843–850. doi.org/10.1016/j.jval.2012.04.012 (opens in a new tab)
- Sullivan SD, et al. Budget impact analysis — principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value in Health 2014;17:5–14. doi.org/10.1016/j.jval.2013.08.2291 (opens in a new tab)
- Bridges JFP, et al. Conjoint analysis applications in health — a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health 2011;14:403–413. doi.org/10.1016/j.jval.2010.11.013 (opens in a new tab)
- Wilkinson T, et al. The International Decision Support Initiative reference case for economic evaluation: an aid to thought. Value in Health 2016;19:921–928. doi.org/10.1016/j.jval.2016.04.015 (opens in a new tab)
- Vassall A, et al. Reference case for estimating the costs of global health services and interventions. Global Health Cost Consortium, 2017. researchonline.lshtm.ac.uk/id/eprint/4653001/ (opens in a new tab)
- ISPOR good practices reports (index). www.ispor.org/heor-resources/good-practices (opens in a new tab)
Want to see our QC checklist?
We are happy to walk through our methods, checklists and a sample parameter table on a short call.