Project title:
How reproducible are economic decision-analytic models? A systematic review and replication study
Type of research:
Secondary data analysis
Background & Scientific Rationale:
Reproducibility refers to the degree to which others can replicate published research to confirm the findings using similar methods. Good science should ensure results are reproducible in order to maintain confidence and trust in scientific research. However, various recent evidence suggests confidence in academic research is waning and that there is much avoidable research waste created in the research process. Chalmers & Glasziou (2009) estimated 85% “of investment in [biomedical] research are lost every year because of correctable problems”. As part of this series Chan et al (2014) identified the need to make research more accessible, and thus to encourage comprehensive reporting. The largest health research funder in the US, the NIH, is now addressing the reproducibility issue.
Currently, the majority of economic evaluations are performed alongside clinical trials, using outcomes and resource use collected within the trial as the primary source of economic data. However, it is also possible to conduct an economic evaluation using a decision-analytic model, defined by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force ‘‘as an analytic methodology that accounts for events over time and across populations, that is based on data drawn from primary and/or secondary sources, and whose purpose is to estimate the effects of an intervention on valued health consequences and costs’’. In doing this, it is possible to draw on multiple sources of evidence as well as to evaluate all relevant comparators, hence reducing the need to perform additional clinical trials to compare interventions previously not compared head to head. In line with current guidelines, it is advised that in reporting decision-analytic models there should be “Technical documentation, written in sufficient detail to enable a reader with necessary expertise to evaluate the model and potentially reproduce it, should be made available openly or under agreements that protect intellectual property, at the discretion of the modelers.” but the degree to which published economic models adhere to this has not to our knowledge been tested. We did find one study that examines the between model consistency, which attempted to see if different models would give the same results with identical inputs (even though model structure may be different). In contrast, this proposal seeks to examine how easy it is to take both the structure and content of a published model and reproduce it.
Methods:
A list of search terms were devised for identifying any published article which considers the reproduction, re-analysis or re-engineering of a decision-analytic model. Using these terms, a search of standard databases was conducted, for example: Medline and EMBASE.
- Inclusion criteria: Any article which details the reproduction, re-analysis or re-engineering of a previously published decision-analytic model.
- Exclusion criteria: Articles published in any language other than English. Articles that rely on information not in the public domain to reproduce, re-analyse or re-engineer the decision-analytic model.
Once the search was conducted, the resulting abstracts were initially screened by two independent reviewers, to determine their eligibility for the review. For those that were deemed potentially eligible, a further screening process took place of the full article. Any disputes will be dealt with by a third reviewer. Following the identification of eligible papers, data will be extracted using a standard data extraction form, including, but not limited to, the reason for replication, the type of model replicated, any problems encountered, as well as the results of the replication. This extraction will be completed by two independent reviewers, with any discrepancies being resolved by a third reviewer, if necessary.
As the aims of the systematic review are largely qualitative in nature, and as it is likely that the papers identified will be replicated models considering different interventions and population groups, it is envisaged that it would be unsuitable to conduct a meta-analysis. Therefore, the results of the systematic review will be presented in a narrative approach and the methodology will be reported according to the “Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015” guidance.
Following this, an attempt to replicate a previously published decision-analytic model will be made, using only information from the published report and information available in the public domain. In the process of doing this, particular challenges will be identified and the feasibility of model replication, as well as the strengths and weaknesses of the research will be discussed. Additionally, the implication of putting models in the public domain on intellectual property rights will be discussed.
Key findings and outputs:
The results of this project are currently being written up for publication.
Preliminary results have found that whilst replication of results is commonplace in other scientific disciplines, there is relatively little discussing the concept within health economics. Going forward, consensus is needed on just how replicable health economic modelling should be, and particularly how a successful replication should be defined. These early findings were orally presented at the Norwich Medical School’s Faculty of Medicine and Health, Post Graduate Research Conference (May, 2018) and also to the Health Economics Study Group conference in Bristol, (July, 2018) in a paper entitled: “When is a model replication successful? Exploring the definition of success.”
Contact:
Emma McManus e-mail: Emma.Mcmanus@uea.ac.uk
References:
- Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374: 86-89.
- Chan, A. W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B.. Increasing value and reducing waste: addressing inaccessible research. The Lancet, 383(9913), 257-266, 2014.
