Potential impact of a new sepsis prediction model for the primary care setting: early health economic evaluation using an observational cohort

STRENGTHS AND LIMITATIONS OF THIS STUDY

  • The prospective data collection resulted in few missing data for the calculation of the sepsis prediction scores and costs.

  • The early economic analysis is enriched with observational study data and a detailed form of expert panel assessment. However, the early nature of the analyses limits the accuracy of the impact on costs for patients with referral states that differ from the observed state during the study.

  • The Hawthorne effect may have biased the results as the general practitioners enrolled patients prospectively in a sepsis study.

Introduction

Sepsis is a life-threatening complication from infection with high mortality and morbidity. The global incidence of sepsis is estimated at 48 million cases per year, resulting in 11 million deaths.1 For the Netherlands, the same study estimated about 59 000 cases and 9400 deaths annually. Hospital-related costs of sepsis are high, estimated between €1101 and €91 951 per patient in different countries and accounting for an average of 2.65% of the total healthcare budget.2 These reported healthcare costs of sepsis are affected by duration of follow-up of healthcare use, included cost components, (study) region, population size and sepsis severity studied.

Early identification of sepsis is one of the key factors to improve outcome.3–5 In the hospital setting, protocolised care for patients with suspected sepsis is increasingly implemented to reduce time to adequate treatment. Not all sepsis-related mortality can be prevented with optimal hospital treatment, for example, due to serious underlying conditions.6 7 However, the simple intervention of rapid intravenous broad-spectrum antibiotics is the most effective treatment of sepsis.4 8 Therefore, early recognition of emerging sepsis and timely referral of patients with sepsis to the hospital are essential to facilitate this.9 10 However, the current prehospital recognition of sepsis is suboptimal.11–14

Recognition of sepsis can potentially be improved by a clinical prediction model consisting of vital signs and other readily available clinical information. Our research group recently showed a simple diagnostic sepsis model enables general practitioners (GPs) to estimate the risk of sepsis in adult patients immediately at the bedside.15 Ideally, all patients with sepsis are referred to the hospital after GP assessment. However, such a strategy aiming not to miss cases of sepsis will also lead to more unnecessary hospital referrals and can potentially increase medical costs. The primary aim of this study was to assess the potential impact on the rate of referral and subsequent costs at different cut-off levels of the prediction model in the primary care setting in a so-called early economic evaluation. Second, we assessed room for improvement in costs if referral decisions would have been perfect.

Methods

We performed the analyses using data of a cohort of 357 patients included in a previously described observational diagnostic study.15 16 As the clinical effect of using the prediction model is not studied yet, these data were used as input for a decision tree to perform an early economic evaluation. The analyses were performed from a third-party payer/healthcare perspective with a time horizon of 1 month. Due to the short time horizon, discounting was deemed unnecessary.

The primary outcome of this study was the estimation of the potential impact of the prediction model on referral rates, mortality and costs; the secondary outcome was to estimate the ‘room for improvement’, calculated as the maximum reduction in costs and mortality in case patients are referred according to their need for hospital assessment. To be able to estimate this impact, we estimated the observed resource use in patients, grouped by their necessity of hospital assessment and final diagnosis category and the impact of referral on mortality and healthcare costs of patients with sepsis.

Utility loss was also measured but not taken into account in the modelling of different scenarios for referral, as it was not deemed possible to accurately estimate utility loss of alternative scenarios.

Study population

In the TeSD-IT Study, 357 patients were included during out-of-hours home visits by GPs at four GP cooperatives in the Netherlands. Out-of-hours primary care is provided by 51 organisations of GP cooperatives, each of which has 50–250 GPs who provide care to 100 000–500 000 citizens. The cooperatives serve 99% of the Dutch population and are available outside business hours.17 All acutely ill, adult patients ≥18 years with fever, confusion or general deterioration or otherwise suspected of a serious infection were eligible for inclusion. We excluded patients if one or more of the following criteria were present: (1) non-infectious cause of the acute complaints (eg, stroke or myocardial infarction); (2) hospitalisation within 7 days before the home visit; (3) condition that requires secondary care assessment if there are any signs of systemic infection (eg, chemotherapy with possible neutropenia); (4) terminal illness or other reasons not to refer the patient to a hospital despite presence of a life-threatening condition. All patients (or their legal representatives in case of mental incapacitation) provided informed consent. Other details of the methods of this study were published previously.15 Candidate predictors for the decision model were collected prospectively by the GPs who included the patients. Sepsis diagnosis and need for hospital assessment of patients (regardless of the final diagnosis) were judged by three expert panels. Each panel consisted of one GP, one emergency physician and one intensivist/acute care internist. The final diagnosis ‘sepsis within 72 hours of inclusion’ was established by the expert panels based on relevant medical health records. Besides the presence of sepsis, the panellists also scored the need for hospital treatment on a scale of 0–10; a mean score above 5 defined the need for hospital assessment.

Previously developed sepsis prediction model

The developed prediction model consisted of six dichotomous variables, each accounting for 1 point when present: age >65 years; temperature >38°C; systolic blood pressure ≤110 mm Hg; heart rate >110/min; saturation ≤95%; altered mental status. The total score ranges from 0 to 6 points. This model showed a C-statistic of 0.80 for the prediction of the outcome ‘sepsis within 72 hours’ according to the Sepsis-3 definition.18

Decision tree

A decision tree was developed to estimate the impact of a change in referral due to the prediction model on resource use (figure 1). The decision tree has branches for the need of hospital assessment, final diagnosis (sepsis, infection without sepsis or no infection) and for patients who were referred or not referred. Referral refers to the GP referring the patient directly to the hospital after the initial assessment during a home visit (usually by ambulance). Some patients were referred to the hospital after initial treatment at home; this indirect referral was labelled as not referred. The implementation of the prediction model was modelled by adjusting the probability to be referred in each branch. Costs as observed in the trial were used to estimate the costs of each branch. The costs of being in a branch were the same, but the total costs of all the branches were influenced by the differing probability. Because referral was not randomised and the probability for referral can be influenced by patient mix and severity of disease, an exception was made for the branch for patients with sepsis needing hospital assessment, as is described in section ‘Expert opinion impact of referral patients with sepsis’. For patients needing hospital assessment for other infections or other causes (without infection), we assumed that referral did not influence their hospital costs.

Figure 1
Figure 1

Structure of the branches and corresponding probabilities of the decision tree.

Input parameters

Probabilities

The proportion of patients who required hospital assessment, and the proportion who had sepsis, infection without sepsis or no infection were used as probabilities in the decision tree (table 1).

Table 1

Observed healthcare costs in the TeSD-IT Study of referred and non-referred patients, divided by diagnosis and need for hospital treatment

Costs

Healthcare costs were estimated for different groups of patients, divided by their need of hospital assessment, final diagnosis (sepsis, infection without sepsis or no infection) and whether or not they were referred.

After 30 days, patients were sent a healthcare consumption questionnaire (Institute for Medical Technology Assessment Medical Consumption Questionnaire)19 to report health-related costs for home care and family care in the previous 30 days. Furthermore, data on the number of consultations, rehabilitation or nursery homes were retrieved retrospectively from the patients’ own GP. In addition, all hospital procedures were collected from the hospital registries. Total costs were calculated by multiplying the procedures with unit costs from the Dutch Healthcare Authority or the Dutch costing manual.20 For 10 patients, hospital registry data were missing because they were referred to a hospital outside the region. However, intensive care unit (ICU) and ward admission days were known in the case report form. Average hospital costs per admission days of other patients were used to impute other hospital procedures in these patients. Multiple imputation was performed with Multiple Imputation by Chained Equations (MICE) for missing data on home care and family care creating 40 imputed datasets since 34.4% of responses were missing. The variables age, sex, all other variables included in the sepsis model, medication use, variables related to medical history and hospital costs were used as predictors.

Impact of referral on mortality and costs of patients with sepsis

The timing of referral in patients with sepsis is likely to influence hospital costs. However, due to the observational study design, observed costs in referred and not referred patients likely reflect a difference in patient mix and severity of disease at the moment of consultation instead of impact of timing in referral. Therefore, a comprehensive form of expert panel opinion was used to assess the impact of referral on costs and mortality for the patients diagnosed with sepsis. This panel consisted of one GP, one emergency physician and one intensivist. The panel was given medical information of patients in the trial and their observed admission days, ICU admission and whether or not they died. Using this information, the panel experts gave, each individually, their estimate on the hospital duration, ICU admission duration and probability of death in each case presented if they would change from ‘hospital referral’ to ‘no referral’ or vice versa of each case. Based on the score of the new sepsis model, the referral advice for 44 patients who were referred and for 16 patients who were not referred could change in the different modelled scenarios. Each expert was presented (1) 16 cases of patients who were not referred but eventually needed sepsis treatment in the hospital and (2) 9 cases of patients who were referred to the hospital. The estimations in admission days and ICU admission duration in the new situation of cases were the average of the answers of the panel. Referred patients are assumed to be transported by ambulance in all scenarios, as this was also observed in the vast majority of referrals during the TeSD-IT Study. Details on selection of the patients presented to the expert panel and questions answered by the panel can be found in the online supplemental methods.

Supplemental material

Measurement of quality of life

In the follow-up questionnaire at 30 days, patients were asked to fill in the EQ-5D-3L for three different moments: (1) for the situation before start of the acute complaints (T0); (2) for the situation when the patient was most severely ill during the 30-day follow-up period (T1); (3) at the end of the 30-day follow-up (T2). For patients who died before the end of follow-up, utility at T2 was assumed to be 0. Other missing data for utilities were imputed using MICE. The handling of inconsistencies in the reported utilities and the calculation of the utility loss are described in the online supplemental methods.

Patient and public involvement

A patient representative was part of the study group, and was involved in the study design, analyses and interpretation of the results.

Analysis

Room for improvement analysis

To estimate the maximum costs that could be saved, a room for improvement analysis was performed. In this analysis, we assumed that none of the patients without a need for hospital assessment would have been referred, while all patients with sepsis with a need for hospital assessment would have been referred.

Impact of prediction model on referral rates and costs

The costs and effects of current care were compared with scenarios in which referral corresponded with the score of the prediction model. The four scenarios differed in the cut-off of the score of the model: (1) a cut-off score of 2 (no referral at a score <2 and referral at ≥2), (2) a cut-off score of 3, (3) a cut-off score of 4, and (4) a mixed scenario: patients below a score of 2 were not referred, while patients with a score of 4 or higher were referred. In case of a score of 2 or 3, referral remained unchanged. For each scenario, the costs of all branches were summed and compared with each other.

Sensitivity analysis

To estimate the uncertainty around the outcome, we performed deterministic and probabilistic sensitivity analyses. Uncertainty ranges of costs for the branches with more than 30 patients were estimated with bootstrapping and were assumed to be normally distributed, while ranges of costs for the branches with less than 30 patients were assumed to range from the cheapest to the most expensive patient with a gamma distribution. Probabilistic sensitivity analysis was performed with 1000 runs on the most positive scenario of the prediction model compared with current care. The outcome of these runs was displayed in violin plots.

Results

The impact of referral and final diagnosis on resource use

Resource use collected during the study

Table 1 shows the healthcare costs in the trial for all groups divided by their need of hospital assessment, final diagnosis and whether or not they were referred. As mentioned in the Methods section, only costs of the patients in whom hospital assessment was not necessary were completely used in the calculation of the impact of the prediction model.

Costs of patients in whom hospital assessment was not necessary were higher for referred patients compared with not referred patients. This increase was due to ambulance and hospital costs. Referral led to an increase in total costs from €1373 to €2515 in patients with sepsis, €1379 to €4838 in patients with infection without sepsis and from €2029 to €6209 in patients without infection.

Costs of patients in whom hospital assessment was necessary consisted primarily of hospital costs (€4937–7938 over the different groups), while family and home care added less to the total costs. In patients with sepsis in whom hospital assessment was necessary, the observed costs of the referred patients were lower with €9298 compared with €9687 in not referred patients. In patients without sepsis, costs were higher in the referred patients. To estimate the impact of referral on costs of patients with sepsis, these observed costs were not used, but instead expert opinion outcomes were used. The minimum and maximum values as used in the deterministic and probabilistic sensitivity analyses for all scenarios can be found in online supplemental table 1.

Expert opinion outcomes

All expert opinion results and agreements are shown in the online supplemental table 2. The experts expected that an increase of 38% in hospital days in the observed referred cases with a standard disease pattern (did not die, had an ICU admission or rare complication) should these patients not have been referred. Expert opinion in these four cases differed from a decrease of 2% (due to an expected early death) to an increase of 89%. Of the three cases that were not referred and did not die nor were admitted to the ICU, a decrease in hospital days of 25% was assumed by the experts if they would have been referred at time of inclusion, differing from a decrease of 48% to an increase of 27% between experts. The expert panel assessed two patients with sepsis who were not referred and died within 30 days, but none of the experts expected death could have been prevented by referral. Corresponding mortality and hospital costs at different cut-offs of the prediction model are shown in the online supplemental tables 3 and 4.

Room for improvement analysis

Hospital assessment was deemed necessary in 136 patients with sepsis in the primary study. A total of €53 796 could have been saved in case all these patients would have been referred according to the expert panel judgement, corresponding to €396 per patient with sepsis. If all 158 patients for whom hospital assessment was not necessary would not have been referred to the hospital, this could have saved €84 086. In total, perfect referral could save €137 882 out of €1 583 732 in these 357 patients (mean €386).

Impact of prediction model on referral rates and costs

Observed referral of patients who needed hospital assessment was 88% for patients with sepsis (120 of 136) and 15% for patients who did not need hospital assessment (see online supplemental table 5). Referral rates at a low cut-off (2 or 3) of the prediction model were higher for patients with sepsis (99% and 91%, respectively). However, referral was also substantially higher for patients who did not need hospital assessment. At a high cut-off score of 4, referral rates for patients who did not need hospital assessment were still higher than the observed referral rate, while at the same time referral in patients with sepsis who needed hospital assessment was lower (60%).

In the mixed scenario, where referral was based on the prediction model below a score of 2 and equal to or above a score of 4 and partly on the GP’s opinion (referral as observed if score was 2 or 3), referral of patients with sepsis in whom hospital assessment was necessary increased to 91%. However, referral of patients in which hospital assessment was not necessary was also increased to 26%.

The decision tree analysis shows that the observed probabilities of referral led to the lowest costs with average patient costs of €5890. The average costs of referral based on the prediction model ranged between €5954 and €6742 (table 2). At higher prediction model cut-offs (scores 3 and 4), sepsis mortality will increase according to the expert opinion analysis. At a cut-off of 2 and the mixed scenario, a benefit in costs of increased referral of patients with sepsis is offset by the increased costs of unnecessary referral. However, the mixed scenario had the lowest costs of the prediction model scenarios resulting in mean costs of €5954 per patient.

Table 2

The total percentage of patients referred, sepsis mortality as estimated with expert opinion and healthcare costs as observed and as modelled using expert opinion at the different prediction model cut-offs

Sensitivity analyses

In a one-way sensitivity analysis, the costs of all different patient groups were varied using both bootstrapped SDs and mean costs (from a minimum of 2 SDs below and above the bootstrapped mean costs). The impact was greatest for patients with sepsis who were not previously referred and for the costs of patients who did not need hospital treatment and had an infection without sepsis (figure 2). This is mainly because the modelled uncertainty was also the largest in these parameters. With maximum costs of €16 614 instead of €9697 per patient, referral of sepsis saves more costs even leading to cost-savings of the prediction model in these extreme assumptions. If the costs of patients who did not need hospital treatment and had an infection without sepsis only added an ambulance ride and emergency department visit (and no hospital days), referral based on the prediction model could save €23 compared with usual care. In the probabilistic sensitivity analyses, costs for the prediction model were higher than usual care in 83% of the runs (figure 3).

Figure 2
Figure 2

Plots of one-way sensitivity analysis; costs for health states were varied independently, and the cost difference, by using the prediction model, is compared with usual care.

Figure 3
Figure 3

Violin plot of probabilistic sensitivity analysis; differences in costs of 1000 runs of implementation of the prediction model costs for health states were varied independently, and the cost difference, by using the prediction model, is compared with usual care.

Utility loss

We observed a median utility at baseline of 0.78 (IQR 0.59–0.89) in the total population and 0.74 (IQR 0.54–0.86) for patients with sepsis who required hospital treatment. Mean utility loss in the first month was 0.36 for these patients, compared with 0.34 in the total population. More detailed results of the utility loss can be found in the online supplemental table 6 and online supplemental figure 1.

Discussion

In this early economic evaluation, we estimated the potential rate of referral and cost impact of the implementation of a new clinical prediction model for adult patients with possible sepsis in the primary care setting. We saw that, when using the model, cost-savings due to referral of patients with sepsis were offset by increased costs due to unnecessary referral for all cut-offs of the prediction model. With ‘perfect’ referral, it is estimated that €137 882 could be saved in the 357 study patients.

It was expected that referral of patients with sepsis previously not referred would have large beneficial effects. However, during expert opinion, 16 of these patients were judged and the impact on hospitalisation the experts gave was smaller than expected. In addition, the impact of referral of patients who did not need hospital assessment on costs was larger than expected. The one-way sensitivity analyses showed that if these costs would only compromise an ambulance ride and emergency department visit, costs of the prediction model and usual care would almost be equal. Together, these costs are crucial for the conclusion of the analysis and should be studied in a randomised setting.

Comparison with literature

Several cost-effectiveness analyses on sepsis prediction models or diagnostics at a hospital level exist,21–23 but we were not able to find other cost-effectiveness analyses of sepsis prediction models in general practice. However, evaluations of prediction models for other acute medical conditions in the primary care setting do exist. A cost-utility analyses of point-of-care troponin testing in patients consulting a GP with chest pain showed a decreased referral rate and cost-savings of €77.25 per patient.24 A cost-effectiveness analysis of a new strategy to rule out deep vein thrombosis in the primary care setting in the Netherlands showed that €138 per patients could be saved at the expense of a very small health loss (0.002 quality-adjusted life-years (QALYs)25). In both studies, the new strategies could safely reduce the number of referrals and decrease hospital costs. In our study population, the proportion of unnecessary referrals is substantially lower, making it more difficult to save costs by decreasing the total number of referrals.

Strengths and limitations

The aim of the study was to perform an early economic evaluation, which is in line with the observational design of the study. Inherently related to the phase of the evaluation, results are less precise than in a full economic evaluation, for instance, due to the lack of observed data on performance of the prediction model in practice, the impact of referral on sepsis costs and the limited number of patients in the observational cohort. In addition, using this observational design as input for the decision tree did result in another two challenges. First, the probability of referral when implementing the sepsis prediction model was based on a hypothetical scenario that could have been different from the actual management of the GP during the study. Second, costs of patients without need for hospital assessment might also not be interchangeable between the referred and not referred patients. For instance, a more frail patient could have a higher probability to be (unnecessarily) referred, but this might also result in other additional resource use that might be omitted if a younger vital patient would be (unnecessarily) referred.

A strength of our study is that costs from different sources were used to gain a broad oversight on different cost components. The necessity to use expert opinion could be seen as either a strength or a limitation. Compared with literature data aligning with our population or prospective data on the impact in a clinical trial, it is a limitation. However, these limitations are also in correspondence with the early nature of the economic evaluation. Therefore, the extensive set-up, including individual detailed patient cases to evaluate the potential impact of referral on patients with sepsis instead of using observed data that could be prone to bias due to the observational design of the study could then be seen as a strength. In the observed data, a smaller than expected cost difference was seen between referred and not referred patients of approximately €400, which we partly allocate to bias. For instance, referred patients could differ in severity of illness and fragility from patients who also needed hospital assessment but were not referred. As the impact referral of sepsis was higher in the expert panel, the cost-saving due to sepsis referral would have been lower when using the observed data and therefore total cost increases even larger. However, cost-savings due to sepsis referral could also still have been underestimated with use of the expert panel. Because the impact of referral is also not known in literature, our expert opinion was the most accurate estimation to our opinion but could have biased results towards the observed management. For example, intravenous antibiotics are appropriate in case of positive blood cultures, but blood cultures were only obtained in referred patients. Possibly, some patients who were successfully treated at home with oral antibiotics would have shown positive blood cultures if they had been collected. Another limitation is the possibility that GPs were biased by the Hawthorne effect. The added value of the model to usual care could be greater than observed during the study, as the study protocol required the measurement of all vital signs and the GPs always considered the diagnosis sepsis in patients included in the study.

Implication for research and clinical practice

As we showed, a reduction in costs is not likely if GPs use only the prediction model to refer patients to the hospital; more research is needed into the effects of our model in routine care comparing with a usual care group in which measurements are up to the GP to perform. As explained in the previous paragraph, beneficial effects of the model may have been underestimated in this study. Also, a reduction in costs is not a precondition for a new intervention, but additional costs per QALY gained should be acceptable. To measure the effects more accurately on health outcomes and costs, a large prospective randomised trial should be performed, in which the effects of the new sepsis model are evaluated in practice. Given the results of the current study, it can be concluded that the discrimination of our new sepsis score is not sufficient to replace the clinical judgement of the GP based on the results of this early economic evaluation. Therefore, we do not propose for GPs to use strict cut-off points of the model to decide to refer a patient, but rather to use the score to estimate the risk of sepsis and incorporate this information in their overall judgement.

In conclusion, guidance for referral of adult patients with suspected sepsis in the primary care setting using any cut-off point of the sepsis prediction model is not likely to save costs. The model should only be incorporated in sepsis guidelines for GPs if improvement of care can be demonstrated in an implementation study.

Data availability statement

Data are available upon reasonable request. De-identified patient data that underlie the presented results in this article will be made available to researchers with a methodological sound proposal. A data dictionary of all variables will be provided. Data will be available from 3 months until 5 years after publication. Requests should be made to the corresponding author.

Ethics statements

Patient consent for publication

Ethics approval

This study involves human participants and was approved by the METC Utrecht, and is registered under number 18/169. Participants gave informed consent to participate in the study before taking part.

Acknowledgments

The authors thank the patients included in the study for their consent and returning the questionnaires. The authors also thank all participating GPs and personnel of the GP cooperatives, and Idelette Nutma for her valuable input as patient representative.

This post was originally published on https://bmjopen.bmj.com