Abstract
Objectives Current guidelines for empirical antibiotic treatment poorly predict the presence of 3rd generation cephalosporin resistant Enterobacteriaceae (3GC-R EB) as a cause of infection, thereby increasing unnecessary carbapenem use. We aimed to develop diagnostic scoring systems to better predict the presence of 3GC-R EB as a cause of bacteraemia.
Methods A retrospective nested case-control study was performed that included patients ≥18 years in whom blood cultures were obtained and intravenous antibiotics were initiated. Each patient with 3GC-R EB bacteraemia was matched to four control infection episodes within the same hospital, based on blood culture date and onset location (community or hospital). Starting from 32 described clinical risk factors at infection onset, selection strategies were used to derive scoring systems for the probability of community- and hospital-onset 3GC-R EB bacteraemia.
Results 3GC-R EB bacteraemia occurred in 90 of 22,506 (0.4%) community-onset and in 82 of 8,110 (1.0%) hospital-onset infections, and these cases were matched to 360 community-onset and 328 hospital-onset control episodes, respectively. The derived community-onset and hospital-onset scoring system consisted of 6 and 9 predictors, respectively, with c-statistics of 0.807 (95% confidence interval 0.756-0.855) and 0.842 (0.794-0.887). With selected score cutoffs, the models identified 3GC-R EB bacteraemia with equal sensitivity as existing guidelines, but reduced the proportion of patients classified as at risk for 3GC-R EB bacteraemia (i.e. eligible for empiric carbapenem therapy) with 40% in patients with community-onset and 49% in patients with hospital-onset infection.
Conclusions These prediction rules for 3GC-R EB bacteraemia may reduce unnecessary empiric carbapenem use.
Introduction
As a consequence of the emergence of infections caused 3rd generation cephalosporin (3GC) resistant Enterobacteriaceae (3GC-R EB; in this manuscript used synonymously with extended-spectrum β-lactamase (ESBL) producing Enterobacteriaceae), physicians are increasingly faced with the question which patients need empiric antibiotic treatment covering these pathogens. Hence, patients and physicians might benefit from prediction rules for 3GC-R EB. Although risk factors for carriage of ESBL-producing Enterobacteriaceae at hospital admission [1–4], and factors distinguishing ESBL- and carbapenemase-producing Enterobacteriaceae as a cause of bacteraemia have been determined [5–8], there are no prediction rules for identifying 3GC-R EB as a cause of bacteraemia at the time that empiric therapy must be started.
Current Dutch empiric treatment guidelines designate patients at risk of infection caused by 3GC-R EB based on prior colonization or infection with 3GC-R EB or on prior exposure to cephalosporins or fluoroquinolones, as these were identified as risk factors in patients with bacteraemia caused by these pathogens [9]. As carbapenems are the treatment of choice for 3GC-R EB, adherence to these guidelines may stimulate overuse of these antibiotics. Indeed, applying these recommendations for all patients needing empiric antibiotic treatment in a population with a pre-test probability for 3GC-R EB of 0.7%, revealed that 19% of all patients were classified as at risk for 3GC-R EB infection and thus eligible for empiric carbapenem therapy (referred to as test positivity rate), while at the same time only 50% of all patients with 3GC-R EB bacteraemia would be classified as at risk (referred to as sensitivity) [10]. Only using prior identification of 3GC-R EB carriage as risk factor, would reduce the test positivity rate to 4%, at the cost of a reduction in sensitivity to 42%.
We aimed to develop prediction rules to better identify, among patients needing intravenous empiric antibiotic therapy, those being infected with 3GC-R EB. We were specifically interested in the balance between sensitivity and test positivity rate. In this derivation study, we compared these quantities to those of the two basic strategies introduced above, which rely on prior identification alone (prior identification model), or in combination with prior exposure to cephalosporins and fluoroquinolones (two-predictor model). We focused on predicting 3GC-R EB bacteraemia, as these infections can be objectively assessed in retrospect, and an immediate start with appropriate antibiotics is indicated. We decided to derive separate prediction rules for community-onset and hospital-onset infections, as we assumed that factors driving spread of 3GC-R EB within these two settings are distinct.
Methods
Setting and patients
This was a retrospective nested case-control study involving 8 hospitals, of which 3 university hospitals, in the Netherlands. Between January 1st 2008 and December 31st 2010, we included all consecutive patients of 18 years of age or older in whom a blood culture was obtained and intravenous broad-spectrum β-lactam antibiotics (i.e. not penicillin or flucloxacillin), aminoglycosides, and/or fluoroquinolones were started on the day of the blood culture or the day after, irrespective of duration. Patients receiving any of the eligible antibiotics on the day of blood culture obtainment were excluded if these had been initiated prior to this day (see Supplementary Table 1 for illustrating examples). In addition, patients with 3GC-R EB bacteraemia in the year prior were excluded, as it was assumed that treating physicians would always provide therapy aimed at these organisms in case of renewed infection. Patients could be included more than once, if a subsequent episode complied with in- and exclusion criteria. Additional information on hospital characteristics, study periods, and databases used in each of the hospitals is provided in Supplementary Table 2.
Infection episodes were separated into two cohorts: the community-onset cohort comprised episodes in which the first blood culture was collected during the first three calendar days of hospitalization, and the hospital-onset cohort consisted of episodes in which blood cultures were obtained later during hospitalization.
The causative pathogen of each episode was based on the results of blood cultures obtained on the day that antibiotics were started and the day before. In both cohorts, the case population comprised all consecutive infection episodes with 3GC-R EB bacteraemia (see Supplementary Table 2 for definition of 3GC resistance in each of the hospitals). We estimated that a study period of three years in the participating hospitals would yield 100 patients with 3GC-R EB bacteraemia in both cohorts, which would allow initial logistic regression with 10 variables, based on the 10 events per variable recommendation [11].
The control population was defined as all other infection episodes, including non-bacteraemic episodes and episodes with blood cultures yielding non-resistant Enterobacteriaceae, other bacteria or fungi. From this population four controls were matched to each case, a ratio chosen because of minimal gains in statistical power with more controls [12]. Controls were matched on hospital, being in the community or hospital-onset cohort, and being closest in time to the blood culture day of the case episode.
Due to its retrospective nature, the Dutch Medical Research Involving Human Subjects Act did not apply to this study. In each of the participating hospitals, applicable local guidelines for noninterventional studies were followed. In accordance with Dutch regulations, informed consent was waived for the study. Reporting of this study was in accordance with the TRIPOD Statement [13,14].
Data collection
All selected cases and controls were subjected to chart review to obtain information that was considered available at the moment that the initial antibiotics were prescribed (referred to as infection onset). Blinding for the outcome during chart review was not considered feasible. Please refer to Supplementary Table 3 for an overview of all collected variables.
Statistical analysis
Two separate prediction models were constructed, one for community-onset and one for hospital-onset infections. Data analyses were performed in R (version 3.2.2) [15], including packages mice 2.25 [16], rms 4.5-0 [17], pROC 1.8 [18], and xtable 1.8-2 [19]. Descriptive analyses of predictors were based on non-missing data only. Some variables were aggregated because of high correlation, low prevalence, and/or similar associations with the outcome (indicated in Table 1). Additionally, the number of categories for suspected sources was reduced to four by combining categories with low frequencies into a single remaining group (original categories in Supplementary Table 3), and categories for antibiotic use were created based on prevalence and assumed predictive power for 3GC-R EB infection. Twenty imputed datasets were created to deal with missing values during the modelling stage. In the Supplementary Material, missing data patterns and the exact imputation procedure are described.
Starting from 32 potential predictors, the first step of model creation involved selection of ten relevant predictors based on (1) observing the strength of their association with 3GC-R EB bacteraemia (without statistical hypothesis testing), and (2) considerations related to coverage of the entire spectrum of known risk factors for 3GC-R EB, and (3) ease-of-use of any resulting model. The second step involved removing redundant variables from the model, which was performed by backward stepwise logistic regression analysis until all remaining predictors had p-values < 0.2 in the Wald test (pooled from 20 imputed datasets by means of Rubin’s rules) [20]. Continuous predictors were initially introduced into models with restricted cubic spline functions with three knots to allow for non-linear associations. Finally, we evaluated by means of the Akaike’s Information Criterion (AIC) if simplification to a linear predictor was possible.
Regression coefficients of the final models were pooled over imputed datasets by means of Rubin’s rules and shrunk according to model optimism (see description further on). Furthermore, developing a model in a case-control study artificially increases the prevalence of the outcome, which means that predicted probabilities generated by the model do not reflect true probabilities within the full cohorts. Test positivity rates, and positive and negative predictive values are similarly affected. Therefore, intercepts of the models were adjusted for the sampling fraction of the controls, and controls were weighted by the inverse of the sampling fraction, as previously described [21]. All quantities presented in this paper reflect the values within the original full cohorts.
Calibration of the predicted and observed probabilities was visually inspected for separate imputed datasets. All other performance parameters were averaged over the imputed datasets. Discrimination was assessed with the area under the curve for receiver operating characteristic curves (referred to as C-statistic). Sensitivity, specificity and positive and negative predictive values, and test positivity rate (i.e. fraction of the population classified as at risk of 3GC-R EB bacteraemia) were calculated for different cutoffs of the predicted risk. These model performance characteristics were compared to those of the prior identification model and two-predictor model. In the prior identification model, patients with identification of 3GC-R EB in the year prior to the infection episode were classified as test-positive. In the two-predictor model, also patients with cephalosporin or fluoroquinolone use during the prior two months were considered test-positive.
A simplified score was created by multiplying the regression coefficients with a constant, followed by rounding to easy-to-use values. Performance of this score was determined similarly.
Estimation of model optimism
Optimism results from the fact that models are developed on a population sample and suffer from overfitting, which jeopardizes generalizability to other populations, including future patients for which a model will be used [22]. By means of a bootstrapping technique, the expected performance loss (e.g. lower sensitivity, specificity, and predictive values, and altered test positivity rate) when applying the model within the total population is quantified. For the two regression models, optimism was estimated by creating 2000 bootstrap samples, creating a new prediction model for each of these samples, and comparing the model’s performance in the original and bootstrapped data. Optimism was estimated for model coefficients, derived odds ratios and C-statistics. During the same procedure, the expected overestimation of sensitivity and underestimation of test positivity rate due to optimism was quantified by applying a probability cutoff above which patients are classified as test-positive. For this evaluation, the probability cutoff was selected such that sensitivity corresponded to the two-predictor model. In the Supplementary Material, technical details of the bootstrapping procedure are presented.
Results
Probabilities of 3GC-R EB bacteraemia were 0.4% (n = 90) for the community-onset infection cohort (22,506 episodes) and 1.0% (n = 82) for the hospital-onset infection cohort (8,110 episodes) (Figure 1). These case populations were matched to 360 community-onset control episodes and 328 hospital-onset control episodes (Table 1). Multiple selection of individual patients, albeit with different episodes, as case and/or control were allowed and occurred 8 times within the community-onset, and 9 times within the hospital-onset dataset. Isolated pathogens from blood cultures and initial antibiotic therapy are presented in Supplementary Tables 4 and 5.
Community-onset infection
The prediction model for 3GC-R EB bacteraemia in community-onset infection consisted of six variables (Table 2). It showed adequate discrimination (C-statistic = 0.808 (95% CI 0.756-0.855), also after correction for optimism (C-statistic = 0.775 (95% CI 0.705-0.839)), and calibration (Supplementary Figure 1).
The derived scoring system had a performance similar to the original model (Supplementary Figure 2a; C-statistic 0.807 (95% CI 0.756-0.855), not corrected for optimism). Table 3 and Figure 2a depict the trade-off between sensitivity and test positivity rate at different cutoffs for being at risk of 3GC-R EB bacteraemia. These can be contrasted to the fixed values for the prior identification model (sensitivity 24.4% and test positivity rate 2.8%), and the two-predictor model (sensitivity 53.9% and test positivity rate 21.5%).
For instance, patients with a score of 120 or higher would have a probability of 1.7% (positive predictive value) of having 3GC-R EB bacteraemia, and with this score as a cutoff 45.7% of all patients with 3GC-R EB bacteraemia would be missed (1 - sensitivity). This sensitivity (or proportion missed) is comparable to the simpler two-predictor model; however, the scoring system reduces eligibility for carbapenem use (test positivity rate) by 40%, from 21.5% to 12.8%.
Bootstrapping of the model indicated that when applying this cutoff in a future patient population some performance loss should be expected due to model optimism. The optimism-corrected sensitivity for future populations was 6.2 percentage points lower, whereas a change in prevalence was hardly noticeably (Table 4; please note that the percentages presented relate to the regression model, not to the score described in the paragraph above).
Hospital-onset infection
The hospital-onset prediction model contained nine variables (Table 5), and also had adequate discrimination (C-statistic = 0.842 (95% CI 0.793-0.886), optimism-corrected 0.811 (95% CI 0.7420.873) and calibration (Supplementary Figure 3).
The derived scoring system again performed very similar to the original model (Supplementary Figure 2b; C-statistic 0.842 (95% CI 0.794-0.887), not corrected for optimism). In Table 6 and Figure 2b, sensitivity and test positivity rate at different scoring cutoffs are compared to the prior identification model (sensitivity 35.4% and test positivity rate 5.2%), and the two-predictor model (sensitivity 79.3% and test positivity rate 52.8%).
Patients with scores of 110 or higher have a 3.1% probability of 3GC-R EB bacteraemia, and with this cutoff 18.5% of all patients with 3GC-R EB bacteraemias would be missed, similarly to the two-predictor model. Yet, carbapenem eligibility would be reduced with 49% (27.0% vs. 52.8%). In this scenario, bootstrapping indicated that sensitivity in future patient populations should again be expected to be somewhat lower (-5.3%; Table 4).
Discussion
We developed scoring systems to more accurately identify patients with bacteraemia caused by 3GC-R EB among those in whom empiric intravenous antibiotic therapy aimed at Gram-negatives is initiated. The scores consist of a limited number of clinical predictors that can easily be assessed based on the information available at the initial examination of a patient presenting with infection, before prescription of initial antibiotics, such as medical history, prior antibiotic usage, prior microbiology results, and infection characteristics. The calculated score can directly be converted to a probability that the patient suffers from 3GC-R EB bacteraemia, and depending on this probability, a decision can be made whether initial antibiotics should include coverage for 3GC-R EB or not. Implementing the scoring systems could improve appropriateness of empiric antibiotic therapy and reduce unnecessary use of broad-spectrum therapy. Compared to a basic model incorporating only prior 3GC-R EB identification and exposure to cephalosporins and/or fluoroquinolones, eligibility for empiric carbapenem use could be reduced by 40%-49% while maintaining a similar risk of missing patients with 3GC-R EB bacteraemia.
With a global emergence of antibiotic resistance, physicians must assess the risks of missing resistant causative pathogens when starting empiric antibiotic treatment [23]. Risk avoidance, albeit imaginable in many situations, is one of the driving forces for broad-spectrum antibiotic use, fuelling the global pandemic of antimicrobial resistance. Better prediction rules for infections caused by antibiotic-resistant pathogens are therefore needed. Prediction systems have been developed for Gram-negative bacteraemia in septic patients [24], carriage of or infection with ESBL-producing Enterobacteriaceae at hospital admission [1,25,26], and distinguishing bacteraemia with ESBL- or carbapenemase-producing pathogens from bacteraemia with susceptible Enterobacteriaceae [5–8].
Yet, guidance on incorporating the risk of 3GC-R EB in selecting empiric antibiotics is currently lacking. A recently published flow chart for initiating empiric therapy with a carbapenem in critically ill patients with suspected Gram-negative infection included predictors for 3GC-R EB carriage at hospital admission and in case of Enterobacteriaceae bacteraemia, without formal evaluation of performance [27]. For clarity, 3GC-R EB bacteraemia is a subset of Enterobacteriaceae bacteraemia, which is a subset of all bacteraemia episodes. Risk factors for any of these overarching categories may alter the probability of bacteraemia caused by antibiotic-resistant Enterobacteriaceae. This is corroborated by the strong predictive role of the suspected source of infection in our prediction models, which likely reflects the likelihood that Enterobacteriaceae play a role as causative pathogens. It emphasizes the need to select a clinically meaningful patient population when deriving a prediction rule. We therefore focused on all patients receiving their first dose of antibiotic therapy aimed at Enterobacteriaceae, rather than selecting patients that had, in retrospect, bacteraemia.
Due to the effect of including all patients with a clinical suspicion of infection, predicted probabilities of 3GC-R EB bacteraemia may seem low (0.4-1.0%). Yet, in a previous Dutch study, an 8.3% 3GC resistance rate among Enterobacteriaceae bacteraemia isolates resulted in a similarly low prior probability of 3GC-R EB bacteraemia in case of suspected Gram-negative infection (0.7%) [10]. Although our data originated from 2008-2010, the prevalence of 3GC resistance among Enterobacteriaceae only marginally increased in the Netherlands since then, and most Western European countries currently have similar prevalence rates of 3GC resistance among Enterobacteriace, namely between five and fifteen percent [28]. Model updating to reflect the local prevalence of resistance will generally improve calibration [22], but our model provides a useful universal backbone due to the incorporation of widely reported risk factors [29].
With the newly developed prediction rules, we aimed to achieve similar sensitivities as in existing prediction schemes, while at the same time reducing the proportion of patients eligible for broad-spectrum antibiotics (test-positives). This leads to diverging performance; for community-acquired infections we were “satisfied” with a sensitivity of 54.3%, where this figure was 81.5% for hospital-onset infections, and this yielded test-positive proportions of 12.8% and 27.0% for community-onset and hospital-onset infections, respectively. Yet, both prediction rules can also be used to increase sensitivity, which will – as a matter of fact – also increase the proportion of test-positivity. The optimal cut-off cannot be defined as each point has a different balance between the risk of overprescribing carbapenems and inappropriate empiric antibiotics.
That balance may be different for certain bug-drug combinations. For instance, the acceptance for a delay in adequate treatment of enterococcal bacteraemia may be different than for carbapenemase-producing Enterobacteriaceae, and might be different in a clinically stable than in a haemodynamically unstable patient [30]. Taking the long-term population effects of, for instance, carbapenem overuse into that equation is difficult, as these effects have not been sufficiently quantified [31], and depend on extraneous factors such as hospital hygiene and the baseline prevalence of carbapenem-resistant micro-organisms [32].
As expected, prior identification of 3GC-R EB was the strongest predictor in both models. In the Netherlands, screening for carriage is only practiced in intensive care units and for highly selected risk groups. Hence, identification was mostly based on previous clinical cultures, and an unknown proportion of actual 3GC-R EB carriers are classified as non-carriers. Naturally, more screening will further increase the sensitivity of this predictor for bacteraemia with 3GC-R EB. Yet, as infection rates among colonized patients are low [33,34], it is unsure whether positive predictive values of models will improve. In fact, if low-risk carriers would be identified by screening more frequently, positive predictive values might even decline.
Prior antibiotic use, on the other hand, had little predictive value in the community-onset model, and was not retained in the hospital-onset model. This seems to contradict the results from other studies. Yet, in such studies associations resulted from comparing infections with resistant Enterobacteriaceae to their sensitive counterparts [29], which exaggerates the role of antibiotic use [35].
We applied a nested case-control design for this study, implying that instead of analysing the full cohort, a representative subset of patients without 3GC-R EB (i.e. the control population) was analysed. The case population, however, (i.e. patients with 3GC-R EB) was analysed in full. This design was chosen for efficiency reasons, reducing the amount of data collection by 95% while accepting a small loss of precision. Knowing the size of the original cohort, we were able to extrapolate the case-control data to the full cohort, resulting in probabilities generalizable to clinical practice. Within the community-onset and hospital-onset cohorts, we matched on hospital to adjust for hospital-specific practices (independent of the incidence of 3GC-R EB) and on date to avoid effects of season-specific fluctuations in incidence and risk factors.
Our study has a limited sample size compared to the initial number of predictors studied. This may simultaneously lead to falsely rejecting predictive variables (a power problem) and selection of spurious predictors (overfitting resulting in overoptimism of model performance) [14]. We applied high p-value thresholds for variable retention in models to overcome our relatively low power, and internal validation by means of bootstrapping to quantify optimism in our model selection strategy. The latter resulted in optimism-adjusted odds ratios and C-statistics, giving insight in values expected when applying models to an external cohort. Expected performance loss when selecting specific probability cutoffs for clinical use has also been calculated (Table 4). Naturally, both models need prospective external validation before clinical implementation, for two reasons. First, even after shrinkage, optimism may still be present, as some steps could not be replicated in the bootstrap procedure, such as aggregation after observing similar associations with the outcome, simplification of continuous variables to linear predictors, and derivation of a scoring system. Second, the current study relied on data available in medical charts. We used pragmatic in- and exclusion criteria, which might not fully reflect intended clinical use, and as data collection was not blinded for outcome, information bias is not excluded. Moreover, potentially relevant predictors, especially for community-onset infection, such as international travel, animal contact, known colonization in household members, and dietary preferences could not be collected [29]. The same holds for determination of colonization pressure, which might be a relevant predictor for hospital-onset infections [32]. External validation studies are currently ongoing.
A study limitation is that the outcome was restricted to bacteraemic episodes, not including non-bacteraemic infections caused by 3GC-R EB, which are more common than bacteraemic infections in patients being empirically treated [10]. Yet, with an overall prevalence of <5% it is unlikely that these infections had a substantial impact on the composition of control groups. Future studies may consider classifying these infections as outcomes. However, due to the more benign course, initial treatment with carbapenems may not have a high priority in non-bacteraemic infections.
Another limitation of our study is that empiric coverage of 3GC-R EB is just one aspect of selection of appropriate empiric therapy. Other potential pathogens and resistance mechanisms, such as Pseudomonas aeruginosa, might justify alterations in empiric treatment even in the absence of risk factors for 3GC-R EB. In some countries, high incidences of infections with carbapenemase-producing Enterobacteriaceae may limit usefulness of our models. On the other hand, escape therapy for 3GC-R EB might not necessarily involve carbapenems, due to underlying resistance mechanisms other than ESBL, or favourable patterns of co-resistance. Ideally, frameworks for selecting empiric therapy should evaluate the probability of success of many different antibiotic agents. An example of such an approach is TREAT, an automated system for recommending antibiotic treatment based on, amongst others, patient and infection characteristics and local epidemiology [36]. TREAT can predict the presence of Gram-negative causative pathogens in infection with some accuracy [37], but performance with regard to resistant variants remains unknown. However, TREAT has not been widely adopted [38], and simple prediction rules may be easier to incorporate into clinical practice.
Furthermore, treating physicians incorporate more factors in their clinical decision making regarding empiric antibiotics than those provided by current risk stratification schemes in guidelines. In both this and our previous study [10], empiric carbapenem use was much lower than it would have been with full guideline adherence (Supplementary Table 4). As a result, achievable reductions in empiric carbapenem use in real life may be lower than anticipated in our study. Yet, we consider it important that antibiotic guidelines do not stimulate unnecessary broad-spectrum antibiotic use [39].
In conclusion, identification of patients with an infection caused by 3GC-R EB amongst all patients that need empiric antibiotic therapy remains a trade-off between acceptably low levels of unnecessary empiric carbapenem use and appropriate treatment in true 3GC-R EB bacteraemia cases. The prediction rules developed quantify this trade-off for patients that need empiric treatment, and might offer improvement in detecting such patients, compared to current international guidelines. As such, they provide useful starting points for optimizing empiric antibiotic strategies.
Transparency declaration
W.C.R. has received research funding from Merck Sharp & Dohme (MSD). All other authors declare no conflicts of interest. This work was supported by a research grant from the Netherlands Organisation for Health Research and Development (project number 205200007) to H. S. M. A.
We would like to thank Evelien Stevenson and Tim Deelen for assistance with the data collection, and Maaike van Mourik and Loes de Bruin for assistance with the statistical analysis.
Results from this study were presented at the 26th European Congress of Clinical Microbiology and Infectious Diseases, Amsterdam, the Netherlands (9-12 April 2016, E-poster eP0083), and at ASM Microbe 2016, Boston, MA, United States of America (16-20 June 2016, session 445). This manuscript is also available as an electronic preprint from www.bioRxiv.org (accessible via doi 10.1101/120550).