- Research article
- Open Access
- Open Peer Review
External validation study of a clinical decision aid to reduce unnecessary antibiotic prescriptions in women with acute cystitis
© The Author(s). 2017
- Received: 27 March 2017
- Accepted: 20 September 2017
- Published: 2 October 2017
Empirical prescribing of antibiotics to women with symptoms of acute cystitis prior to culture results is common, but subsequent culture results are often negative. A clinical decision aid for prescribing decisions in acute cystitis was previously developed that could reduce these unnecessary antibiotic prescriptions but has not been validated. This study sought to validate this decision aid for empirical antibiotic prescribing decisions in a new cohort of women with suspected acute cystitis.
External validation study of a clinical decision aid in 397 women with symptoms of acute cystitis, involving 230 Canadian family practitioners across Canada between 2009 and 2011. The sensitivity and specificity of the decision aid compared to a gold standard positive urine culture (defined as ≥102 cfu/ml (≥105 CFU/L)) was determined, and compared with physician management, and the earlier development cohort study estimates. Other outcomes assessed were total antibiotic prescriptions, unnecessary antibiotics for negative urine cultures, and recommendations for urine culture testing. Chi-square tests were used for unpaired comparisons, adjusted for physician clustering. McNemar’s test was used for paired comparisons.
There were 245/397 (61.7%) positive urine cultures. The cystitis aid sensitivity was 202/245 (82.5%, 95% Confidence Interval (CI)) = 77.1%, 86.8%), compared to 167/208 (80.3%) in the previous development cohort (p = 0.54), and 239/245 (97.6%) by family physicians in the current study (p < 0.001). Specificity was low for physicians (10/152, 6.6%) compared to the decision aid (54/152, 35.5%; p < 0.001, resulting in more antibiotic prescriptions by physicians (381/397, 96.0%) than would occur with decision aid recommendations (300/397, 75.6%, p < 0.001). Unnecessary antibiotic prescriptions where urine cultures were negative would be reduced an absolute 11.1% with cystitis aid recommendations (98/397, 24.7%) compared to usual physician care (142/397, 35.8%; p = 0.001). Urine cultures would also be reduced (97/397, 24.4% decision aid vs 351/397, 88.4% physicians; p < 0.001).
A 3-item clinical decision aid demonstrated reproducible accuracy in two cohorts of women with acute cystitis symptoms. Clinically important reductions in total and unnecessary antibiotic use, as well as urine culture testing, could result with routine clinical use compared to current empirical physician management practices.
- Antibiotic prescriptions
- Decision aid
Uncomplicated urinary tract infection (UTI) is one of the most common bacterial infections encountered in the community . Most are cases of acute cystitis in women where the causative organism is Escherichia coli in 75 to 95% of instances . Over time, the prevalence of antibiotic resistant E.coli has increased . The volume of antibiotics prescribed in the community is known to be associated with resistance at both a national and individual level [3, 4]. Reducing unnecessary antibiotic use in primary care can reduce community resistance levels . However, determining whether antibiotics are needed can sometimes be challenging.
In acute cystitis, women present with uncomfortable symptoms whereas a urine culture result that could identify if a uropathogenic bacteria is present may take up to 3 days. Physicians typically prescribe antibiotics empirically based on clinical judgement , an approach endorsed in expert guidelines . However, in 40% of empirically prescribed antibiotics for acute cystitis, urine cultures are negative or there is no significant growth .
Previously, a clinical decision aid was developed that demonstrated the potential to reduce total and unnecessary antibiotic use in managing acute cystitis compared to usual family physician care . However, an attempt to validate the aid in community practice found one of the variables was not reliable, necessitating a modification to the original aid . The modified acute cystitis decision aid criteria are dysuria (burning or discomfort with urination), a positive urine for leukocytes (>trace amount), and urinary nitrites (any present) . If two or more criteria are present, a > 70% prevalence of a positive culture was demonstrated  and empirical antibiotics are suggested, consistent with expert recommendations [1, 6]. With one or fewer criteria (26–38% positive cultures) [7, 8], a culture is recommended with antibiotic treatment based on culture results.
Standards for validating clinical decision aids require independent validation in different populations to ensure reproducibility prior to clinical application [9, 10]. The objective of this study was to assess the validity of this modified acute cystitis decision aid by determining its performance in a new cohort of women who visited family physicians with cystitis symptoms. Specifically, we sought to determine the sensitivity and specificity of the cystitis decision aid for identifying bacterial infection, the reproducibility of these estimates compared with the development cohort estimates , and the potential impact of the decision aid on antibiotic use compared with usual clinical care by physicians.
Study design and setting
A pragmatic, prospective observational study of women presenting to family physicians with symptoms of cystitis was conducted. Pragmatic studies seek to include typical patients encountered in routine clinical care settings and managed with usual expertise and resources . The TRIPOD statement (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) format for reporting prediction model validation study results was followed .
Between 2009 and 2011, 15,742 family and general practitioners from across Canada were randomly selected (2008 Scott’s Canadian Medical Directory) and contacted about a study of acute cystitis. Sampling was stratified by province, proportional to the national distribution of physicians. Physicians were eligible if they conducted community family or general practice, provided care to adult women, and did not limit their practice to psychotherapy, emergency medicine, palliative care, sports medicine, or hospitalist care. Materials were available in English and French.
Study participants and data collection
Participating physicians were asked to assess women presenting with new urinary tract symptoms where they considered acute cystitis a possible diagnosis. Physicians prospectively enrolled up to four women at their discretion, according to specified inclusion and exclusion criteria. Females 16 years of age or older presenting with new urinary symptoms suggestive of uncomplicated cystitis were eligible. Excluded were children, men, pregnant women, nursing home residents and those unable to understand English or French.
Standardized forms to document clinical findings and urine chemical strips (Siemens Multistix 8 SG, Siemens Healthcare Diagnostics Inc., USA), to test for leukocyte esterase and nitrites, were provided. A urine culture was requested for all women. Test strips and urine cultures physicians indicated they would not normally have ordered were paid for from study funds. No company or funding body had any role in the design, collection or analysis of the data, or drafting of the manuscript. Urine cultures were submitted to community laboratories that physicians’ routinely utilized. Testing procedures were not standardized in order to reflect regular clinical practice.
Physicians documented selected co-morbidities and whether an antibiotic was prescribed. The cystitis decision aid criteria were not provided and were not identifiable on the data collection form. As cultures were requested in all cases, physicians recorded whether they would have normally performed a dip urinalysis or ordered a urine culture, to reflect their usual practice. Women provided consent separately for the urine culture result.
The reference standard for diagnosis of a urine infection was a positive urine culture with a colony count ≥102 cfu/ml (105 CFU/L), as recommended for symptomatic women . Each culture report was reviewed by two investigators, one of whom was a microbiologist (TM), to determine if an infection with a significant uropathogen was present. The laboratory culture reports were reviewed blinded to the clinical variables that family physicians had recorded on a separate form.
Outcomes and statistical analysis
The primary outcome to assess the cystitis decision aid performance was the accuracy for identifying infection, indicated by the sensitivity and specificity of the aid. These estimates were compared to those from the original development cohort study to assess reproducibility . The proportion of unnecessary urine cultures, total antibiotics and unnecessary antibiotics recommended were also compared with development cohort estimates , and for usual clinical care as recorded by physicians in the current study, to assess potential clinical impact.
Sensitivity of the decision aid was determined as the proportion of all positive urine cultures that would be identified in women with two or more criteria (antibiotic recommended), and specificity as the proportion of negative cultures in those with 1 or fewer criteria (culture recommended). Physician sensitivity and specificity was estimated similarly using their recommendation for an antibiotic. An antibiotic prescription was considered unnecessary if the subsequent urine culture result was negative.
The sample size was based on estimating resistance rates for a separate report using this data . All women from that study with clinical information needed to apply the decision aid, and a culture result, were eligible for the current validation study. Categorical variables were described using frequencies and percentages, with means and ranges for continuous variables. The sensitivity and specificity of the decision aid threshold of 2 or more criteria present was determined in relation to the gold standard positive urine culture result. These estimates were compared to development cohort estimates  using a Pearson’s chi-square test, adjusted for the clustering of cases by physician (STATA version 12; Statacorp, Texas). McNemar’s test for paired comparisons was used to compare the proportion of urine cultures, unnecessary cultures, total antibiotics prescribed and unnecessary antibiotics recommended by the decision aid to observed care by physicians. Complete–case analysis was used to handle missing data . Women not included because of missing culture results were compared with women included in the analysis for differences in predictor or outcome variables. The Research Ethics Board of the Mount Sinai Hospital (Toronto, Ontario) approved the study.
Women with clinical records (n = 722) excluded from the validation cohort analysis (n = 325) due to missing urine cultures, dip urinalysis or clinical information needed to calculate the decision aid data did not differ from included women in the proportion with dysuria or urine nitrites, but were less likely to have positive leukocytes (Additional file 1: Table S1). Otherwise, excluded women were similar in clinical characteristics and management by physicians. Excluded women were younger than those included in the validation cohort analysis, but similar in age to women in the development cohort. No association was found between age and having a positive urine culture in either the validation (p = 0.39) or development cohorts (p = 0.69).
Characteristics of women with clinically suspected acute uncomplicated cystitis: External validation cohort and development cohort 
≤ 50 years old
> 50 years of age
Prince Edward Island
History of fever
Would order culture
Urine leukocytes (>trace)
Urine nitrites (any)
Positive Urine Culture
Compared to the development cohort  (Table 1), there were no differences in the distributions of decision aid predictor variables (dysuria, leukocytes, nitrites). The development cohort was younger (45.2 years) than the validation cohort (50.8 year, p < 0.001). More culture results were obtained for the development cohort (84.6%) than the validation cohort (52.8%, p < 0.001). However, the prevalence of positive cultures in each cohort was not different (62.8 vs 61.7%, p = 0.76). Physicians were more likely to prescribe an antibiotic and order a urine culture in the validation cohort (p < 0.01).
The prevalence of positive urine cultures according to decision aid criteria and management recommendations
1. Number of Cystitis Decision Aid Criteria*
2. Cystitis Decision Management Categories*
No antibiotic, culture (0,1 criteria)
Antibiotic (2,3 criteria)
Cystitis decision aid performance in the validation and development cohorts (Table 3)
Comparison of acute cystitis decision aid performance in the development  and validation cohorts of women with acute cystitis
Development Cohort  (2002)
Validation Cohort (2009–2011)
Number of Family Physicians
Women with complete data
Prevalence of positive cultures
Comparison of cystitis decision aid recommendations with observed physician care (Table 4)
Comparison of physician management and acute cystitis decision aid recommendations management: validation cohort of women with acute cystitis, 2009–11
Cystitis Decision Aid
Absolute Difference (MD - Aid)
The decision aid accuracy and impact on antibiotic use compared to physician care in both cohorts is shown in Additional file 1: Table S2. The reduction in decision aid specificity in the two cohorts was also seen with usual physician management (22.1% development cohort vs 6.6% validation study (p < 0.001), and was lower in both cohorts for physician care than decision aid recommendations (53.3 and 35.5% respectively). Urine culture testing would also be reduced compared to physician care (p < 0.0001). Empirical antibiotics would have been reduced with decision aid recommendations by an absolute 20.8% (development cohort) and 20.4% (validation cohort) compared to physician management, and unnecessary antibiotics by 11.4 and 11.1%, respectively.
A clinical decision aid to inform empirical decisions about antibiotic treatment for women with suspected acute cystitis demonstrated reproducible accuracy and potential reductions in total and unnecessary antibiotic prescribing, and urine culture testing, in two national cohorts of women compared to observed care by family physicians.
The sensitivity of the decision aid for identifying women with culture positive acute cystitis for empirical antibiotic treatment was the same in two national, but different cohorts of women with cystitis symptoms, assessed 4 years apart. While family physicians were more sensitive with their prescribing decision in identifying women likely to have a positive urine culture, they demonstrated little specificity, resulting in almost all women receiving antibiotics. The specificity of the decision aid varied in the two cohorts, but this was also true for physicians, and the decision aid specificity was significantly higher than physician specificity in both cohorts. The superior specificity of the decision aid compared to usual empirical prescribing decisions translates into clinically important reductions in both total and unnecessary antibiotic prescriptions that would be realized in these women with typical cystitis presentations.
The impact of the cystitis decision aid on unnecessary antibiotic prescriptions was a clinically important 11% absolute reduction, and a 31% relative reduction. As physicians prescribe antibiotics empirically to most women with suspected cystitis, this corresponds to large reductions in total antibiotics and unnecessary antibiotic prescriptions. In a study from Wales, clinics that achieved large reductions in antibiotic prescribing over 7 years were able to reduce ampicillin and trimethoprim resistance in coliform urinary tract isolates . Secondary benefits from the decision aid strategy are potential reductions in urine culture testing.
One question is whether women would be willing to wait for culture results before receiving antibiotics. This would occur in 24% of women assessed using the cystitis decision aid. A Dutch study found more than a third of women with cystitis were willing to delay antibiotic treatment . In another randomized trial, no difference in symptom severity or duration was found with delayed antibiotics for cystitis compared to immediate treatment . Ibuprofen has also been demonstrated to reduce symptom discomfort similar to antibiotics . This could provide an practical alternative for women advised to wait for culture results.
A limitation of the current study was a large number of culture results that were not able to be obtained. In the development cohort,  85% of culture results were obtained compared to 53% in the current study. This was due to a change in the method for obtaining consent in the validation study that relied on physicians giving women a consent form. Despite this, the prevalence of positive cultures in the two studies was the same. This is important as decision aid accuracy can be affected when disease prevalence varies in different populations . As the prevalence of positive urine cultures and decision aid sensitivity in these two national cohorts of women with cystitis symptoms was the same, despite missing culture results, this suggests the decision aid is likely to apply in usual cystitis cases seen in primary care office settings. However, additional validation of the decision aid may be warranted given the missing cultures in this study. Validation in emergency departments or walk-in clinics would also be necessary to ensure the prevalence of culture-positive cystitis is similar, prior to use of the decision aid in those settings. Similarly, additional study of the performance of decision aid in developing countries would be needed prior to clinical use.
A second limitation was the recruitment of only 2% of physicians approached. Nonetheless, there were more than 230 different physicians involved from cities and towns across the entire country. In addition, the clinical characteristics of the women they assessed were typical of cases of acute cystitis that any family physician would see. Further, the similarity of the current results with those in the development study of a different cohort of women and family physicians provides additional reassurance that the decision aid is likely applicable in community family practice settings. The use of mid-stream urine samples as a reference standard could also be criticized. One study reported a poor correlation of mid-stream samples with catheter specimens, primarily for gram positive organisms . However, for E.coli, which causes 75–95% of acute cystitis [1, 12] the positive predictive value was greater than 90% .
Different studies have used different microbiologic definitions for a positive culture. We chose a definition of ≥102 cfu/ml (≥105 CFU/L) as per North American standards . Others have followed European microbiologic standards (> 103 cfu/ml) [19, 20]. Nonetheless, the prevalence of culture positive UTI in the latter primary care studies (61% , 66% ) were comparable to the current study (62%). This may be because most women with established cystitis symptoms will have colony counts above both the ≥102 and 103 thresholds, resulting in little impact on the final prevalence of positive cultures. However, further assessments of the performance of the decision aid with different microbiologic definitions is advisable. One study reported some women with negative urine cultures improved when treated with antibiotics . However, the definition of a positive culture in that study was a less sensitive microbiologic threshold of ≥105 cfu/ml. It is possible some women deemed culture negative in that study may have been found to be culture positive using the lower 102cfu/ml threshold used in the current study.
Other clinical decision aids have been proposed for acute cystitis [19, 20, 22]. Some involve complex scoring [19, 22], or have not undergone external validation in a new population to ensure reproducibility [19, 22]. One study compared a clinical symptom score and a urine dipstick score separately , whereas physicians utilize both clinical symptoms and urine dip stick results in deciding about a course of action. Symptoms combined with dipstick tests have been demonstrated to have better diagnostic accuracy than symptoms alone . The study by Little et al. concluded that clinical and urine based scores had poor predictive accuracy, but did not compare their performance with that of usual physician care for diagnosis or antibiotic prescribing . The current and previous studies [7, 8] found that cystitis decision aids have superior diagnostic performance compared to physicians, and the potential for clinically relevant reductions in unnecessary antibiotic use. Empirical antibiotic treatment by telephone without an office visit or urine culture has also been suggested as an appropriate strategy . However, the impact of telephone management on antibiotic use in acute cystitis was not considered . In a study of cystitis where symptoms and cultures were collected, 40% of women prescribed empirical antibiotics based on symptoms that could be assessed by telephone were associated with a negative urine cuture .
This study provided external validation of a simple 3-item clinical decision aid for empirical antibiotic prescribing decisions in a new cohort of women with symptoms of acute cystitis, compared to a previous cohort used to develop the aid . In addition, the potential of the aid to reduce antibiotic prescriptions and urine culture testing compared to usual care by family physicians was demonstrated. Given the frequency of cystitis and volume of antibiotics prescribed to manage this common condition, a randomized trial of the cystitis decision aid in practice seems warranted to further assess if this strategy can reduce unnecessary antibiotic utilization in primary care settings.
This study was funded by the Canadian Institutes of Health Research, Grant no.MOP-86494.
Availability of data and materials
The datasets used and/or analysed in the current study are available from the corresponding author on reasonable request.
WM,TM, and RM conceived the study, participated in the design and data collection, and WM, TM, RM and IG analysed the data, and revised the manuscript. All authors approved the final manuscript.
Ethics approval and consent to participate
The Research Ethics Board of the Mount Sinai Hospital (Toronto, Ontario) approved the study. (no 08-0280-E) Written informed consent was obtained from all participating family physicians and women.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Gupta K, Hooton TM, Naber KG, Wullt B, Colgan R, Miller LG, et al. International clinical practice guidelines for the treatment of acute uncomplicated cystitis and pyelonephritis in women: A 2010 update by Infectious Diseases Society of America and the European Society for Microbiology and Infectious Diseases. Clin Infect Dis. 2011;52(5):e103–20.View ArticlePubMedGoogle Scholar
- Gupta K, Scholes D, Stamm WE. Increasing prevalence of antimicrobial resistance among uropathogens causing acute uncomplicated cystitis in women. JAMA. 1999;281:736–8.View ArticlePubMedGoogle Scholar
- Goossens H, Ferech M, Vander Stichele R, Elseviers M, for the ESAC Project Group. Outpatient antibiotic use in Europe and its association with resistance: a cross-national database study. Lancet. 2005;365:579–87.View ArticlePubMedGoogle Scholar
- Costelloe C, Metcalfe C, Lovering A, Mant D, Hay AD. Effect of antibiotic prescribing in primary care on antimicrobial resistance in individual patients: systematic review and meta-analysis. BMJ. 2010;340:c2906.View ArticleGoogle Scholar
- Butler CC, Dunstan F, Heginbothom M, Mason B, Roberts Z, Hillier S, Howe R, Palmer S, Howard A. Containing antibiotic resistance: decreased antibiotic-resistant coliform urinary tract infections with reduction in antibiotic prescribing by general practices. Br J Gen Pract. 2007;57:785–92.PubMedPubMed CentralGoogle Scholar
- Grigoryan L, Trautner BW, Gupta K. Diagnosis and management of urinary tract infections in the outpatient setting. A review. JAMA. 2014;312(16):1677–84.View ArticlePubMedGoogle Scholar
- McIsaac WJ, Low DE, Biringer A, Pimlott N, Evans M, Glazier R. The impact of empirical management of acute cystitis on unnecessary antibiotic use. Arch Intern Med. 2002;162:600–5.View ArticlePubMedGoogle Scholar
- McIsaac WJ, Moineddin R, Ross S. Validation of a decision aid to assist physicians in reducing unnecessary antibiotic drug use for acute cystitis. Arch Intern Med. 2007;167(20):2201–6.View ArticlePubMedGoogle Scholar
- Adams ST, Leveson SH. Research Methods and Reporting. Clinical prediction rules BMJ. 2012;344:d8312. 10.1136/bmj.d8312.View ArticlePubMedGoogle Scholar
- Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and elaboration. Ann Intern Med. 2015;162:W1–W73. 10.7326/M14-0698.View ArticlePubMedGoogle Scholar
- Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375:454–63.View ArticlePubMedGoogle Scholar
- Warren JW, Abrutyn E, Hebel JR, Johnson JR, Schaeffer AJ, Stamm WE. Guidelines from the Infectious Diseases Society of America. Guidelines for antimicrobial treatment of uncomplicated acute bacterial cystitis and acute pyelonephritis in women. CID. 1999;29:745–58.View ArticleGoogle Scholar
- McIsaac WJ, McIsaac WJ, Moineddin R, Meaney C, Mazzulli T. Antibiotic-resistant Escherichia coli in women with acute cystitis in Canada. Can J Infect Dis Med Microbiol. 2013;24(3):143–9.PubMedPubMed CentralGoogle Scholar
- Knottnerus BJ, Geerlings SE, Moll van Charante EP, ter Riet G. Women with symptoms of uncomplicated urinary tract infection are often willing to delay antibiotic treatment: a prospective cohort study. BMC Fam Pract. 2013;14:71.View ArticlePubMedPubMed CentralGoogle Scholar
- Little P, Moore MV, Turner S, Rumsby K, Warner G, Lowes JA, et al. Effectiveness of five different approaches in management of urinary tract infection: randomized controlled trial. BMJ. 2010;340:c199. 10.1136/bmj.c199.View ArticlePubMedPubMed CentralGoogle Scholar
- Gágyor I, Bleidorn J, Kochen MM, Wegscheider K, Hummers-Pradier E. Ibuprofen versus fosfomycin for uncomplicated urinary tract infection in women: randomized controlled trial. BMJ. 2015;351:h6544.View ArticlePubMedPubMed CentralGoogle Scholar
- Poses RM, Cebul RD, Collins M, Fager SS. The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis. Ann Intern Med. 1986;105:586–91.View ArticlePubMedGoogle Scholar
- Hooton TM, Roberts PL, Cox ME, Stapleton AE. Voided midstream urine culture and acute cystitis in premenopausal women. NEJM. 2013;369:1883–91.View ArticlePubMedPubMed CentralGoogle Scholar
- Knottnerus BJ, Geerlings SE, Moll van Charante EP, ter Riet G. Toward a simple diagnostic index for acute uncomplicated urinary tract infections. Ann Fam Med. 2013;11(5):442–51.View ArticlePubMedPubMed CentralGoogle Scholar
- Little P, Turner S, Rumsby K, Jones R, Warner G, Moore M, Lowes JA, Smith H, Hawke C, Leydon G, Mullee M. Validating the prediction of lower urinary infection in primary care: sensitivity and specificity of urinary dipsticks and clinical scores in women. Br J Gen Pract. 2010;60:495–500.View ArticlePubMedPubMed CentralGoogle Scholar
- Richards D, Toop L, Chambers S, Fletcher L. Response to antibiotics of women with symptoms of urinary tract infection but negative dipstick urine test results: double blind randomized controlled trial. BMJ. 10.1136/bmj.38496.452581.8F.
- Dobbs FF, Fleming DM. A simple scoring system for evaluating symptoms, history and urine dipstick testing in the diagnosis of urinary tract infection. J Roy Coll Gen Pract. 1987;37:100–4.Google Scholar
- Giesen LGM, Cousins G, Dimitrov BD, van de Laar F, Fahey T. Predicting acute uncomplicated urinary tract infection in women: a systematic review of the diagnostic accuracy of symptoms and signs. BMC Fam Pract. 2010;11:78.View ArticlePubMedPubMed CentralGoogle Scholar