- Research article
- Open Access
- Open Peer Review
Assessing quality improvement capacity in primary care practices
BMC Family Practice volume 20, Article number: 103 (2019)
Healthy Hearts Northwest (H2N) is a study of external support strategies to build quality improvement (QI) capacity in primary care with a focus on cardiovascular risk factors: appropriate aspirin use, blood pressure control, and tobacco screening/cessation.
To guide practice facilitator support, experts in practice transformation identified seven domains of QI capacity and mapped items from a previously validated medical home assessment tool to them. A practice facilitator (PF) met with clinicians and staff in each practice to discuss each item on the Quality Improvement Capacity Assessment (QICA) resulting in a practice-level response to each item. We examined the association between the QICA total and sub-scale scores, practice characteristics, a measure of prior experience with managing practice change, and performance on clinical quality measures (CQMs) for the three cardiovascular risk factors.
The QICA score was associated with prior experience managing change and moderately associated with two of the three CQMs: aspirin use (r = 0.16, p = 0.049) and blood pressure control (r = 0.18, p = 0.013). Rural practices and those with 2–5 clinicians had lower QICA scores..
The QICA is useful for assessing QI capacity within a practice and may serve as a guide for both facilitators and primary care practices in efforts to build this capacity and improve measures of clinical quality.
This trial is registered with www.clinicaltrials.gov Identifier# NCT02839382, retrospectively registered on July 21, 2016.
Although primary care practices are under increased pressure to improve care quality , little attention has been devoted to assessing and building the inherent ability of primary care practices to effectively conduct quality improvement activities [1,2,3]. This “QI capacity” has been defined as “…the ability of a primary care practice to engage in QI activities in a continuous and effective way.”  QI capacity requires that teams of individuals who work in primary care settings possess an in-depth knowledge of approaches to QI, have the requisite skills and commitment to apply that knowledge to clinical care improvement, and are supported by clinical information systems in that effort [5, 6]. A focus on building QI capacity/capability is one of the key characteristics of high performing health care organizations [7,8,9].
Smaller primary care practices comprise over half of all primary care practices in the U.S.  These smaller practices often struggle with improving their capacity to improve care quality . Health care professionals in these settings are often not prepared to promote QI efforts or lead change activities [12, 13]. They also often lack the health IT infrastructure, training, resources and time required to build their capacity to do QI well [11, 14,15,16,17]. Given the shortage of primary care providers, clinicians in these practices often have large patient panels requiring a focus on day-to-day patient care activities rather than continually improving the way care is delivered. These factors have led to calls for external support infrastructure to build QI capacity in primary care practice settings [18,19,20].
In 2015, the Agency for Healthcare Research & Quality (AHRQ) launched the EvidenceNOW initiative across seven regional cooperatives in the United States [21,22,23] to expand our knowledge about how to effectively assist smaller primary care practices in building their QI capacity with a focus on improving cardiovascular disease (CVD) risk factors . For each of these cooperatives AHRQ required the use of practice facilitation as a unifying strategy [18, 24] to provide the necessary external support. Healthy Hearts Northwest is the EvidenceNOW cooperative across Washington, Oregon and Idaho [25, 26]. To guide practices and PFs in this study, we needed to define the specific elements and behaviors that reflect the presence of QI capacity specific to improving clinical care quality in a practice. Such a self-assessment tool should provide important insights to PFs about the current state of QI capacity in the practice and inform both the practice and the PF about opportunities to build QI capacity. Without such a ‘roadmap’ to serve as a guide, practice PFs have noted challenges when supporting improvement in primary care practices .
Is such a self-assessment tool needed? A recent systematic review of approaches to evaluate QI capacity found substantial variation in approaches to measure or observe QI capacity and no organizational-level QI capacity building evaluations were found . Specific to primary care, some tools and instruments exist that assess organizational characteristics more broadly associated with quality improvement [28,29,30,31,32] such as overall organizational maturity or development , organizational culture [34,35,36], or readiness for change [37, 38]. However, the concept of QI capacity is distinct from organizational culture or readiness . QI capacity is reflected in the presence of capabilities and competencies possessed by a primary care team that directly impinge on the delivery of clinical care to a patient. It is an emergent property of the underlying knowledge, skills, culture and experience of those who work in a practice . For example, the Change Process Capacity Questionnaire (CPCQ) was developed to measure organizational capability to manage change within a primary care practice setting . It identifies a set of processes used to manage change (e.g. using opinion leaders) that might be useful in quality improvement initiatives. It was not explicitly developed to assess the presence of specific structural capabilities required to improve clinical performance on clinical quality measures, and a recent publication found no association between the CPCQ score and practice performance on clinical quality measures as would be expected from an assessment of QI capacity .
In response to this need, we identify and define a set of domains that would inform the selection of key activities practices could undertake to improve their QI capacity . We map those domains to survey items and introduce the twenty-item Quality Improvement Capacity Assessment (QICA) tool used to guide PFs and practices in assessing and building their QI capacity.
Study setting and subjects
Healthy Hearts Northwest (H2N) took place within 209 smaller (10 or fewer clinicians) primary care practices across Washington State, Oregon and Idaho and the results of this randomized trial have been previously reported . All practices received support from a practice facilitator (PF) for 15 months. In addition to PF, practices were randomized to receive educational outreach visits (based on principle of academic detailing), shared learning opportunities or both. The published trial results adhered to CONSORT guidelines for reporting clinical trials.
Development of a QI capacity assessment instrument
To identify and define core concepts or “domains” that contribute to QI capacity two members of the study team (KC, LM) along with two outside experts with considerable prior experience in patient centered medical home transformation efforts was convened. They first reviewed key articles in the literature related to the definitions and characteristics of QI capacity [5, 44], principles of transformation to a patient-centered medical home [45,46,47,48,49], and elements of the chronic care model [50, 51] . .After reviewing the literature and discussing their prior medical home transformation experience, the experts used a nominal group process to nominate and build consensus about the domains . A key consideration in selecting a domain was their assessment that a practice could engage in activities with support from a practice facilitator that would improve that domain within the planned 15-months of practice support. In addition, because the study was focused in improving CVD risk factors with support from a PF, there was a focus on domains that were felt to directly impact QI capacity in a manner that would improve performance on clinical quality measures of these risk factors . These domains were:
Embed clinical evidence into daily work to guide how care is delivered to patients
Utilize data to understand and improve clinical performance measures
Establish a regular QI process involving cross-functional teams
Identify at-risk patients through pro-active population management for outreach
Define roles and responsibilities across the team to improve care
Deepen patient self-management support to improve clinical outcomes
Link patients to resources outside of the clinic to support patients
To assist both practices and their PF in assessing their current state of QI capacity and guiding efforts to improve, a self-assessment survey was developed by identifying items relevant to each of the seven change concepts. The Safety Net Medical Home Initiative had previously validated the 36-item Patient-Centered Medical Home Assessment (PCMH-A) tool for medical home transformation . Since this instrument was intended to measure areas where practice change would result in improving care delivery, and because it was used by practice facilitators to support practice change, it was identified as a promising source of items. Study team members (MLP, KC) reviewed the 36 items in the PCMH-A independently, and agreed that 19 aligned well with the seven change concepts described above. A twentieth item was created by study team members (MLP, KC) to assess the availability of a standard method or tool to stratify patients by level of clinical risk. This new item was included with three other items to help assess the fourth domain listed above, identification of at-risk patients. Consistent with the PCMH-A instrument , and with the more widely used and validated Assessment of Chronic Illness Care (ACIC) instrument [27, 55], each item on the QICA is rated on a 1 to 12 scale with higher scores indicating higher QI capacity (see Additional file 1). The 20 items were grouped into seven subscales corresponding to the change concepts identified above.
Measures and data collection
During a face-to-face visit by the PF at the start of the study intervention, the QICA survey was completed individually by clinicians and staff in each H2N practice. With the PF present, team members discussed their response to each item and came to a consensus on a practice-level response. These meetings took place between December 2015 and July of 2016. Prior to this meeting, practice managers completed a survey describing practice characteristics, which included the Change Process Capacity Questionnaire (CPCQ) . In addition, each practice was asked to submit clinical quality measures (CQMs) data derived from the electronic health record on three CVD risk factors in the year prior to completing the QICA: appropriate aspirin use, blood pressure control, and tobacco screening/cessation [56, 57]. After the in-person visit to facilitate completion of the QICA, PFs kept detailed field notes describing the encounter.
We first described practice characteristics, and summarized baseline QICA scores and clinic performance on each CQM. We then used Kruskal-Wallis tests to assess the association between total QICA score and practice characteristics including practice size, ownership and location (rural v. urban). To examine the reliability of the QICA, we estimated Cronbach’s alpha for the total scale and each subscale. We assessed scale validity by reporting the Spearman correlation between the total score and subscale scores, with the CPCQ total score, and practice performance on each of the three CQMs in the year prior to administering the QICA survey.
The QICA was completed by 202 of the 209 enrolled practices. Of these, 178 (88.1%) provided data on the CPCQ and 192 (95.0%) reported on one or more of the CVD risk factor CQMs. Practice characteristics, QICA scores and CQMs are shown in Table 1. Of the seven change domains, practices scored the lowest on utilizing data, having a regular QI process, and identifying at-risk patients. Practices in rural locations, and those with 2–5 clinicians compared to solo or larger group practices had lower total QICA scores (Table 2).
The Cronbach’s alpha was 0.90 for the total score and ranged from 0.62 to 0.87 for the domain subscales. (Table 3) All QICA sub-scores, as well as the total QICA score were significantly related to the practice’s CPCQ score. The QICA total score was associated with two of the three clinical quality measures: the proportion with adequate blood pressure control (r = 0.18, p = 0.013) and the proportion with appropriate aspirin use (r = 0.16, p = 0.049). Five of the seven QICA domain sub-scales were associated with at least one of the three CQMs.
These results suggest that the QICA is associated both with a practices’ prior experience with specific change strategies (as measured by the CPCQ) and their current level of clinical performance as assessed by two of the three CVD CQMs collected for this study. These findings provide partial support for the “criterion validity” of the QICA, the degree to which a measure is associated with an outcome. Overall, practices scored near the middle of the range of possible scores on the total QICA score, 6.55 (S.D. 1.49), out of a possible range of 1 to 12, with no evidence of either floor or ceiling effects. Scores on utilizing data to improve, establishing a regular QI process, and identifying at-risk patients for outreach were lower than other QI capacity domains. These three domains represent what some consider to be core QI competencies , reflect significant opportunities for development of their QI capacity, and support prior observations that smaller primary care practices struggle with building their QI capacity ..
The total QICA score was moderately associated with two of the three CQMs. In addition to QI capacity, it is possible that other inner setting characteristics of the clinic, its patient population, and other external/contextual factors play an important role in predicting performance on CQMs . For example, some practices may have had external incentives to improve performance, financial or otherwise, that interacted with their QI capacity to predict their performance. QI capacity is also dependent on the interrelationships between QI training; individual capability and capacity; opportunities to apply QI skills; exposure to further coaching/support in the workplace; a supportive organizational culture and leadership; and access to resources . Cohen and colleagues highlighted the role of a complex interaction of internal and external factors on implementing improvement strategies in primary care . Those factors include motivations of practice members to improve, the resources available, external influences on improvement options, and the perspective of practice members about where opportunities for improvement exist. Ultimately, although the domains represented in the QICA reflect the internal practice setting and may be necessary for improvement to occur, they may not be sufficient to fully explain current performance on CQMs.
In addition to these limitations it is worth noting that some practices may have scored themselves higher on some items on the QICA survey because they did not fully understand what the highest state or level would look like in their setting, or their practice’s true ability in relation to this target highest level. This overestimation may have been more common in practices with less QI experience and less appreciation for the doing the work of QI. The investigators are cognizant of the possibility that practices with higher QI capacity had employed that capacity to improve other measures of care quality but had not focused improvement efforts on their performance on the cardiovascular risk factors reported in this study. For example, some practices may have done prior work on improving performance on measures of cancer screening or immunizations in response to external influences, work that improved their QI capacity, but had never engaged in work to improve the cardiovascular CQMs measured here.
The QICA is useful for assessing QI capacity within a practice and may serve as a guide for both facilitators and primary care practices in efforts to improve this capacity and improve performance on clinical quality measures.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Agency for Healthcare Research & Quality
Clinical Quality Measure
Healthy Hearts Northwest study
Quality Improvement Capacity Assessment
McGlynn EA, Adams JL, Kerr EA. The quest to improve quality: measurement is necessary but not sufficient. JAMA Intern Med. 2016;176(12):1790–1.
Hudson SM, Hiott DB, Cole J, Davis R, Egan BM, Laken MA. Increasing capacity for quality improvement in underresourced primary care settings. Qual Manag Health Care. 2014;23(3):155–62.
Solberg LI, Elward KS, Phillips WR, et al. How can primary care cross the quality chasm? Ann Fam Med. 2009;7(2):164–9.
Taylor EF, Peikes D, Genevro J, Meyers D. Creating capacity for improvement in primary care. Rockville: Agency for Healthcare Research and Quality; 2013. http://www.ahrq.gov/professionals/prevention-chronic-care/improve/capacity-building/pcmhqi1.html. Accessed July 10, 2019
Taylor EF, Genevro J, Peikes D, Gennotti WW, Meyers D. Bulding quality improvement capacity in primary care: supports and resources. Agency for Healthcare Research & Quality: Rockville; 2013. http://www.ahrq.gov/professionals/prevention-chronic-care/improve/capacity-building/pcmhqi2.html. Accessed 10 Jul 2019
Mery G, Dobrow MJ, Baker GR, Im J, Brown A. Evaluating investment in quality improvement capacity building: a systematic review. BMJ Open. 2017;7(2):e012431.
Bevan H. How can we build skills to transform the healthcare system? J Res Nurs. 2010;15(2):139–48.
Baker GR, MacIntosh-Murrary A, Porcellato C, Dionne L, Stelmacovich K, Born K. Higher performing healthcare systems: delivering quality by design. Toronto: Longwoods Publishing; 2008.
Baker GR, Denis J. A comparative study of three transformative healthcare systems: lessons for Canada. Ottawa: Canadian Foundation for Healthcare Improvement; 2011.
Liaw WR, Jetty A, Petterson SM, Peterson LE, Bazemore AW. Solo and small practices: a vital, diverse part of primary care. Ann Fam Med. 2016;14(1):8–15.
Nutting PA, Crabtree BF, McDaniel RR. Small primary care practices face four hurdles--including a physician-centric mind-set--in becoming medical homes. Health Aff (Millwood). 2012;31(11):2417–22.
Robert Wood Johnson Foundation. Spreading quality improvement: Institute for Healthcare Improvement Open School. 2011; https://www.rwjf.org/en/library/research/2011/12/spreading-quality-improvement-.html. Accessed 10 Jul 2019.
Dobrow M, Neeson J, Sullivan T. Canadian chief executive officers' prescription for higher quality: more clinical engagement, shared accountability and capacity development. Healthc Q. 2011;14(4):18–21.
Cohen DJ, Dorr DA, Knierim K, et al. Primary care practices' abilities and challenges in using electronic health record data for quality improvement. Health Aff (Millwood). 2018;37(4):635–43.
Nutting PA, Crabtree BF, Miller WL, Stange KC, Stewart E, Jaen C. Transforming physician practices to patient-centered medical homes: lessons from the national demonstration project. Health Aff (Millwood). 2011;30(3):439–45.
Rittenhouse DR, Casalino LP, Shortell SM, et al. Small and medium-size physician practices use few patient-centered medical home processes. Health Aff (Millwood). 2011;30(8):1575–84.
Wolfson D, Bernabeo E, Leas B, Sofaer S, Pawlson G, Pillittere D. Quality improvement in small office settings: an examination of successful practices. BMC Fam Pract. 2009;10:14.
Phillips RL Jr, Kaufman A, Mold JW, et al. The primary care extension program: a catalyst for change. Ann Fam Med. 2013;11(2):173–8.
Grumbach K, Bainbridge E, Bodenheimer T. Facilitating improvement in primary care: the promise of practice coaching. Issue Brief (Commonw Fund). 2012;15:1–14.
Grumbach K, Mold JW. A health care cooperative extension service: transforming primary care and community health. JAMA. 2009;301(24):2589–91.
Shoemaker SJ, McNellis RJ, DeWalt DA. The capacity of primary care for improving evidence-based care: early findings from AHRQ's EvidenceNOW. Ann Fam Med. 2018;16(Suppl 1):S2–s4.
Cohen DJ, Balasubramanian BA, Gordon L, et al. A national evaluation of a dissemination and implementation initiative to enhance primary care practice capacity and improve cardiovascular disease care: the ESCALATES study protocol. Implement Sci. 2016;11(1):86.
Meyers D, Miller T, Genevro J, et al. EvidenceNOW: balancing primary care implementation and implementation research. Ann Fam Med. 2018;16(Suppl 1):S5–S11.
Baskerville NB, Liddy C, Hogg W. Systematic review and meta-analysis of practice facilitation within primary care settings. Ann Fam Med. 2012;10(1):63–74.
Parchman ML, Fagnan LJ, Dorr DA, et al. Study protocol for “healthy Hearts northwest”: a 2 x 2 randomized factorial trial to build quality improvement capacity in primary care. Implement Sci. 2016;11(1):138.
Parchman M, Anderson M, Door D, et al. A randomized trial of external practice support to improve cardiovascular risk factors in primary care.Ann Fam Med In Press.
Parchman ML, Noel PH, Culler SD, et al. A randomized trial of practice facilitation to improve the delivery of chronic illness care in primary care: initial and sustained effects. Implement Sci. 2013;8:93.
Hardison CD. Readiness, action, and resolve for change: do health care leaders have what it takes? Qual Manag Health Care. 1998;6(2):44–51.
Lehman WE, Greener JM, Simpson DD. Assessing organizational readiness for change. J Subst Abus Treat. 2002;22(4):197–209.
Steel R, Jennings K. Measuring progress toward quality excellence. ASQC Qual Congr Trans. 1992;46:620–6.
Flynn B, Schroeder R, Sakakibara S. Determinants of quality performance in high- and low-quality plants. Qual Manag J. 1995;2:8–25.
Tamimi N, Gershon M, Currall S. Assessing the psychometric properties of Deming's 14 principles. Qual Manag J. 1995;2:38–52.
Elwyn G, Rhydderch M, Edwards A, et al. Assessing organisational development in primary medical care using a group based assessment: the maturity matrix. Qual Saf Health Care. 2004;13(4):287–94.
Deo S, McInnes K, Corbett CJ, et al. Associations between organizational characteristics and quality improvement activities of clinics participating in a quality improvement collaborative. Med Care. 2009;47(9):1026–30.
Kralewski J, Dowd BE, Kaissi A, Curoe A, Rockwood T. Measuring the culture of medical group practices. Health Care Manag Rev. 2005;30(3):184–93.
Larson P, Sinha A. The TQM impact: a study of quality managers' perceptions. Qual Manag J. 1995;2:53–66.
Ingersoll GL, Kirsch JC, Merk SE, Lightfoot J. Relationship of organizational culture and readiness for change to employee commitment to the organization. J Nurs Adm. 2000;30(1):11–20.
Weeks DL, Polello JM, Hansen DT, Keeney BJ, Conrad DA. Measuring primary care organizational capacity for diabetes care coordination: the diabetes care coordination readiness assessment. J Gen Intern Med. 2014;29(1):98–103.
Flaspohler P, Duffy J, Wandersman A, Stillman L, Maras MA. Unpacking prevention capacity: an intersection of research-to-practice models and community-centered models. Am J Community Psychol. 2008;41(3–4):182–96.
Geonnotti K, Taylor EF, Peikes D, et al. Engaging primary care practices in quality improvement: strategies for practice facilitators. AHRQ Publication No. 15–0015-EF. Rockville: Agency for Healthcare and Research; 2015.
Solberg LI, Asche SE, Margolis KL, Whitebird RR. Measuring an organization's ability to manage change: the change process capability questionnaire and its use for improving depression care. Am J Med Qual. 2008;23(3):193–200.
Shelley D, Blechter B, Siman N, et al. Quality of cardiovascular disease care in small urban practices. Ann Fam Med. 2018;16(Suppl 1):S21–s28.
Langley G, Nolan K, Nolan C, Provost L. The improvement guide: a practical approach to enhancing organizational performance. 2nd ed. San Francisco: Jossey-Bass; 2009.
Michaels L, Anastas T, Waddell EN, Fagnan L, Dorr DA. A randomized trial of high-value change using practice facilitation. J Am Board Fam Med. 2017;30(5):572–82.
Ferrante JM, Balasubramanian BA, Hudson SV, Crabtree BF. Principles of the patient-centered medical home and preventive services delivery. Ann Fam Med. 2010;8(2):108–16.
O'Malley A, Peikes D, Ginsburg P. Making medical homes work: moving from concept to Practice. Policy Perspective: Insights into Health Policy Issues, vol. Vol 1. Washington: Center for Studying Health System Change; 2008. p. 1–19.
Bodenheimer T, Ghorob A, Willard-Grace R, Grumbach K. The 10 building blocks of high-performing primary care. Ann Fam Med. 2014;12(2):166–71.
Stange KC, Nutting PA, Miller WL, et al. Defining and measuring the patient-centered medical home. J Gen Intern Med. 2010;25(6):601–12.
Simon M, Choudhry NK, Frankfort J, et al. Exploring attributes of high-value primary care. Ann Fam Med. 2017;15(6):529–34.
Wagner EH. Chronic disease management: what will it take to improve care for chronic illness? Eff Clin Pract. 1998;1(1):2–4.
Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness: the chronic care model, part 2. JAMA. 2002;288(15):1909–14.
Wagner EH, Coleman K, Reid RJ, Phillips K, Abrams MK, Sugarman JR. The changes involved in patient-centered medical home transformation. Prim Care. 2012;39(2):241–59.
Jones J, Hunter D. Consensus methods for medical and health services research. BMJ (Clinical research ed). 1995;311(7001):376–80.
Daniel DM, Wagner EH, Coleman K, et al. Assessing progress toward becoming a patient-centered medical home: an assessment tool for practice transformation. Health Serv Res. 2013;48(6 Pt 1):1879–97.
Bonomi AE, Wagner EH, Glasgow RE, VonKorff M. Assessment of chronic illness care (ACIC): a practical tool to measure quality improvement. Health Serv Res. 2002;37(3):791–820.
Million Hearts. Clinical quality measures. https://millionhearts.hhs.gov/files/mh_cqm.pdf. Accessed 19 Apr 2019.
National Quality Forum (NQF). Measures, reports & tools. http://www.qualityforum.org/Measures_Reports_Tools.aspx. Accessed 19 Apr 2019.
Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington: National Academies Press (US); 2001.
Kaplan HC, Brady PW, Dritz MC, et al. The influence of context on quality improvement success in health care: a systematic review of the literature. Milbank Q. 2010;88(4):500–59.
Cohen D, McDaniel RR Jr, Crabtree BF, et al. A practice change model for quality improvement in primary care practice. J Healthc Manag. 2004;49(3):155–68 discussion 169-170.
The authors would like to express appreciation to Kathryn Phillips, Beth Sommers and Nicole Van Borkulo for their role in developing the domains of QI capacity as well as the staff and clinicians in participating primary care practices across Washington, Oregon and Idaho.
This project was supported by grant number R18HS023908 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
Ethics approval and consent to participate
This study was determined to be exempt (category 2) by the Kaiser Permanente Washington Health Research Institute’s Institutional Review Board, waiving the requirement for informed consent but not ethics review.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.