Skip to main content

Learning health systems in primary care: a systematic scoping review

Abstract

Background

Learning health systems have been gaining traction over the past decade. The purpose of this study was to understand the spread of learning health systems in primary care, including where they have been implemented, how they are operating, and potential challenges and solutions.

Methods

We completed a scoping review by systematically searching OVID Medline®, Embase®, IEEE Xplore®, and reviewing specific journals from 2007 to 2020. We also completed a Google search to identify gray literature.

Results

We reviewed 1924 articles through our database search and 51 articles from other sources, from which we identified 21 unique learning health systems based on 62 data sources. Only one of these learning health systems was implemented exclusively in a primary care setting, where all others were integrated health systems or networks that also included other care settings. Eighteen of the 21 were in the United States. Examples of how these learning health systems were being used included real-time clinical surveillance, quality improvement initiatives, pragmatic trials at the point of care, and decision support. Many challenges and potential solutions were identified regarding data, sustainability, promoting a learning culture, prioritization processes, involvement of community, and balancing quality improvement versus research.

Conclusions

We identified 21 learning health systems, which all appear at an early stage of development, and only one was primary care only. We summarized and provided examples of integrated health systems and data networks that can be considered early models in the growing global movement to advance learning health systems in primary care.

Peer Review reports

Background

Health system improvement in primary care is generally slow, partly because of the dependence on passive knowledge dissemination, but also because of the lack of a systematic approach to identify gaps between evidence and practice, and implementation of interventions to close these gaps [1, 2]. The spread of electronic health records (EHR) in primary care across North America in the past decade [3,4,5], and the ability to rapidly analyze these data and act on their findings, provides a strategy to potentially close the loop between practice gaps and implementation strategies. In 2007, the Institute of Medicine (IOM) first described the concept of a learning health system (LHS) [6], which is an organization or a network with a culture of health system improvement where internal data is integrated with existing evidence and rapidly analyzed, this knowledge is put into practice, and its effectiveness at closing practice gaps is evaluated [7].

Budrionis and Bellika (2016) completed the most recent systematic review on LHSs and found 13 papers that described implementation of an LHS [8]. They categorized these into clinical data reuse (nine), patient-reported outcomes (three), and collaborative learning (one). Most of these LHSs were in the hospital setting, and it’s not clear how many also included primary care [8]. There are no systematic or scoping reviews summarizing LHSs in primary care.

Primary care is a patient’s first point of contact with the healthcare system, and typically providers focus on primary and secondary prevention of chronic diseases and acute episodes. Many primary care providers now use EHRs [3,4,5], which can be used to track illness and healthcare patterns, provide decision support, and for quality improvement initiatives. On the other hand, many primary care providers work in small independent community practices and use one of various EHR vendors, making it difficult to link data across different practices, especially in Canada [9]. Therefore, the primary care setting is both an opportune yet challenging setting to develop an LHS.

The purpose of this scoping review was to understand the extent of LHSs’ spread in primary care and their characteristics. We decided to conduct a scoping review to capture a broad overview on this topic. Our research objectives were to understand: 1) where LHSs in primary care settings have been implemented or are being planned, 2) how these LHSs are operating, and 3) the challenges and solutions to implementing or sustaining an LHS in primary care.

Methods

We completed a scoping review following guidelines outlined by the Joanna Briggs Institute and the PRISMA extension for scoping reviews (Additional file 1) [10, 11]. We do not have a published protocol for this study.

Eligibility criteria

We included data sources that described the implementation or plans for developing an LHS within a primary care setting. We defined an LHS as an organization or a network, which rapidly analyzes health data while incorporating best-practice guidelines to directly feedback to and improve clinical care (either through clinical research, quality improvement or decision support tools). In this case, clinical research included comparative effectiveness analyses, prediction modelling, pragmatic trials, or other big data analytics where the findings could be readily applied in practice. We excluded research networks where only conventional knowledge translation through academic publications was occurring without more direct feedback of the knowledge to improve care. We included integrated health systems or networks that also had inpatient care, or care in other settings, if they included primary care in their health system or network. We did not include pediatric LHSs, since this is considered specialty care in some countries including Canada. We only included studies published after the IOM report in 2007 [6], and we restricted the search to English language articles only. Finally, we did not include sources describing only specific technical components of an LHS, such as EHRs, software, hardware or interoperability.

Information sources

We searched OVID Medline®, Embase® and IEEE Xplore® from January 2007 to January 2020. We originally searched databases on March 24, 2019 and updated searches on January 2, 2020. We also reviewed all papers published in the following journals: Leaning Health Systems, eGEMs, and BMJ Quality and Safety. We then reviewed all references from Budrionis and Bellika (2016) and other systematic reviews that were found from our database search [8]. We also used Google Scholar to find articles that cited the IOM’s (2007) LHS report [6]. To capture gray literature we also performed a Google search in August 2019 for ‘learning health systems’. To confirm whether an LHS was eligible for inclusion in our review, we reached out to some authors or website contacts.

Search strategy

We consulted a librarian at Western University on the search strategy (Additional file 2). For Medline® and Embase® we used keywords for different variations of LHSs including ‘health’ versus ‘health care’. We also included Medical Subject Headings (MeSH) terms for ‘learning’ and ‘health care delivery’. We used search terms for primary care in the IEEE Xplore® database search but not for the other databases, since this narrowed the articles retrieved and did not capture integrated health systems. After all eligible data sources were found we completed a targeted search using Pubmed and Google for other sources of information further describing these LHSs. This was to ensure completeness for our data charting. For example, an article found through our search may have discussed the LHS in detail but lacked information on the organization or health system in which it was based.

Selection and data charting processes

All articles from the database search were uploaded to Covidence [12]. DMN screened all article titles and abstracts through Covidence and selected ones eligible for full-text review, and then completed the full-text review. ZB further reviewed full-text articles for eligibility, and both authors discussed discrepancies. DMN created the data charting form and tested it on five articles to ensure all information would be captured. ZB charted the data from the eligible articles and websites, and DMN reviewed and confirmed all information entered in the spreadsheet against the original data source. The charted data elements are described in Additional file 3. We summarized data in table format (overview of LHSs, and challenges and solutions) and included a description of each learning health system with examples of primary care specific learning initiatives in the text. In some cases, we reached out to authors or website contacts to further confirm details.

Results

Selection of sources of evidence

Through our database search we retrieved 1924 unique articles plus an additional 51 from other sources. The number of articles excluded from the database screening, and the reasons for exclusion are shown in Fig. 1. Confirmation of eligibility by reaching out to authors confirmed inclusion for one data source and confirmed exclusion for two. We only searched up to page 8 in Google for our internet search, since the results were not relevant after this.

Fig. 1
figure 1

The number of data sources included and excluded in review, and reasons for exclusions. *Ineligible sources refer to data networks and organizations that did not meet our definition of a learning health system but were included as an additional file of those with potential to become one

Characteristics of sources of evidence

We identified 21 LHSs including primary care that met our definition, from 40 peer-reviewed research articles and 22 websites (Table 1). Most (18/21) self-identified as an LHS, where three did not: Health Care Systems Research Network (HCSRN), High Value Healthcare Collaborative (HVHC), and Intermountain Healthcare [13,14,15].

Table 1 Overview of learning health systems included in the review

Translational medicine and patient safety (TRANSFoRm) was the only LHS included in our review that operated only in primary care [61]. The remaining LHSs are vertically integrated health systems or networks that deliver care through hospitals, universities and primary care. Although both are integrated health systems, the papers describing Johns Hopkins’ and University of Wisconsin’s LHSs focus on primary care. Three of the LHSs identified were in Europe [53, 59, 61], where the remaining were in the United States.

All but one of the included LHSs appear to be financially sustainable, with the learning activities from most organizations funded through care delivery, rather than external, project-based or time-limited funding. The exception is TRANSFoRm, which is a time-limited research project dependent on external funding [18, 61]. However, even internally-funded LHSs, such as Geisinger Health System, raised concerns about financial sustainability of their learning activities [75].

We also identified 16 additional data networks / organizations (based on 14 peer-reviewed research articles and 16 websites) that included primary care data, but did not meet our definition of a fully-functioning LHS, since they were mostly platforms for conventional research and knowledge translation, rather than building internal systems for directly identifying gaps and cycling knowledge to improve care (Additional file 4). The majority of these (12/16) were projects funded through the National Patient-Centered Clinical Research Network (PCORnet), with the initial aim to conduct large-scale comparative effectiveness research [76]. We included all 16 in an additional file as “potential LHSs”, rather than excluding them from our review, since many are working towards an LHS by establishing a data sharing process. However, our review and search strategy targeted LHSs, so the identified research / data networks and organizations may not be comprehensive.

Integrated health systems

We identified 13 integrated LHSs in our review, all of which are in the United States.

Geisinger Health System – a large not-for-profit integrated health system in Central Pennsylvania and Southern New Jersey encompassing eight hospital campuses – was an early adopter of LHS approaches [19]. In Psek (2015), they reported a framework for nine components of an LHS and outlined how Geisinger meets these components including data and analytics, people and partnerships, patients and family engagement, ethics and oversight, evaluation and methodology, funding, organization, prioritization, and deliverables [20]. We did not find any documented primary care-specific LHS initiatives for Geisinger.

Kaiser Permanente is another large not-for-profit integrated health system serving members across eight states and the District of Columbia [77]. We identified many examples of LHS activities among the different Kaiser Permanente sites [29, 32]. Kaiser Permanente Southern California developed an Outpatient Safety Net Program, which uses clinical surveillance software to regularly scan outpatient EHR data to identify care gaps [33]. Similar initiatives included screening for new diagnoses and ensuring follow-up on positive tests, such as cancer screening, kidney disease, and Hepatitis C, and identification of potentially harmful medications and interactions, or those that require laboratory monitoring.

To address an identified problem of missed appointments, Kaiser Permanente Colorado investigators conducted a randomized clinical trial and showed that phone call and text message based reminders to patients significantly reduced missed appointments [27]. They also developed and externally validated a prediction model to identify those most likely to miss their appointment.

The Veterans Health Administration (VHA) is the largest integrated health system in the United States consisting of a national network with more than 1200 centers (including over 1000 outpatient clinics) serving 9 million veterans [43]. The VHA uses a combination of research and quality improvement to improve care [44, 45]. They demonstrated one way to embed research into clinical practice through an initiative called Point of Care Research (POC-R) [46]. This initiative facilitates pragmatic trials using EHR data by fully embedding these trials into practice. For example, when a patient is eligible for one of the ongoing trials, a notification is sent through the EHR asking the clinician if they want to randomize the patient or not. A research coordinator then obtains patient consent and all data is collected through the EHR.

Another VHA initiative called Quality Enhancement Research Initiative (QUERI) is an internally-funded program to help the VHA translate research into practice more quickly and efficiently than traditional knowledge translation [45, 47, 48]. Four priority research projects were: 1) home- and community-based care for Veterans at risk of nursing home placement, 2) risk mitigation for patients receiving opioid prescriptions, 3) targeting care for patients at high risk for suicide, and 4) a telehealth tool to improve access to dermatology services. An Implementation Roadmap was also developed to scale up these initiatives and to help clinicians sustain them in practice [48]. This Roadmap was developed based on a review of existing frameworks and successful implementation strategies and based on the expertise of QUERI programs. However, further research and evaluation is needed to confirm and further refine Roadmap components [48].

Intermountain Healthcare is a not-for-profit integrated health system serving patients primarily in Utah, southern Idaho, and southern Nevada [23]. Intermountain Healthcare was an early adopter of a data warehouse in 1998 to measure best practices and costs, and to inform quality improvement initiatives [15]. They continue to use their data to identify gaps and quality improvement efforts to inform clinical decision making [24]. One example is the development of an Area Deprivation Index (a proxy for socioeconomic status), which was evaluated in a prediction model and found to help identify those who would benefit most from enhanced care management services [78].

At Johns Hopkins there are 240 primary care physicians serving over 250,000 patients across Maryland [26]. McGuire (2019) describes the evolution of developing an LHS within primary care at Johns Hopkins, including clinician experts and a quality analyst who provides EHR support, development of a team-based learning culture among staff and clinicians, improvement of patient experience, and incentives for participation in learning activities among clinicians [26]. We could not identify any documented specific LHS projects for Johns Hopkins.

The University of Wisconsin developed a framework for learning health system development and sustainability through their Health Innovation Program, which focuses on research being an equal component to health care [41]. One example using this Health Innovation Program framework is the development of a health case management program to improve primary care for patients with complex medical or social care considerations. Through this program, 20 nurses and social workers were hired as case managers to provide telephone support to these patients. They also developed a prediction model to identify patients who would benefit most from this program. This health case management program is being scaled up across all their primary care clinics, and future evaluations will assess the impact of this scale up [41].

NYU Langone Health has six hospitals and eight primary care and specialty centers [38]. They implemented a rapid-cycle initiative to evaluate existing practices in randomized trials to determine whether they should continue or be modified, and to eventually test new quality improvement initiatives. Over a one-year period, ten existing delivery practices were evaluated – most within primary care – including prompts for flu vaccines and smoking cessation, mailed reminders for appointments, comparison of different telephone scripts for annual visits, and messages to patients to complete health surveys [39].

The Agency for Healthcare Research and Quality (AHRQ) in the United States highlighted the following three large integrated healthcare organizations on their journeys to becoming LHSs: 1) Baylor Scott and White Health – the largest not-for-profit health system in Texas, 2) Denver Health – providing comprehensive care to a third of all Denver residents, and 3) University of Utah Health – an academic healthcare system in Salt Lake City, Utah [16, 17, 40]. Baylor Scott and White Health is planning to transition all their sites to a single EHR and create value-based dashboards for all primary care physicians to standardize information [16]. Denver Health has launched a Quality Improvement Review Committee to review proposals for new initiatives [17]. Denver Health and University of Utah are both using their EHRs to benchmark progress in comparison to baseline data and other leading institutions [17, 40]. University of Utah Health is also using monitoring systems to provide real-time feedback on system-wide issues [40]. However, all three organizations do not seem to have used internal data systems to evaluate the impact on patient care, outcomes or costs of these changes to their practices.

Research / data networks and networks of networks

TRANSFoRm is a network of 21 partner organizations from 10 European countries focusing on primary care research [61]. TRANSFoRm was originally developed for three purposes: 1) prospective study recruitment including randomized trials, 2) retrospective analyses, and 3) decision support for clinical care [62, 63]. Through TRANSFoRm, an electronic solution was developed and validated for standardized and automatic recruitment and data collection for pragmatic clinical trials embedded in EHRs, which is now being deployed in the United Kingdom [79]. The decision support component focused on embedding diagnostic support for primary care clinicians within their EHRs advising on earlier cancer diagnoses [64]. An expansion of TRANSFoRm includes ROAD2H, which is an LHS that will provide decision support combined with local clinical guidelines for primary care to low- and middle-income countries [65].

We identified two other learning health data networks also in Europe: the Swiss Learning Health System (SLHS) in Lucerne, Switzerland and Connected Health Cities in North England, United Kingdom [53, 54, 59]. Projects are being advanced within the SLHS including themes of innovation in service delivery, health promotion and prevention, and health systems guidance and intelligence [60]. The Connected Health Cities was a pilot project leading to over 10 million citizens across North England with connected health records [53]. Sixteen clinical pathway projects have been developed using these health records to help improve patient care. One example is the development of a National Antibiotic Prescribing Dashboard, which uses anonymous data to allow primary care physicians to compare their antibiotic prescribing practices to national and local averages, and allows them to identify higher risk patients [53].

In the United States, we identified two university-based learning health networks that include primary care: Northwestern University Clinical and Translational Sciences Institute (NUCATS) and Health Sciences South Carolina [55, 57]. NUCATS supports clinical trials, community-based research, as well as dissemination and implementation of findings [57]. We could not identify any documented primary care-specific LHS initiatives for NUCATS and Health Sciences South Carolina.

We identified three network of networks, where two of these – the Healthcare Systems Research Network (HCSRN) and the High Value Healthcare Collaborative (HVHC) – each consists of 19 different healthcare systems across the United States, and Optum Labs which standardizes EHR data from across 52 health systems in the United States [13, 14, 72]. Optum Labs describes the ability to use their healthcare data to conduct “N of 1” studies, so healthcare providers can generate evidence that is directly applicable to complex, unique patients at the point of care [72]. We could not identify any documented system level improvement activities nor any primary care-specific LHS initiatives for HCSRN and HVHC.

Challenges and potential solutions

The LHSs included in this review identified challenges that they experienced or anticipated and potential solutions to these challenges in regards to data standardization or quality, ease of data access and use, financial sustainability, promoting a culture of learning, involvement of patients and the community, consistency across different sites within the same organization, prioritization of learning initiatives, the use of EHRs for quality improvement, and the move toward embedding evaluation and implementation of improved practices into usual care rather than traditional research projects (Table 2).

Table 2 Summary of challenges and potential solutions identified by the included learning health systems

Discussion

Summary of evidence

Large integrated health systems in the United States and research / data networks in the United States and Europe provide some of the leading examples on developing an LHS within primary care. We identified only one LHS that operates exclusively in primary care, which is a research-funded initiative in Europe, TRANSFoRm. It is not clear from the data sources how integrated TRANSFoRm is with healthcare delivery organizations, since their partners are primarily academic organizations [62,63,64,65, 79]. The PCORnet projects demonstrate the United States’ motivation to move towards a national-level LHS, yet many of these networks need to apply more rapid quality improvement initiatives to keep moving in this direction rather than relying on traditional research [76, 81].

We identified challenges to initiating or sustaining an LHS in primary care and some potential solutions to these challenges. However, these solutions were presented at a high level and may need to be described in more detail to be useful for other organizations and networks looking to apply them.

Implications and recommendations

Integrated health systems are at an advantage of having access to large amounts of healthcare data and delivering care to patients, compared to organizations that have access to healthcare data but are not directly responsible for delivering care. Furthermore, learning activities are generally funded by patient care revenue, so savings or improvements can be directly applied, thus providing an ideal platform for a self-sustaining LHS.

On the other hand, data networks develop the mechanisms and find the resources to link and analyze healthcare data, and although the data contributors to the network may be healthcare organizations, the networks themselves do not provide patient care, making it difficult for networks to directly develop interventions to improve care across all member organizations. This was why many of the data networks identified in this review did not meet our definition of a fully-functioning LHS. The data networks that were eligible provided some examples of using data to improve healthcare or at least plans for how they will become a fully-functioning LHS, but they generally showed less maturity than the integrated health systems. Network of networks, such as Optum Labs [72, 73], also use a different approach than the integrated health systems, which has some advantages and disadvantages. They provide a service for data integration and analysis, and a platform for collaboration and research. Although initial investments are required to establish these networks, they could become financially sustainable through the researchers and organizations who pay for their services. Similar to data networks, a disadvantage is their disconnect between healthcare provision and thus their limited ability to directly impact patient care at the system level, as compared to integrated health systems. For both types of networks, the separation between data analysis and responsibility for care improvement may limit their ability to design and evaluate effective interventions to improve care, along the same lines that integrated health systems have achieved.

The purpose of this review was to describe LHSs in primary care, however, we found most of the identified LHSs included other healthcare settings. This could be an advantage, since integrated organizations can apply learnings to the appropriate level more easily than primary care providers who are not closely linked to the rest of the system. Furthermore, many of the learning initiatives described included specific projects and not overall plans for whole system improvement in primary care. Geisinger is one of the longest established self-identified LHSs, and not surprisingly describes how their organization met all the criteria for an LHS, but even here there is little information on how this extends to the primary care setting [20]. Finally, the learning initiatives described in this review mainly include sequential research projects or quality improvement activities, but for an LHS to be as productive as possible it needs to move beyond sequential projects to continual improvement without sacrificing quality. NYU Langone Health seems to be moving in this direction with their rapid-cycle testing of various healthcare interventions using randomized clinical trials [39].

Limitations

There are some limitations of our review that should be noted. We used keywords for LHSs to identify eligible studies. However, this may have biased our review to identifying more LHSs in the United States where this concept is more established, rather than other parts of the world where organizations may not realize that they are functioning as an LHS. It was beyond the scope of this review to seek out LHSs that were not self-identifying as such, although we did find a few using our search strategy. This presented another challenge of identifying organizations and networks that are true LHSs that go beyond just a data network, as there is not a clear definition of an LHS in the literature that can be easily applied to assess organizations as such. We created a definition based on existing literature and then refined this through the current search as we identified organizations that exemplified this criteria. We recognize that our definition may be a simplification of the concept. For example, Geisinger identified nine criteria that are required for an LHS [20]. However, most of the identified data sources did not provide enough details to assess whether or not they met all nine criteria, so if we applied this definition we would have underestimated the LHSs included in our review. Furthermore, LHSs also exist on a smaller scale, including solo-practicing physicians who use their data to learn and provide better care, but this was beyond the scope of this review.

We restricted our searches to English language only, which may also have biased the review towards English-speaking countries such as the United States and the United Kingdom. Finally, we only had one individual available at the time to perform the title and abstract review, even though guidelines recommend that two people should be involved at this step.

Conclusions

This is the first scoping review to identify the extent and characteristics of LHSs in primary care. We identified 21 LHSs that included primary care, although all but one included care from other settings, and most were in the United States. We presented some example projects and some challenges and potential solutions that can be applied to future primary care LHSs. The potential of LHSs in primary care has started to be realized and will hopefully be expanded on in the future as more data networks and organizations move toward an LHS and focus on quality improvement that goes beyond traditional research.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

AHRQ:

Agency for Healthcare Research and Quality

EHRs:

Electronic Health Records

HCSRN:

Health Care Systems Research Network

HVHC:

High Value Healthcare Collaborative

IOM:

Institute of Medicine

LHS:

Learning Health System

NUCATS:

Northwestern University Clinical and Translational Sciences Institute

PCORnet:

Patient-Centered Clinical Research Network

POC-R:

Point of Care Research

QUERI:

Quality Enhancement Research Initiative

SLHS:

Swiss Learning Health System

VHA:

Veterans Health Administration

TRANSFoRm:

Translational medicine and patient safety

References

  1. Lenfant C. Shattuck lecture–clinical research to clinical practice–lost in translation? N Engl J Med. 2003;349(9):868–74. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12944573.

    Article  PubMed  Google Scholar 

  2. Davis D, Evans M, Jadad A, Perrier L, Rath D, Ryan D, et al. The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 2003;327(7405):33–5 Available from: http://www.ncbi.nlm.nih.gov/pubmed/12842955.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Collier R. National Physician Survey: EMR use at 75%. CMAJ. 2015;187(1):E17-8 Available from: http://www.ncbi.nlm.nih.gov/pubmed/25487665.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Nguyen L, Bellucci E, Nguyen LT. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int J Med Inform. 2014;83(11):779–96 Available from: http://www.ncbi.nlm.nih.gov/pubmed/25085286.

    Article  PubMed  Google Scholar 

  5. Canada Health Infoway. Commonwealth Fund 2019 Survey [Internet]. 2020 [cited 2020 Mar 19]. Available from: https://www.infoway-inforoute.ca/en/commonwealth-fund-2019-survey.

  6. Institute of Medicine Roundtable on Evidence-Based Medicine, Olsen L, Aisner D, McGinnis J. The learning health care system: workshop summary. 2007.

  7. AHRQ. About learning health systems [Internet]. 2019 [cited 2020 Mar 19]. Available from: https://www.ahrq.gov/learning-health-systems/about.html.

  8. Budrionis A, Bellika JG. The learning healthcare system: where are we now? A systematic review. J Biomed Inform. 2016;64:87–92.

    Article  PubMed  Google Scholar 

  9. Ministry of Health and Long-Term Care, eHealth Ontario. 2016 annual report of the office of the auditor general of Ontario: electronic health records’ implementation status [Internet]. 2016. Available from: http://www.auditor.on.ca/en/content/annualreports/arreports/en16/v1_303en16.pdf.

  10. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

    Article  PubMed  Google Scholar 

  11. Peters MD, Godfrey CM, McInerney P, Khalil H, Parker D, Baldini SC. 2017 Guidance for the conduct of JBI scoping reviews: chapter 11 scoping reviews. Joana Briggs Inst Rev Man. 2015;13(3):141–6.

    Google Scholar 

  12. Covidence. Sign in to covidence [Internet]. 2019 [cited 2019 Nov 25]. Available from: https://www.covidence.org/sign_in.

  13. Rahm AK, Ladd I, Burnett-Hartman AN, Epstein MM, Lowery JT, Lu CY, et al. The Healthcare Systems Research Network (HCSRN) as an environment for dissemination and implementation research: a case study of developing a multi-site research study in precision medicine. eGEMs. 2019;7(1):1–9.

    Article  Google Scholar 

  14. Priest EL, Klekar C, Cantu G, Berryman C, Garinger G, Hall L, et al. Developing electronic data methods infrastructure to participate in collaborative research networks. eGEMs. 2014;2(1):18.

    Article  Google Scholar 

  15. Evans RS, Lloyd JF, Pierce LA. Clinical use of an enterprise data warehouse. AMIA Annu Symp Proc. 2012;2012:189–98.

    PubMed  PubMed Central  Google Scholar 

  16. Agency for Healthcare Research and Quality. Baylor Scott & White Health: building a foundation for continuous improvement [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.ahrq.gov/sites/default/files/wysiwyg/lhs/lhs_case_studies_bsw.pdf.

  17. Agency for Healthcare Research and Quality. Denver health: how a safety net system maximizes its value [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.ahrq.gov/sites/default/files/wysiwyg/lhs/lhs_case_studies_denver_health.pdf.

  18. Psek W, Davis FD, Gerrity G, Stametz R, Bailey-Davis L, Henninger D, et al. Operationalizing the learning health care system in an integrated delivery system. eGEMs. 2016;4(3):6.

    Article  Google Scholar 

  19. Geisinger. Learning healthcare system [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.geisinger.edu/research/research-and-innovation/learning-healthcare-system.

  20. Psek WA, Stametz RA, Bailey-Davis LD, Davis D. Operationalizing the learning health care system in an integrated delivery system. eGEMs. 2015;3(1):6.

    Article  Google Scholar 

  21. Clarke D, Gerrity G, Stametz R, Young A, Davis D. Organizational learning in an integrated health system: informing operations for a learning health care system. J Patient Cent Res Rev. 2017;4:193–4.

    Article  Google Scholar 

  22. Williams MS, Buchanan AH, Davis FD, Faucett WA, Hallquist MLG, Leader JB, et al. Patient-centered precision health in a learning health care system: Geisinger’s genomic medicine experience. Health Aff. 2018;37(5):757–64 Available from: 29733722 NS-.

    Article  Google Scholar 

  23. Intermountain Healthcare. About intermountain [Internet]. 2019 [cited 2019 Nov 19]. Available from: https://intermountainhealthcare.org/about/.

  24. Brunisholz KD, Kim J, Savitz LA, Hashibe M, Gren LH, Hamilton S, et al. A formative evaluation of a diabetes prevention program using the RE-AIM framework in a learning health care system, Utah, 2013–2015. Prev Chronic Dis. 2017;14(7). Available from: http://www.cdc.gov/pcd/issues/2017/16_0556.htm.

  25. Knighton AJ, Savitz L, Belnap T, Stephenson B, VanDerslice J. Introduction of an area deprivation index measuring patient socio-economic status in an integrated health system: implications for population health. eGEMs. 2016;4(3):9.

    Article  Google Scholar 

  26. McGuire MJ. Building learning health care systems in primary care. Qual Manag Health Care. 2019;28(4):252–3.

    Article  PubMed  Google Scholar 

  27. Steiner JF, Shainline MR, Bishop MC, Xu S. Reducing missed primary care appointments in a learning health system. Med Care. 2016;54(7):689–96.

    Article  PubMed  Google Scholar 

  28. Kaiser Permanente Colorado. Care and coverage together help Colorado thrive [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://thrive.kaiserpermanente.org/care-near-colorado.

  29. Kaiser Permanente Northern California. Care and coverage together help Northern California Thrive [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://thrive.kaiserpermanente.org/care-near-you/northern-california/.

  30. Grant RW, Uratsu CS, Estacio KR, Altschuler A, Kim E, Fireman B, et al. Pre-visit prioritization for complex patients with diabetes: randomized trial design and implementation within an integrated health care system. Contemp Clin Trials. 2016;47:196–201. https://doi.org/10.1016/j.cct.2016.01.012.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Schmittdiel JA, Dlott RS, Young JD, Rothman MB, Dyer W, Adams AS. The delivery science rapid analysis program: a research and operational partnership at Kaiser Permanente Northern California. Learn Heal Syst. 2017;1(4):e10035.

    Article  Google Scholar 

  32. Kaiser Permanente Southern California. Care and coverage together makes Southern California healthier [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://thrive.kaiserpermanente.org/care-near-you/southern-california/.

  33. Danforth KN, Smith AE, Loo RK, Jacobsen SJ, Mittman BS, Kanter MH, et al. Electronic clinical surveillance to improve outpatient care: diverse applications within an integrated delivery system. eGEMs. 2014;2(1):9.

    Article  Google Scholar 

  34. Munoz-Plaza CE, Parry C, Hahn EE, Tang T, Nguyen HQ, Gould MK, et al. Integrating qualitative research methods into care improvement efforts within a learning health system: addressing antibiotic overuse. Heal Res Policy Syst. 2016;14(63):1–10. https://doi.org/10.1186/s12961-016-0122-3.

    Article  Google Scholar 

  35. Henry SL, Mohan Y, Whittaker JL, Koster MA, Schottinger JE, Kanter MH. E-SCOPE: a strategic approach to identify and accelerate implementation of evidence-based best practices. Med Care. 2019;57(10):S239–45.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157:207–10.

    Article  PubMed  Google Scholar 

  37. Kaiser Permanente Washington. Learning health system program [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.kpwashingtonresearch.org/about-us/capabilities/learning-health-system-program.

  38. NYU Langone Health. Family Health Centers at NYU Langone [Internet]. 2019 [cited 2020 Jan 17]. Available from: https://nyulangone.org/locations/family-health-centers-at-nyu-langone.

  39. Horwitz LI, Kuznetsova M, Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med. 2019;381(12):1175–9.

    Article  PubMed  Google Scholar 

  40. Agency for Healthcare Research and Quality. University of Utah Health: creating a formula for value-based care [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.ahrq.gov/sites/default/files/wysiwyg/lhs/lhs_case_studies_utah_health.pdf.

  41. Smith MA, Nordby PA, Yu M, Jaffery J. A practical model for research with learning health systems: Building and implementing effective complex case management. Appl Ergon. 2020;84(September 2019): 103023. https://doi.org/10.1016/j.apergo.2019.103023.

    Article  PubMed  Google Scholar 

  42. University of Wisconsin. Health innovation program [Internet]. 2019 [cited 2020 Jan 17]. Available from: https://hip.wisc.edu/.

  43. U.S. Department of Veterans Affairs. Veterans health administration [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.va.gov/health/.

  44. Atkins D, Kilbourne AM, Shulkin D. Moving from discovery to system-wide change: the role of research in a learning health care system: experience from three decades of health systems research in the Veterans Health Administration. Annu Rev Public Heal. 2017;38:467–87.

    Article  Google Scholar 

  45. Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating research impact in a learning health care system. Med Care. 2017;55(7):S4-12.

    Article  PubMed  Google Scholar 

  46. Weir CR, Butler J, Thraen I, Woods PA, Hermos J, Ferguson R, et al. Veterans Healthcare Administration providers’ attitudes and perceptions regarding pragmatic trials embedded at the point of care. Clin Trials. 2014;11(3):292–9. https://doi.org/10.1177/1740774514523848.

    Article  PubMed  Google Scholar 

  47. Frakt AB, Prentice JC, Pizer SD, Elwy AR, Garrido MM, Kilbourne AM, et al. Overcoming challenges to evidence-based policy development in a large, integrated delivery system. Health Serv Res. 2018;53(6):4789–807.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality enhancement research initiative implementation roadmap: toward sustainability of evidence-based practices in a learning health system. Med Care. 2019;57(10):S286–93.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Garrido MM, Allman RM, Pizer SD, Rudolph JL, Thomas KS, Sperber NR, et al. Innovation in a learning health care system: veteran-directed home- and community-based services. J Am Geriatr Soc. 2017;65(11):2446–51.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Minegishi T, Frakt AB, Garrido MM, Gellad WF, Hausmann LRM, Lewis ET, et al. Randomized program evaluation of the Veterans Health Administration Stratification Tool for Opioid Risk Mitigation (STORM): a research and clinical operations partnership to examine effectiveness. Subst Abus. 2019;40(1):14–9. https://doi.org/10.1080/08897077.2018.1540376.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Wu RR, Kinsinger LS, Provenzale D, King HA, Akerly P, Barnes LK, et al. Implementation of new clinical programs in the VHA Healthcare System: the importance of early collaboration between clinical leadership and research. J Gen Intern Med. 2014;29(4):825–30.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Kilbourne AM, Braganza MZ, Bowersox NW, Goodrich DE, Miake-Lye I, Floyd N, et al. Research lifecycle to increase the substantial real-world impact of research: accelerating innovations to application. Med Care. 2019;57(10):S206–12.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Connected Health Cities. Connected health cities [Internet]. 2016 [cited 2019 Nov 14]. Available from: https://www.connectedhealthcities.org/.

  54. Steels S, Van Staa T. Evaluation protocol of the implementation of a learning healthcare system in clinical practice: The Connected Health Cities programme in the north of England. BMJ Open. 2019;9(6):e025484.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Turley CB. Leveraging a statewide clinical data warehouse to expand boundaries of the learning health system. eGEMs. 2016;4(1):25.

    Article  Google Scholar 

  56. Health Sciences South Carolina. Health Sciences South Carolina [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.healthsciencessc.org/.

  57. Starren JB, Winter AQ, Lloyd-Jones DM. Enabling a learning health system through a unified enterprise data warehouse: the experience of the Northwestern University Clinical and Translational Sciences (NUCATS) Institute. Clin Transl Sci. 2015;8(4):269–71.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Northwestern University. Northwestern University Clinical and Translational Sciences Institute [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.nucats.northwestern.edu/.

  59. Boes S, Mantwill S, Kaufmann C, Brach M, Bickenbach J, Rubinelli S, et al. Swiss Learning Health System: a national initiative to establish learning cycles for continuous health system improvement. Learn Heal Syst. 2018;2(3):2–7.

    Google Scholar 

  60. Swiss Learning Health System. About the project [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.slhs.ch/en/about-the-project.

  61. i~HD. TRANSFoRm [Internet]. 2016 [cited 2019 Nov 14]. Available from: https://www.i-hd.eu/index.cfm/resources/ec-projects-results/transform/.

  62. Delaney BC, Curcin V, Andreasson A, Arvanitis TN, Bastiaens H, Corrigan D, et al. Translational medicine and patient safety in Europe: TRANSFoRm - architecture for the learning health system in Europe. Biomed Res Int. 2015;2015:1–8.

    Article  Google Scholar 

  63. Ethier JF, McGilchrist M, Barton A, Cloutier AM, Curcin V, Delaney BC, et al. The TRANSFoRm project: experience and lessons learned regarding functional and interoperability requirements to support primary care. Learn Heal Syst. 2018;2(2):1–7.

    Google Scholar 

  64. Cancer Research UK. Writing a successful population research proposal – Professor Brendan Delaney [Internet]. 2019 [cited 2019 Nov 15]. Available from: https://www.cancerresearchuk.org/funding-for-researchers/research-features/2019-03-07-writing-a-successful-population-research-proposal-professor-brendan-delaney?_ga=2.267090246.913723477.1572876178-858616111.1572876178.

  65. ROAD2H. Developing novel learning health system techniques to facilitate Universal Health Coverage (UHC) in low- and middle-income countries [Internet]. 2019 [cited 2019 Nov 15]. Available from: http://www.road2h.org/.

  66. Delaney BC. TRANSFoRm: translational medicine and patient safety in Europe. In: Institute of Medicine, editor. Digital infrastructure for the learning health system: the foundation for continuous improvement in health and health care: workshop series summary [Internet]. Washington, D.C.: National Academy of Sciences; 2011. p. 198–202. Available from: https://www.ncbi.nlm.nih.gov/books/NBK83569/pdf/Bookshelf_NBK83569.pdf#page=224.

  67. Mastellos N, Bliźniuk G, Czopnik D, McGilchrist M, Misiaszek A, Bródka P, et al. Feasibility and acceptability of TRANSFoRm to improve clinical trial recruitment in primary care. Fam Pract. 2016;33(2):186–91.

    Article  PubMed  Google Scholar 

  68. Saganowski S, Misiaszek A, Bródka P, Andreasson A, Curcin V, Delaney B, et al. TRANSFoRm eHealth solution for quality of life monitoring. AMIA Jt Summits Transl Sci proceedings AMIA Jt Summits Transl Sci. 2016;2016(247787):231–9.

    Google Scholar 

  69. HCSRN. Health Care Systems Research Network [Internet]. 2019 [cited 2019 Nov 14]. Available from: http://www.hcsrn.org/en/.

  70. Tomek I, Sabel A, Froimson M, Koenig K, Lewallen D, Naessens J, et al. A collaborative of leading health systems finds wide variations in total knee replacement delivery and takes steps to improve value. Health Aff. 2012;31(6):1–19.

    Article  Google Scholar 

  71. High Value Healthcare Collaborative. High value healthcare collaborative [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.highvaluehealthcare.org/.

  72. Wallace PJ, Shah ND, Dennen T, Bleicher PA, Crown WH. Optum labs: building a novel node in the learning health care system. Health Aff. 2014;33(7):1187–94.

    Article  Google Scholar 

  73. Dore D, Ciofani D, Davis S, Nunes A, Bradley J, Seeger J, et al. Creating a large-scale physically integrated electronic health record data system to support a learning healthcare system. Value Heal. 2017;20(5):A321 Available from: https://linkinghub.elsevier.com/retrieve/pii/S1098301517302516.

    Google Scholar 

  74. Optum. Optum Labs [Internet]. 2019 [cited 2019 Nov 14]. Available from: https://www.optumlabs.com/.

  75. Psek W, Davis FD, Gerrity G, Stametz R, Bailey-Davis L, Henninger D, et al. Leadership perspectives on operationalizing the learning health care system in an integrated delivery system. eGEMs. 2016;4(3):6.

    Article  Google Scholar 

  76. Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. Launching PCORnet, a national patient-centered clinical research network. J Am Med Informatics Assoc. 2014;21(4):578–82.

    Article  Google Scholar 

  77. Kaiser Foundation Health Plan Inc. Kaiser Permanente [Internet]. 2019 [cited 2019 Nov 15]. Available from: https://healthy.kaiserpermanente.org/.

  78. Knighton AJ, Savitz L, Belnap T, Stephenson B, VanDerslice J. Introduction of an area deprivation index measuring patient socio-economic status in an integrated health system: implications for population health. eGEMs. 2016;4(3):9.

    Article  Google Scholar 

  79. Ethier JF, Curcin V, McGilchrist MM, Choi Keung SNL, Zhao L, Andreasson A, et al. eSource for clinical trials: implementation and evaluation of a standards-based approach in a real world trial. Int J Med Inform. 2017;106(January):17–24. https://doi.org/10.1016/j.ijmedinf.2017.06.006.

    Article  PubMed  Google Scholar 

  80. Jeffries M, Keers RN, Phipps DL, Williams R, Brown B, Avery AJ, et al. Developing a learning health system: insights from a qualitative process evaluation of a pharmacist-led electronic audit and feedback intervention to improve medication safety in primary care. PLoS One. 2018;13(10):e0205419.https://doi.org/10.1371/journal.pone.0205419.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  81. Califf RM. The patient-centered outcomes research network. N C Med J. 2014;75(3):204–10.

    PubMed  Google Scholar 

Download references

Acknowledgements

We would like to express our sincere appreciation to Marissa Tippett at Western University for assisting with the literature search strategy, and Lorri Zagar – a Quality Improvement Consultant – and Dr. Christopher Forrest – a Pediatrician and Director of the Center for Applied Clinical Research at Children’s Hospital of Philadelphia – for reading and providing feedback on our manuscript.

Funding

DMN was supported by a Canadian Institutes of Health Research Fellowship. MZ and JR received a Canadian Institutes of Health Research Planning / Dissemination Grant.

Author information

Authors and Affiliations

Authors

Contributions

MZ and DMN were responsible for the study conception. MZ and DMN developed the search strategy in collaboration with JR. DMN did the screening, created the data charting form and completed the full-text review. ZB further reviewed full-text articles for eligibility and charted the data from the eligible articles, while DMN reviewed and confirmed all information. MZ supervised the execution of this review and JR was also involved throughout the process. DMN and MZ drafted the initial manuscript. All authors read and approved of the final manuscript.

Corresponding author

Correspondence to Danielle M. Nash.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors have no competing or conflicting interests to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1

. Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist. Reporting checklist for systematic reviews

Additional file 2

. Search strategies for Medline®, Embase® and IEEE Xplore®. Screenshots of the searches completed through different databases, including the keywords and mesh terms used

Additional file 3

. Charted data elements and definitions or scenarios for meeting certain criteria. List of charted data elements abstracted from the included articles and definitions or scenarios for meeting certain criteria

Additional file 4

. Integrated health systems and data networks with potential to become learning health systems. Summary of the identified integrated health systems and data networks with potential to become learning health systems and explanation for why they were not considered fully-functioning learning health systems

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nash, D.M., Bhimani, Z., Rayner, J. et al. Learning health systems in primary care: a systematic scoping review. BMC Fam Pract 22, 126 (2021). https://doi.org/10.1186/s12875-021-01483-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12875-021-01483-z

Keywords