This article has Open Peer Review reports available.
It could be a ‘Golden Goose’: a qualitative study of views in primary care on an emergency admission risk prediction tool prior to implementation
© Porter et al. 2016
Received: 28 August 2015
Accepted: 24 December 2015
Published: 6 January 2016
The Erratum to this article has been published in BMC Family Practice 2016 17:14
Rising demand for health care has prompted interest in new technologies to support a shift of care from hospital to community and primary care, which may require clinicians to undertake new working practices. A predictive risk stratification tool (Prism) was developed for use in primary care to estimate patients’ risk of an emergency hospital admission. As part of an evaluation of Prism, we aimed to understand what might be needed to bring Prism into effective use by exploring clinicians and practice managers’ attitudes and expectations about using it. We were informed by Normalisation Process Theory (NPT) which examines the work needed to bring an innovation into use.
We conducted 4 focus groups and 10 interviews with a total of 43 primary care doctors and colleagues from 32 general practices. All were recorded and transcribed. Analysis focussed in particular on the construct of ‘coherence’ within NPT, which examines how people understand an innovation and its purpose.
Respondents were in agreement that Prism was a technological formalisation of existing practice, and that it would function as a support to clinical judgment, rather than replacing it. There was broad consensus about the role it might have in delivering new models of care based on active management, but there were doubts about the scope for making a difference to some patients and about whether Prism could identify at-risk patients not already known to the clinical team. Respondents did not expect using the tool to be onerous, but were concerned about the work which might follow in delivering care. Any potential value would not be of the tool in isolation, but would depend on the availability of support services.
Policy imperatives and the pressure of rising demand meant respondents were open to trying out Prism, despite underlying uncertainty about what difference it could make.
Controlled Clinical Trials no. ISRCTN55538212.
KeywordsNormalisation process theory Process evaluation Intervention studies Primary care Clinical prediction rule Risk assessment Health services/utilisation Emergency admissions Chronic disease
Rising demand for health care, particularly from people living with long term conditions, is prompting service providers and policy makers to turn to new technologies in order to improve efficiency and effectiveness. Clinical risk prediction models use demographic information, diagnoses, and service use data to stratify a population’s risk of having or developing a specified disease, or experiencing an outcome such as emergency admission to hospital [1, 2]. In combination with a software platform, the models provide clinical risk prediction tools , whose two functions in supporting health care are case finding for improved clinical management, and resource allocation through budgeting or service planning . In terms of case finding, such tools are seen as potentially more accurate and consistent than clinical opinion in identifying patients at risk of unscheduled hospital admission . In recent years, contractual mechanisms in all four nations of the UK have incentivised risk profiling and stratification as part of a proactive model of care for patients at risk of emergency admission [6, 7]. The funding commitment is considerable: £160 million per year in England alone .
Though a systematic review identified 27 validated risk models for emergency hospital admissions in use internationally , their clinical and cost effectiveness have not been demonstrated and practical aspects of adoption and use in general practice are poorly understood . A range of risk prediction tools—including the Combined Model  developed for NHS England, and QAdmissions —are intended to produce stratified lists of patients, but there is some variation in the underlying modelling (for example, whether or not secondary care data is used alongside primary care data). Prism  is a tool developed specifically for NHS Wales, using a 37-variable model based on Welsh data sets, and updated monthly with data from GP practices, and designed to reflect the particular service configuration of the Welsh NHS and the service needs of the Welsh population. We are currently evaluating Prism in general practice in Wales , including examining how Prism is being implemented into practice.
Implementation into health care of new technology such as risk prediction tools can be slow and difficult [14, 15]. There is an increasing acknowledgement of the complexities of the process of bringing new technology into routine use: what is implemented is a package of behaviour change, not just a piece of equipment or protocol ; it is a process over time [16, 17] and it involves a complex social system, in which multiple parties are involved [17–19], some of which may be resistant .
Normalisation Process Theory (NPT)  is increasingly being used as a conceptual framework to examine and explain implementation of innovations in health care . NPT emphasises the nature of implementation as a process which entails sustained work by the parties involved, and suggests that four constructs help us understand how innovation is brought into normal everyday practice: how people understand the innovation and its purpose (coherence); what decisions are taken about using it, based on perceived advantages (cognitive participation); what people do to bring the innovation into everyday use (collective action); how an innovation is reviewed, modified or abandoned (reflexive monitoring) .
Our qualitative study aims to understand what might be needed to bring Prism into effective use by exploring clinicians’ and practice managers’ attitudes and expectations about using it. We used NPT to inform our analysis, focussing in particular on the first construct, coherence. Our study contributes new understanding of the ambiguities of the real-world expectations of primary care practitioners when presented with an opportunity to adopt a technology into practice. It will both help to inform effective implementation, and to bring new insights to process evaluation of interventions which are being trialled to assess their impact.
Setting and intervention
We undertook this qualitative study of the process of implementing the Prism risk prediction tool in primary care as part of a progressive cluster randomised trial of the tool, in line with recent MRC recommendations on process evaluation . PRISMATIC (ISRCTN55538212) (Prism Risk Stratification Model: A Trial in Chronic conditions management) aims to estimate Prism’s effects on delivery of care, patient satisfaction, quality of life and resources used . PRISMATIC is under way in one Welsh health board area where 32 practices volunteered to participate. The intervention consisted of Prism software, installed on the practice computer system and providing a range of searchable graphical and numerical presentations of risk information at individual or practice level, automatically updated monthly from routine data. To support this, practices were also provided with a training session of approximately 1 h, delivered in the practice; clinical support through two locally appointed ‘GP champions’; a telephone ‘help desk’ at the NHS Wales Informatics Service (NWIS), during working hours; and a user-friendly handbook of guidance on using Prism. Participating practices received a payment of £1250 to cover costs associated with taking part in the trial.
Qualitative data collection took place between October 2012 and April 2013, after the Prism software was installed on the practice computer system, but before Prism was activated for use or training provided. During the course of the data collection, NHS Wales announced that the contract with primary care providers for 2013–14 would include an additional payment for active care management of at least 0.5 % of the practice list predicted to be at high risk of hospital admission . Some data collection took place before this announcement, and some after it.
We invited each GP who had been nominated as lead for Prism in participating practices to attend one of four focus groups, along with other staff members (such as practice manager or nurse) if desired. 33 respondents attended focus groups—Group A (5 GPs; 4 practice managers; 1 nurse), Group B (4:3:1), Group C (7:1:0), Group D (5:2:0). We also interviewed 10 GPs who were unable to attend, by telephone (n = 8) or in person (n = 2), giving a total of 43 participants from across all 32 practices. Participants represented a mix of urban, rural and semi-rural practices. 21 were female. Mean number of years of experience in their role was 14 (range: 3 to 28).
We devised a topic guide (piloted before use) for focus groups and interviews, covering knowledge of existing predictive risk tools; approaches currently used to identify patients at risk of hospitalisation; motivations for participating in the PRISMATIC trial; expectations of Prism use and impact; and any concerns. During the course of the focus group/interview, we presented a summary of the Prism tool with screenshots to enable respondents to understand and visualise how they might use it in practice. Focus groups, conducted by BAE or AP supported by MK, lasted 65–90 min. Interviews, conducted by the same researchers, lasted 28–40 min. All were recorded, with consent, and fully transcribed, except for one interview where detailed notes were taken.
differentiating it from other practices;
agreeing its purpose;
understanding how they might use it;
identifying its potential value.
NPT suggests that each of these tasks is shaped by factors that promote or inhibit the extent to which participants look on a new practice as meaningful.
In line with the method outlined by Bradley et al., our analysis integrated inductive and deductive approaches . We developed an initial coding frame for analysis, informed by NPT and by the major concepts identified by BAE, AP, MK and service user SW during an initial review of transcripts, and revised and refined this frame during analysis. We used NVIVO 9 software to organise and manage data, to code into groups and connect data into meaningful themes. AP and MK led coding, comparing results and meeting with all authors to confirm consistency and resolve discrepancies through discussion and comparison with raw data. To strengthen credibility, we presented findings to three GPs associated with the study who agreed they were ‘recognisable’ and seemed to ‘fit’ with their experience .
The PRISMATIC trial, including this qualitative component, received approval from the National Research Ethics Service for Wales (10/MRE09/25). We gained informed consent from all participants before data collection. We followed recommended ethical standards for conducting this study and storing all data .
Defining and differentiating Prism from current practice
We want a systematic way rather than just seeing people opportunistically because they are ill, we want to get there before they’re ill and on the point of going into hospital. We want to do it in a more orderly manner. [GP 30, Group D]
The Prism tool was not seen as something entirely new. Many respondents made comparison between the principles and process of Prism and condition-specific risk prediction tools with which they were already familiar, such as the Wells score (estimating the probability of deep vein thrombosis or pulmonary embolism) and Framingham score (cardiovascular risk).
We can always refer back to that tool that “this is the scoring system, that was the reason I admitted this patient or they did not need admission”. [GP 25, Group B]
We have been thinking for a while about looking more in depth at some of the—certain individuals. And I think this might give us the impetus to actually carry that through. [GP 15, Interview]
Agreeing a purpose for Prism
That’s how I would go about it, you see, identifying the patients that may have a problem, put in place, or mobilise forces that might be able to go in, and help a little bit—you know, if somebody has a chest infection and they’re having difficulty in breathing, they’re in need of—intravenous antibiotics, the [community support] team can do that at home—she doesn’t have to go in. [GP 9, Interview]
I suppose, one of the overwhelming senses, was that if somebody were to give us a list of our top 100 patients, we’d say, “Well, yeah, you know, we already know about them.” ....We felt that the chance of this churning out unexpected patients was probably pretty low. [GP 15, Interview]
The idea is that you can predict who they are and you can do something about it, so they don’t need to be going into hospital. And that sort of assumes lots of things, really. I mean, it assumes first of all that you can predict it, but secondly it presumes that some of these things are modifiable. [GP 15, Interview]
So, if your patient has a certain level of risk, in the top 5 %…and you haven’t [been able] to get them lower…that’s my worry, because it’ll be like a stick to beat you. [GP 12, Group C]
If this helps us define and gives evidence to what we need, that’s where I see it coming from. [GP 13, Group C]
In interviews and focus groups, most respondents reflected on Prism from an individual perspective, rather than reporting any collective viewpoint in their practice. However, a minority of respondents did report on discussions with colleagues about Prism, with GPs saying that they had persuaded partners, and some practice managers that they had convinced GPs, to take the opportunity to try the technology.
Understanding how they might use Prism
If it looks like that on the webpage, then that’s very usable, isn’t it. It’s clicking and it’s very organised and the figures are there nice and neatly and you can scroll up and down over a cup of coffee. [GP 19, Group D]
We’ll involve the whole practice. We’ll involve the nurses—the clinical side, isn’t it? And we’ll have a look at it in clinical meetings, which we have in any case, and we’ll go from there, I think. [GP 1, Interview]
It wouldn’t be everybody using it, but I think as long as we’ve got a doctor onboard…I’m there, and then it’s the nurse—a small group…Sometimes if you get too many people involved, nothing gets done. [Practice Manager 7, Group A]
I think obviously a meeting first of all…to lead us in the right direction…[work with the patient] will come my way. [Practice Nurse, Group B]
If we say, “Well, we’re gonna do this, but it might make X hours extra work,” [my partners] will say to me, “Well, where’s that X hours gonna come from?”. [GP 9, Interview]
We struggle at the moment anyway—everybody’s the same with time, with clinics, scripts, mail—everything. After all of this there’s not much time to put in. [Practice Manager 4, Group C].
If you are presented with a list of patients who are deemed to be high risk and there may be people that you haven’t had contact with, that obviously is gonna generate work because you can’t really ignore the fact that they’re deemed at risk and you’d then have to visit or contact them.[GP 25, Group C]
Especially if this tool identified patients that are unknown to us, then you’ve gotta go digging…Because we know—we all know Dai down the road, with COPD, who’s back and forth all the time, because he’s in the surgery all the time. But, then, Mr Thomas down here, who we said, “Well, who the hell is he?”. [Practice Manager 31, Group A]
Identifying potential value for Prism
GPs and practice staff showed a willingness and open-mindedness about trying the Prism risk prediction tool. Some were curious, keen to find out about new technology or to avoid missing extra resources or information which could improve their patient care. Most saw an opportunity to help their general practice by supplementing their access to data and supporting the move towards proactive health care. But underlying the apparent curiosity, there was an urgent and anxious search for ways to change working practices and pressures, to move away from current practices which were seen as unsustainable, a case of ‘trying to keep our head above water.’ [GP 18, Group B]
I’ll give it a go, see if it does work, if it doesn’t at least we tried, otherwise we just stay in the status quo, which we can’t stay in as the current health service is being funded. [GP 20, Group C]
So part of me coming [to the focus group] is to see what we can do with it…we all do it [identifying people at risk] on a sort of informal basis. We recognise who’s ill and who’s the really worrying patients…if you can use that [Prism], then it has a benefit but what that is, is a Golden Goose so to speak. [GP 6, Group D]
We have to have a system in place, where there is an alternative…if there’s no alternative, then all of our GPs, all we’re gonna do is just say “admit, admit, admit”. [GP 11, Interview]
You know, there’s nothing worse than saying to somebody, “Well, actually, you need this, but we can’t give it to you.” [GP 26, Interview]
And it may identify work—a workload—or it may identify needs, that we may feel unable to address. That can be demoralising, can’t it?…And frustrating. [GP 9, Interview]
GPs will generally use anything that they feel is of benefit to their patients. I think the difficulty I would have is not really knowing if it is gonna be of benefit. I’m quite happy to give it a go. [GP 21, Group C]
New health care technologies are promoted as a way to improve the quality of patient care and effectiveness and efficiency of health services . However, uptake and use of new technologies can be slow, even when clinical effectiveness is already known [14, 29]. NPT provides a useful sensitising theory to inform analysis of implementation, identifying coherence—the mechanism by which participants understand an innovation and its purpose—as one of four essential components to the work of bringing an innovation into routine practice. Informed by this, we explored attitudes of general practice staff before they began to pilot the Prism tool. We were interested to know how respondents understood what Prism was and what it was for.
NPT suggests that achieving the embedding of innovation into routine use is dependent on coherence , yet previous applications of the theory have identified that achieving coherence can be the most common obstacle to implementation [30–32].
We found that, in terms of the four tasks which May and Finch  suggested make up coherence, there was a mixed story from our respondents. There seemed to be clear agreement on what the tool was—a technological formalisation of existing practice—and a common understanding that it would function as a support to clinical judgment, rather than a replacement for it. Respondents generally saw Prism as something additional to their existing workload, rather than at the core of it. Though Prism stratifies the whole patient population, respondents tended to focus in their discussion on those at the highest risk. When it came to agreeing a purpose, there was broad consensus about the role Prism might have in delivering new models of care based on active management of patients, but also some scepticism. Scepticism about the potential impact of Prism went along two contradictory lines: one was that it would not be useful in identifying high-risk patients, because these were already known; the other was that it would uncover previously unknown high risk patients for whom no practical support could be delivered.
In terms of the work needed to put Prism into use, the majority of respondents did not anticipate that gathering information from the tool would be a major task in itself. The anticipated workload came with what followed—since the tool has no scope to make an impact in isolation, it would lead to a new workload in terms of reviewing patient needs, care planning, and identifying and setting up support services. There was some anxiety about inappropriately raised expectations, and, to a small extent, liability for responding to identified needs. These concerns spilled over into the consideration of the potential value of Prism—because respondents were well aware of the constrained supply of community based care, which would limit the scope for delivering care to patients identified through using Prism, and created uncertainty in what Prism could achieve. One respondent’s comment about the ‘Golden Goose’ referenced, deliberately or not, the world of fairytales.1 Our findings seem to concur with previous work on case management models—which include identification of patients at risk as one of their core tasks—which has found impact to be limited for many reasons, including difficulty of engaging with GPs .
Although we focussed on the NPT component of coherence, there were clear references to other components of the theory—for example, the comments about respondents’ plans to use the tool in clinical meetings relates to the NPT component of collective action. The potential for overlap between constructs, and the influence they can have on each other in the dynamic process of implementation, has previously been noted ; we intend to return to the story of Prism implementation to see how the four constructs have played out over time.
What was striking was, on balance, how the pressures of a rising numbers of patients with long term conditions, together with national, regional and local policy drivers to anticipate patient needs and reducing demand for unscheduled and emergency care, meant that they were open to trying out Prism as it offered at least a hope of improvement. Open-minded willingness to try the tool seemed to over-ride the scepticism, anxiety and uncertainty. A stronger and clearer message to practitioners about the potential benefits of the tool would have been a useful support, but the challenge to offering this was the absence of evidence—which is what the PRISMATIC trial itself was designed to produce.
During the course of the fieldwork, changes to the GP contract introduced new rewards for identifying high risk patients, and Prism was well placed to support practices’ response to this. However, what was encouraged by the new payment scheme was identification of patients at highest risk of emergency admission, and interventions for only a small proportion of these. This was a very narrow focus compared with the much wider range of functions and possibilities available through Prism, and one which is at odds with a literature which suggests that the biggest scope for impact is in identifying and working with the groups of patients in the risk band below the very top, whose needs may be less immediate or extreme .
Strengths and limitations of the study
We interviewed 43 primary care staff with a wide range of experiences, from 32 general practices across rural and urban, prosperous and deprived communities. The data provide a comprehensive spread of views in terms of practices and roles represented. However, in most cases we talked to only one person from each practice and so were not able to explore in full the extent to which those people’s views were representative of their colleagues, nor to discuss in any depth the relative merits of using the tool in an individual or group setting. It may be that there were wide ranges in attitude and expectations among colleagues within a single practice.
These practices had volunteered to take part in the PRISMATIC trial, committing themselves to using the tool for a fixed time period, and receiving a small payment to acknowledge the additional demands on time of practice staff associated with participation in the study. Their views may not be typical of doctors and other staff in practices which did not opt to take part in the study.
This paper examines coherence among the potential front-line users. Their views may differ from those of local commissioners and managers with responsibility for services required by patients at risk of emergency admission and likely to be identified by the technology.
During the course of our fieldwork, practices were informed that a new payment would be available for identifying and managing patients at high risk of hospital admission. In those interviews which took place after this announcement, responses would have been informed and potentially influenced by awareness of this payment and the potential role of Prism in earning it, while in the focus groups and interviews before the announcement, respondents would not necessarily have known that such a payment innovation was planned.
NPT provided a useful framework to help us to examine attitudes of practice staff to a predictive risk tool at the point of its introduction. We know that stages of adoption and implementation are part of a process that comprises both formal organisational decisions and informal decisions by individual users . Overall, respondents’ views were not a coherent story but a mix of open-minded willingness, scepticism, anxiety and uncertainty. This mix of views was observed within individual GPs’, nurses’ and practice managers’ responses, as well as being reflected in a range of viewpoints from different individuals. The fact that Prism was offered to practices as part of a trial meant that it could be perceived as a low risk, ‘no obligation’ option. The small scale incentives which came with participation in the Prismatic trial were simply for taking part, and did not depend on successfully identifying new patients or delivering interventions to them. Taking the first step of agreeing to take part in the trial and installing Prism in the practice required a relatively small investment of time and effort, and—unlike the use of many other risk prediction tools—no cost.
Further research will explore how expectations at the pre-adoption stage compare to decisions and actual use of the technology in general practice. Our PRISMATIC study will follow up general practice staff over 18 months to explore whether and how they use the new technology, allowing us to consider the other mechanisms identified in NPT.
Grimm’s Fairy Tales contain two stories about geese and gold: the goose who lays the golden eggs is the source of undreamt of bounty; but another story has a golden goose to which the unwary stick when they grab it, with unfortunate consequences. We assume that our respondent was thinking of the former.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Castro V, McCoy T, Cagan H, Rosenfield H, Murphy S, Churchill S, et al. Stratification of risk for hospital admissions for injury related to fall: cohort study. BMJ. 2014;349:g5863.PubMed CentralView ArticlePubMedGoogle Scholar
- Janssen K, Vergouwe Y, Kalkman J, Diederick C, Grobbee D, Karel K, et al. A simple method to adjust clinical prediction models to local circumstances. Can J Anesth/J Can Anesth. 2009;56:194–201.View ArticleGoogle Scholar
- Lewis G, Curry N, Bardsley M. Choosing a predictive risk model: a guide for commissioners in England. London: The Nuffield Trust; 2011.Google Scholar
- Georghiou T, Blunt I, Steventon A, Lewis G, Billings J, Bardsley M. Predictive risk and healthcare: an overview. London: The Nuffield Trust; 2011.Google Scholar
- Curry N, Billings J, Darin B, Dixon J, Williams M, Wennberg D. Predictive Risk Project Literature Review. London: The King’s Fund; 2005.Google Scholar
- NHS England. Enhanced Service Specification: Avoiding unplanned admissions: proactive case finding and patient review for vulnerable people 2015. http://www.england.nhs.uk/wp-content/uploads/2014/08/avoid-unplanned-admissions.pdf. Accessed 15 June 2015.
- NHS Wales. Proposed Changes to the GMS Contract 2013/14. 2013. http://www.wales.nhs.uk/sites3/documents/480/Contract%20Revisions%202013-14.pdf. Accessed 13 June 2015.
- Wallace E, Stuart E, Vaughan N, Bennett K, Fahey T, Smith SM. Risk prediction models to predict emergency hospital admission in community-dwelling adults: a systematic review. Med Care. 2014;52:751–65. doi:10.1097/mlr.0000000000000171.PubMed CentralView ArticlePubMedGoogle Scholar
- Lewis G. Next steps for Risk Stratification in the NHS2015. http://www.england.nhs.uk/wp-content/uploads/2015/01/nxt-steps-risk-strat-glewis.pdf. Accessed 17 June 2015.
- Wennberg D, Siegel M, Darin B, Filipova, N Russell, R, et al. Combined predictive model: final report and technical documentation. London: Health Dialog/King's Fund/New York University, 2006. http://www.kingsfund.org.uk/sites/files/kf/Combined_Predictive_Model_Final_Report.pdf. Accessed 10 July 2015.
- Hippisley-Cox J. Coupland C (2013) Predicting risk of emergency admission to hospital using primary care data: derivation and validation of QAdmissions score. BMJ Open. 2013;3:e003482. doi:10.1136/bmjopen-2013-003482.PubMed CentralView ArticlePubMedGoogle Scholar
- Health Dialog, NHS Wales, Informing Healthcare: Wales Predictive Model. Final Report and Technical Documentation. 2008.Google Scholar
- Hutchings H, Evans BA, Fitzsimmons D, Harrison J, Heaven H, Huxley P, et al. Predictive risk stratification model: a progressive cluster-randomised trial in chronic conditions management (PRISMATIC) research protocol. Trials. 2013;14:301.PubMed CentralView ArticlePubMedGoogle Scholar
- Berwick DM. Disseminating innovations in health care. JAMA. 2003;289:1969e75.View ArticleGoogle Scholar
- Ferlie E, Fitzgerald L, Wood M. The nonspread of innovations: the mediating role of professionals. Acad Manag J. 2005;48:117e34.View ArticleGoogle Scholar
- May C. Towards a general theory of implementation. Implement Sci. 2013;8:18.PubMed CentralView ArticlePubMedGoogle Scholar
- Greenhalgh T, Robert G, Bate P, Kyriakidou O, Macfarlane F, Peacock R. How to spread good ideas: a systematic review of the literature on diffusion, dissemination and sustainability of innovations in health service delivery and organisation. London: National Co-ordinating Centre for NHS Service Delivery and Organisation R & D; 2004.Google Scholar
- Greenhalgh T, Swinglehurst D. Studying technology use as social practice: the untapped potential of ethnography. BMC Med. 2011;9:45.PubMed CentralView ArticlePubMedGoogle Scholar
- Arce R, De Ormijana A, Orueta J, Gagnon M-P, Nuno-Solinis R. A qualitative study on clinicians’ perceptions about the implementation of a population risk stratification tool in primary care practice of the Basque Health Service. BMC Fam Pract. 2014;15:150.PubMed CentralView ArticleGoogle Scholar
- Pope C, Halford S, Turnbull J, Prichard J, Calestani M, May C. Using computer decision support systems in NHS emergency and urgent care: ethnographic study using normalisation process theory. BMC HSR. 2013;13:111.Google Scholar
- May C, Finch T. Implementation, embedding, and integration: an outline of Normalization Process Theory. Sociology. 2009;43:535–54.View ArticleGoogle Scholar
- McEvoy R, Ballini L, Maltoni S, O’Donnell C, Mair F, MacFarlane A. A qualitative systematic review of studies using the normalization process theory to research implementation processes. Implement Sci. 2014;9:2.PubMed CentralView ArticlePubMedGoogle Scholar
- Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8:63.PubMed CentralView ArticlePubMedGoogle Scholar
- Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. ‘Process evaluation of complex interventions: Medical Research Council guidance’. BMJ. 2015;350:h1258.PubMed CentralView ArticlePubMedGoogle Scholar
- Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: Developing taxonomy, themes, and theory. Health Serv Res. 2007;42:1758–72.PubMed CentralView ArticlePubMedGoogle Scholar
- Corbin J, Strauss A. Basics of qualitative research. 3rd ed. San Jose: Sage; 2008.Google Scholar
- Silverman D. Doing Qualitative Research. 3rd ed. London: Sage; 2010.Google Scholar
- Darzi A. High Quality Care for All. London: Department of Health; 2008.Google Scholar
- Dixon-Woods M, Amalberti R, Goodman S, Bergman B, Glasziou P. Problems and promises of innovation: why healthcare needs to rethink its love/hate relationship with the new. BMJ QualSafe. 2011;20 Suppl 1:i47–51.View ArticleGoogle Scholar
- Elwyn G, Légaré F, van der Weijden T, Edwards A, May C. Arduous implementation: does the Normalisation Process Model explain why it’s so difficult to embed decision support technologies for patients in routine clinical practice. Implement Sci. 2008;3:57.PubMed CentralView ArticlePubMedGoogle Scholar
- Lloyd A, Joseph-Williams N, Edwards A, Rix A, Elwyn G. Patchy ‘coherence’: using normalization process theory to evaluate a multi-faceted shared decision making implementation program (MAGIC). Implement Sci. 2013;8:102.PubMed CentralView ArticlePubMedGoogle Scholar
- Sanders T, Foster N, Ong BN. Perceptions of general practitioners towards the use of a new system for treating back pain: a qualitative interview study. BMC Med. 2011;9:49.PubMed CentralView ArticlePubMedGoogle Scholar
- Ross S, Curry N, Goodwin N. Case management. What it is and how it can best be implemented. London: The King's Fund; 2011.Google Scholar
- Lewis G. ‘Impactibility models’: identifying the sub-group of high risk patients most amenable to hospital avoidance programs. Milbank Q. 2010;88(2):240–55.PubMed CentralView ArticlePubMedGoogle Scholar
- Robert G, Greenhalgh T, MacFarland F, Peacock R. Adopting and assimilating new non-pharmaceutical technologies into health care: a systematic review. J Health Serv Res Policy. 2010;15(4):243–50. doi:10.1258/jhsrp.2010.009137. Epub 2010 Jun 30.View ArticlePubMedGoogle Scholar