This article has Open Peer Review reports available.
Implementing referral guidelines: lessons from a negative outcome cluster randomised factorial trial in general practice
- Moyez Jiwa†1Email author,
- Paul Skinner†2,
- Akinoso Olujimi Coker†3,
- Lindsey Shaw†4,
- Michael J Campbell†4 and
- Joanne Thompson†4
© Jiwa et al; licensee BioMed Central Ltd. 2006
Received: 20 July 2006
Accepted: 02 November 2006
Published: 02 November 2006
Few patients with lower bowel symptoms who consult their general practitioner need a specialist opinion. However data from referred patients suggest that those who are referred would benefit from detailed assessment before referral.
A cluster randomised factorial trial. 44 general practices in North Trent, UK. Practices were offered either an electronic interactive referral pro forma, an educational outreach visit by a local colorectal surgeon, both or neither. The main outcome measure was the proportion of cases with severe diverticular disease, cancer or precancerous lesions and inflammatory bowel disease in those referred by each group. A secondary outcome was a referral letter quality score. Semi-structured interviews were conducted to identify key themes relating to the use of the software
From 150 invitations, 44 practices were recruited with a total list size of 265,707. There were 716 consecutive referrals recorded over a six-month period, for which a diagnosis was available for 514. In the combined software arms 14% (37/261) had significant pathology, compared with 19% (49/253) in the non-software arms, relative risk 0.73 (95% CI: 0.46 to 1.15). In the combined educational outreach arms 15% (38/258) had significant pathology compared with 19% (48/256) in the non-educational arms, relative risk 0.79 (95% CI: 0.50 to 1.24). Pro forma practices documented better assessment of patients at referral.
There was a lack of evidence that either intervention increased the proportion of patients with organic pathology among those referred. The interactive software did improve the amount of information relayed in referral letters although we were unable to confirm if this made a significant difference to patients or their health care providers. The potential value of either intervention may have been diminished by their limited uptake within the context of a cluster randomised clinical trial. A number of lessons were learned in this trial of novel innovations.
Few patients with diarrhoea or rectal bleeding consult a general practitioner and fewer still need referral for specialist advice . In the UK guidelines describe patients who need to be referred urgently . Here general practitioners are audited on their efforts to identify such cases with reference to guidelines published by the Department of Health . It has been reported that over ninety percent of patients with colorectal cancer satisfy the published criteria for urgent referral and yet most patients who merit an urgent referral do not have cancer . It has also been suggested that delayed diagnosis may result in part for a failure to elicit the relevant history or perform the pertinent examination of symptomatic patients . Therefore the detailed assessment of patients with colorectal symptoms is considered to be vital when choosing which patients to refer in order to achieve a timely diagnosis.
We report a trial in which an electronic interactive referral pro forma prompting the assessment of patients with lower bowel symptoms was introduced in general practice as part of the workflow. It was anticipated that practitioners would learn which signs and symptoms were important and which patients required urgent referral. Adult learning theories predict that when practitioners are offered guidance at the time of making a decision, 'learning' or a 'relatively permanent change to the frequency of actions brought about by instruction or reinforced practice' will take place [6, 7]. Empirical evidence from Patel and Kaufman indicate that the integration of software systems into clinical settings fundamentally change not only how physicians view their daily work practice but also the very process of medical reasoning itself . Mugford and colleagues identified 36 published studies of interventions of feedback of information, concluding that information feedback was most likely to influence clinical practice if the information was presented close to the time of decision-making and the clinicians had previously agreed to review their practice .
Data from the US suggests that in-office education of primary care physicians may be effective in improving awareness of significant clinical features . Similarly a Cochrane review concluded that educational outreach visits, may be a promising approach to modifying health professional behaviour . Presentations by invited clinical experts are often feature in educational events aimed at general practitioners. Thus we also included a visit by a colorectal surgeon as part of an educational outreach programme as a second intervention. The research question is whether the introduction of an electronic interactive referral pro forma or educational outreach visits by a colorectal surgeon to general practice can alter the case mix of patients referred to lower bowel specialists.
Practices were recruited from Doncaster and Sheffield. Ethical review was by the North Sheffield Research Ethics Committee. Referrals to colorectal surgeons in this region were not required on any existing pro forma. A total of 150 practices were offered the opportunity to participate and 44 practices were recruited (29%) from August to December 2003 (see Additional File 1).
Development of software intervention
Educational outreach (EO)
A colorectal surgeon delivered short educational sessions. The design of the package was modified following pilot presentations to small numbers of practitioners and subsequently tailored to the needs of the target audience as expressed in the feedback to the team. The outreach visits were designed to reflect the model for content developed by Soumerai and Avron and incorporated general principles of education and behaviour change in the training sessions . During the 45 minute EO meeting, the presenter summarised the features of significant organic colorectal disease and encouraged questions. The published guidelines and the potential for the improvement to the management of patients were emphasised . This intervention was delivered from December 2003 to March 2004.
Design and sample size calculations
The design is a 2 × 2 factorial design, clustered by practice. The fours arms are: no educational outreach or software; educational outreach only; interactive pro forma only; educational outreach and interactive pro forma. This design enables one to estimate the effect of the software independently of the effect of the educational outreach. We assume a priori that there is no interaction between the two interventions, which is reasonable since they have different modalities. Having consented to take part, the practices were randomised to one of the four groups, stratified by median practice size. A computer program was used to allocate practices at random, using a fixed block size to ensure approximately equal numbers in the four arms. A pilot study suggested that the proportion of subjects referred by GPs who had 'significant' pathology was approximately 0.14 and these values are assumed for the 'no educational outreach-no software' group of the proposed trial. These pathologies were listed by consensus among local specialists and included: cancer; large polyps (≥3 cm); moderate to severe diverticular disease, and inflammatory bowel disease. We anticipated that the intervention would be valuable if it increased the proportion to 0.25 (i.e. a relative risk of about 2). A non-clustered trial would require about 630 referrals to achieve 80% power at two-sided 5% significance. The interclass correlation coefficient from previous data was calculated as 0.03 . On average, each GP refers six patients per year. Therefore the Design Effect is 1.15 and the required number of referrals is 725. With approximately three GPs per practice, this suggests that around 40 practices were required.
Clinical data that should be relayed to the specialist.
Duration of symptoms
History of change in bowel habit
History of rectal bleeding
History of tenesmus
History of passing mucous per rectum
History of abdominal pain
History of weight loss or patient's weight.
History of peri-anal symptoms e.g. itch or pain
Rectal mass or results of rectal examination
Abdominal mass or results of abdominal examination
Iron deficiency anaemia or results of full blood count
History of inflammatory bowel disease
Relevant family history
History of lower bowel investigations or existing colorectal conditions
General practitioner's opinion as to likely diagnosis
The outcome measures were: the proportion of cases with significant colorectal organic pathology referred in each arm of the study, an 'Assessment Score' of the quality of the referral letters reflecting the number of signs, symptoms and other data recorded in the referral letters and the proportion of cases from each arm referred on the appropriate pathway. The relative risk is an appropriate outcome in prospective studies, and we used a method described recently by Zou based on a modified Poisson regression, which also allowed for clustering of the data . The 'Assessment Score' was found to be skewed. Therefore a square root transformation was used to give a more symmetrical distribution. The data were collected at the hospitals receiving the referrals.
Six months after the referral pro forma was introduced practitioners using the software were asked to complete a questionnaire relating to their use of the software. We then conducted a series of in depth interviews on selected GPs on the basis of age; gender and self reported use of software. The practitioners were aged from 30 to 60 years. Interviews were audiotaped and transcribed. The transcripts from the groups were analysed by two researchers working independently. Analysis was by the framework method drawing on a priori issues and questions derived from the aims of the study as well as issues raised by the respondents themselves and views and experiences that recurred in the data . Sampling was terminated once saturation of themes was considered to have been achieved. The analysis followed the prescribed steps, including: familiarisation, indexing, charting, mapping and interpretation. Following analysis interviewees were asked to review a summary of the themes and to offer any further ideas on reflection. We present key themes only in this report.
Practices by intervention group
EO & Software
List size (mean)
Mean WTE GPs
Uptake of the interventions
Proportion of cases with significant pathology referred by each arm of the trial
Relative Risk (95%CI)* P-value
0.73 (0.46 to 1.15) P = 0.18
0.79 (0.50 to 1.24) P = 0.30
Assessment score of letters for patients referred by each arm of the trial (square root)
Mean difference (95% CI)
0.3 (0.17 to 0.42)
0.08 (-.04 to 0.2)
We recorded a 60% response rate to the questionnaires about the use of the software. Eight practitioners were interviewed about their use of the software before saturation of themes was achieved. Two key themes were identified relating to poor adoption of the software in the interviews with users these were:
i) Duplication of effort
Typically one GP expressed concern that the pro forma created an additional task in the process:
"I suspect there are a lot of GPs ... that are not interested in that they just want to make the referral. It is very much quicker for me to dictate a letter to my secretary and send it – a lot less effort. GRAF is installed on all the computers here and I guess there are only two of us that use it on a regular basis."
ii) Doubts about the software
The final question in the pro forma invites the GP to say if he/she thinks the patients' symptoms are those of cancer. Three choices were offered on the drop down menu: 'yes', 'no' and 'don't know'. If the practitioner answered 'yes' the referral was processed as an urgent referral. If they answered 'no' and the clinical details, with reference to the guidelines suggested otherwise, the program warned the practitioner that they had chosen to over rule guideline recommendations. The software was programmed to respond to the answer 'don't know' by recommending the most appropriate pathway based on the clinical details entered with reference to the published guidelines. Commenting on this function one respondent expressed a frequently voiced misgiving about GRAF.
"There's a lot of people that you don't think have got cancer, but ... you're not going to be sure. You don't know they definitely haven't got cancer, and that's partly why you refer most people who've got bowel symptoms, especially, you know, there's a bit of you might be worried that there may be a malignancy. So quite often you put the "Don't know", and if they all then become two-week wait (i.e. urgent referrals), then I think a lot of people are getting two-week waits that probably shouldn't get two-week waits. I don't want it (the hospital clinic) to sort of get overloaded with urgent two-week appointments that aren't really appropriate."
We are disappointed to report little or no benefit from the outreach visits or from the adoption of the pro forma software, which was limited to one in five referrals from the software practices. The purpose of the electronic innovation was to increase consultation of guidelines. The design of the pro forma as a workflow innovation was to enable practitioners to complete a task, namely write a referral letter. The data suggests there was minimal enthusiasm for the letter writing facility offered in the program. A previous study similarly concluded that despite their apparent benefits computerised referral pro forma systems are unpopular on the face of it because they transferred the administrative burden of the referral process from clerks and secretaries to clinicians . Similarly experience with web-based programmes in primary care has been disappointing . It is possible, perhaps probable, that we did not offer interventions that were acceptable in practice rather than cast doubt on the potential value of electronic referral pro formas and outreach visits per se. The fact that a reasonable number of practices were recruited but failed to participate echoes the experience of others who similarly reported disappointing results from studies in which practitioners remained ambivalent toward the proposed innovations . We suspect however that more valuable lessons can be learned from this negative experiment in addition to the difficulty of conducting large clinical trials in general practice.
Firstly our electronic innovation and to some extent the outreach visits were aimed at influencing medical decision making by focusing on the task of documenting and relaying medical information, that is on the medical record. We hypothesised, with reference to Adult learning theory that as clinicians were led to be more comprehensive in the data relayed to specialists that their clinical acumen would increase. We sought evidence for this from the greater number of referrals of patients with 'significant' bowel pathology. It is possible that the designation of 'significant' pathologies, those that were thought to be the root cause of the symptoms, including severe diverticular disease, identified the 'wrong' group of disorders and so we were unable to detect any 'significant' improvement in diagnostic acumen. In addition applying this innovation to other professional groups, for example practice nurses or involving the patient directly in documenting the clinical details may have yielded different results. We believe the software did identify the patients who met the referral criteria for urgent referral insofar as the software was programmed with the relevant algorithms. However this presupposes that the clinician systematically elicited the relevant clinical features for all symptomatic patients referred or otherwise.
Secondly we did not formally evaluate whether the educational visits resulted in any improvement in knowledge or change in practitioner attitude and so could not confidently confirm that the visits were effective. Thirdly the literature on the purpose of medical records suggests that the document genres evolve as their purpose and content evolves . The purpose of the 'referral letter', at the time of the project and as stated in the interviews was to seek a specialist opinion, not as our innovation was fashioned as a prompt for the detailed clinical assessment of symptomatic patients and a way to consult guidelines. The role of the referral letter did not 'evolve' to this purpose.
Finally it was apparent in this trial that general practitioners were uncomfortable with challenges to their clinical judgement. It is such attitudes that might hold the key to the implementation of guidelines more than, or in addition to, a technical fix-it or educational exchange. A key concept in the relation to the adoption of complex technologies is that they involve groups and networks of people . Technology both supports and shapes social practices such as sending a patient for a specialist opinion. In such communities, knowledge " rides along the rails laid by shared practice" . Where practices are canonical (agreed, shared) knowledge flows readily; where practices are noncanonical (innovative, challenging), knowledge may 'stick' rather than flow . It was apparent in our data that general practitioners sometimes had very different ideas about what constitutes an 'urgent' referral and that this further cast doubt on the value of the software for them. The result was a rejection of the innovation and by default the guidelines.
Along with other negative IT projects we did not take account of how our innovation was to be used in practice and perhaps more importantly how its use might threaten established professional identities and work practices . We need a clearer understanding of what clinicians do in relation to specialist referrals and at the 'macro' level what organisational routines, scripts and structures support them. This trial ignored the 'messiness' and ad-hoc nature of clinical work. We support those who argue that this is not a minor issue, which will be tidied up if only clinicians learn to adopt electronic innovations more consistently and rationally. We echo Greenhalgh and colleagues in their plea for further research into the spread and sustainability of technology based innovations in health care . We also recommend that any future trial of this approach in general practice is more closely integrated with the electronic health record.
We would like to thank the general practitioners who participated in this trial. With special thanks to the many colleagues and referees who reviewed the manuscript and offered cogent and constructive criticism. Funded by the Fred and Ann Green Legacy Mexborough.
- Thompson JA, Pond CL, Ellis BG, Beach A, Thompson MR: Rectal bleeding in general and hospital practice: "the tip of the iceberg. Colorect Dis. 2000, 2: 288-93. 10.1046/j.1463-1318.2000.00141.x.View ArticleGoogle Scholar
- Department of Health: The NHS Cancer Plan: A Plan for Investment. A Plan for Reform. 2000, London: HMSOGoogle Scholar
- Department of Health: Referral Guidelines for Suspected Cancer. 2000, London: Department of HealthGoogle Scholar
- Chohan DPK, Goodwin K, Wilkinson S, Miller R, Hall NR: How had the 'two-week' rule affected the presentation of colorectal cancer?. Colorectal Disease. 2005, 7: 450-453. 10.1111/j.1463-1318.2005.00821.x.View ArticlePubMedGoogle Scholar
- Springall RG, Todd IP: General practitioner referral of patients with lower gastrointestinal symptoms. J of the Royal Soc of Med. 1988, 81: 87-8.Google Scholar
- Knowles M: Andragogy in Action. 1984, San Francisco: Jossey-BassGoogle Scholar
- Instructional systems glossary. (Accessed June 2005), [http://members.aol.com/JohnEshleman/glossary.html]
- Patel VL, Kaufman DR: Medical informatics and the science of cognition. Medical informatics and the science of cognition. J Am Med Inform Assoc. 1998, 5: 493-502.View ArticlePubMedPubMed CentralGoogle Scholar
- Mugford M, Banfield P, O'Hanlon M: Effects of feedback of information on clinical practice: a review. BMJ. 1991, 303: 398-402.View ArticlePubMedPubMed CentralGoogle Scholar
- Daly M, Balshem B, Sands M, James C, Workman J, Engstrom S, P F: Academic detailing: a model for in-office CME. Journal of Cancer Education. 1993, 8 (4): 273-80.View ArticlePubMedGoogle Scholar
- Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL: Educational outreach visits: effects on professional practice and health care outcomes. The Cochrane Database of Systematic Reviews: Reviews. 1997, John Wiley & Sons, Ltd Chichester, UK, 4Google Scholar
- Jiwa M, Mathers N, Walters S: The quality of referrals to colorectal surgery: towards consensus. Current Medical Research and Opinion. 2002, 18 (2): 72-7. 10.1185/030079902125000309.View ArticlePubMedGoogle Scholar
- Avron J, Soumerai SB: Improving drug-therapy decisions through educational outreach: a randomised controlled trial of academically based ' detailing'. NEJM. 1983, 308: 1457-1463.View ArticleGoogle Scholar
- Jiwa M, Walters S, Mathers N: Referral letters to colorectal surgeons – the impact of peer mediated feedback. British Journal of General Practice. 2004, 54: 123-126.PubMedPubMed CentralGoogle Scholar
- Zou GY: A modified Poisson regression approach to prospective studies with binary data. Am J Epidemiol. 2004, 159: 702-706. 10.1093/aje/kwh090.View ArticlePubMedGoogle Scholar
- Ritchie J, Spencer L: Qualitative data analysis for applied policy research. Analysing Qualitative data. Edited by: Bryman A, Burgess R. 1993, Routledge: LondonGoogle Scholar
- Sittig DF, Gandhi TK, Franklin M, Turetsky M, Sussman AJ, Fairchild DG, Bates DW, Komaroff AL, Teich JM: A computer-based outpatient referral system. Int J of Med Informatics. 1999, 2: 149-158. 10.1016/S1386-5056(99)00027-1.View ArticleGoogle Scholar
- Jeannot JG, Scherer F, Pittet V, Burnand B, Vader JP: Use of the World Wide Web to implement clinical practice guidelines: a feasibility study. J Med Internet Res. 2003, 5 (2): e12-10.2196/jmir.5.2.e12.View ArticlePubMedPubMed CentralGoogle Scholar
- Modell M, Khan M, Modell B: Why do some practices respond to an educational intervention?. BJGP. 2000, 50: 826-827.PubMedPubMed CentralGoogle Scholar
- Østerlund C: Combining genres: How practice matters. 2006, Hawaii: HICSS, 50a-Google Scholar
- Østerlund C, Carlile P: Relations in practice: Sorting through practice theories on knowledge sharing in complex organizations. The Information Society. 2005, 21: 91-107. 10.1080/01972240590925294.View ArticleGoogle Scholar
- Brown JS, Duguid P: Knowledge and organization: A social practice perspective. Organization Science. 2001, 12: 198-213. 10.1287/orsc.18.104.22.16816.View ArticleGoogle Scholar
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organisations: systematic literature review and recommendations for future research. Milbank Quarterly. 2004, 82: 581-629. 10.1111/j.0887-378X.2004.00325.x.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2296/7/65/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.