Maximising harm reduction in early specialty training for general practice: validation of a safety checklist

Background Making health care safer is a key policy priority worldwide. In specialty training, medical educators may unintentionally impact on patient safety e.g. through failures of supervision; providing limited feedback on performance; and letting poorly developed behaviours continue unchecked. Doctors-in-training are also known to be susceptible to medical error. Ensuring that all essential educational issues are addressed during training is problematic given the scale of the tasks to be undertaken. Human error and the reliability of local systems may increase the risk of safety-critical topics being inadequately covered. However adherence to a checklist reminder may improve the reliability of task delivery and maximise harm reduction. We aimed to prioritise the most safety-critical issues to be addressed in the first 12-weeks of specialty training in the general practice environment and validate a related checklist reminder. Methods We used mixed methods with different groups of GP educators (n = 127) and specialty trainees (n = 9) in two Scottish regions to prioritise, develop and validate checklist content. Generation and refinement of checklist themes and items were undertaken on an iterative basis using a range of methods including small group work in dedicated workshops; a modified-Delphi process; and telephone interviews. The relevance of potential checklist items was rated using a 4-point scale content validity index to inform final inclusion. Results 14 themes (e.g. prescribing safely; dealing with medical emergency; implications of poor record keeping; and effective & safe communication) and 47 related items (e.g. how to safety-net face-to-face or over the telephone; knowledge of practice systems for results handling; recognition of harm in children) were judged to be essential safety-critical educational issues to be covered. The mean content validity index ratio was 0.98. Conclusion A checklist was developed and validated for educational supervisors to assist in the reliable delivery of safety-critical educational issues in the opening 12-week period of training, and aligned with national curriculum competencies. The tool can also be adapted for use as a self-assessment instrument by trainees to guide patient safety-related learning needs. Dissemination and implementation of the checklist and self-rating scale are proceeding on a national, voluntary basis with plans to evaluate its feasibility and educational impact.


Background
Patients worldwide are unintentionally but avoidably harmed as a result of their interactions with health care [1]. A global campaign to highlight the patient safety problem and recommend potential solutions is being led by the World Health Organisation (WHO) [2]. In the United Kingdom (UK), making patient care safer has been an explicit policy priority in the National Health Service (NHS) for over a decade [3]. Until recently the predominant safety improvement focus was on acute hospital settings where there is well-established evidence of the scale and consequences of adverse health care events [4]. In UK general medical practice comparable research and knowledge of the nature and impact of avoidable harm is growing, but is methodologically limited compared with secondary care [5,6]. However, the emerging evidence from a range of international sources suggests that the safety of general practice may be compromised in a significant minority of cases [7][8][9]. Specialty trainees are known to be involved in a proportion of these incidents largely because of a range of systems, knowledge, cognitive, training and behaviour based reasons.
In the NHS, a raft of interventions designed to make patient care safer are being implemented or are under development. For example: multi-centre collaborative programmes of safety improvement that aim to reduce harm in specific areas of acute hospital care and, more recently, primary care are currently underway [10]; a high-profile and long-running campaign to minimize healthcare acquired infections is ongoing [11]; while a concerted attempt to improve the reporting and learning from patient safety incidents has also taken place [12].
From an educational perspective, it is now recognised that there is a need for patient safety education to be more explicitly integrated and prominently positioned within existing undergraduate and postgraduate training curricula for all health care professions [13]. The UK Royal College of General Practitioners (RCGP) -which has responsibility for the content of the specialty training curriculum -has responded by developing a curriculum statement on 'patient safety' [14] and defining specific learning objectives (Table 1). However, it is left to individual GP educational supervisors to determine how the RCGP curriculum is best delivered and the related learning needs of trainees are identified and acted upon during the training period. The role of the postgraduate deaneries in this regard is to verify that the evidence provided by training practices to support the delivery of the patient safety element of the curriculum is of an adequate standard.
Ensuring that all essential educational issues are identified, prioritised and satisfactorily covered during training is not straightforward given the complexity of the tasks to be undertaken and the high volume of topics to be addressed. Within the framework of the RCGP curriculum, GP educational supervisors currently guide the activities undertaken by the trainee using a combination of locally developed induction packs, nationally promoted learning interventions and assessments, and professional experience in the workplace to match across to RCGP curriculum competencies (Table 2). However, it is currently unclear how the patient safety-related learning needs of trainees are specifically addressed when in the training environment. Given the lack of standardized guidance on how and what specific learning issues are to be covered, it is possible there is variation in GP educational supervisors' interpretation and delivery of this safety-critical element of the curriculum at the 'sharp end' of frontline educational practice.
Overall there appears to be a paucity of evidence on how postgraduate training in general practice (or lack of ) may have a visible and negative impact on the safety of patientsarguably it may be impossible to ever demonstrate clear causation. However, we know that a range of issues connected with the postgraduate training

A Comprehensive Approach
(e.g. Describe the risks to patient safety by considering an illness pathway/journey in which a variety of healthcare professionals have been involved)

Community Orientation
(e.g. Describe how patient groups may be put at increased risk of mishap by virtue of their particular characteristics, such as language, literacy, culture and health beliefs)

A Holistic Approach
(e.g. Describe how the lessons of patient safety can be applied prospectively to doctor-patient interactions, especially through the identification and discussion of risk).

Contextual Aspects
(e.g. Describe the impact of the working environment on the care the doctor provides and the likelihood of adverse incidents as a result of this)

Attitudinal Aspects
(e.g. Help to shape an organisational culture that prioritises safety and quality through openness, honesty, shared learning and continual incremental improvement)

Scientific Aspects
(e.g. Describe the basic principles of risk assessment) *UK GP Specialty Trainees are required to spend 18 months in a GP setting as part of a 3 or 4 year programme. The teaching required is governed by the RCGP curriculum and one area that is increasingly being highlighted is UK general practice is patient safety. environment may act as proxy indicators of the safety of patient care being compromised unnecessarily and avoidably. For instance, medical educators may be involved in failures of, or inadequate, clinical supervision [15]; or fail to respond appropriately to trainees' seeking professional guidance [16]; or conduct insufficient joint reviews of the management of complex clinical cases [17]; or provide limited feedback on drug prescribing performance [18]; and may let poorly developed attitudes and behaviour (such as lack of insight) continue unchecked [19]. Other salient issues that are potentially safety-critical include trainees' possessing different levels of clinical knowledge [20] and an inability to prioritise their clinical workloads and manage time [17]. The quality of the learning environment in which trainees are based may also affect the safety of patient care [21], while doctors-in-training are known to be susceptible to medical errors [6,22].
Given what is known about human error theory in the healthcare workplace [23], and the marked differences in local safety cultures [24] and the reliability of practice systems [5][6][7][8][9], it is inevitable that variation in the quality of training provision exists and that some issues will be inadequately covered or even overlooked completely. If this happens with fundamental training topics which are considered to be safety-critical then there is a likelihood that the risk of patients being harmed and other quality of care issues arising could potentially increase.
Evidence is accumulating from healthcare safety and improvement initiatives that routine adherence to the adoption and use of checklist reminders can improve the overall reliability with which care processes and tasks

Practising holistically
(This competency is about the ability of the doctor to operate in physical, psychological, socio-economic and cultural dimensions, taking into account feelings as well as thoughts)

Data gathering and interpretation
(This competency is about the gathering and use of data for clinical judgement, the choice of examination and investigations and their interpretation)

Making a diagnosis/making decisions
(This competency is about a conscious, structured approach to decision-making)

Clinical management
(This competency is about the recognition and management of common medical conditions in primary care)

Managing medical complexity
(This competency is about aspects of care beyond managing straightforward problems, including the management of co-morbidity, uncertainty and risk, and the approach to health rather than just illness).

Primary care administration and information management & technology
(This competency is about the appropriate use of primary care administration systems, effective record keeping and information technology for the benefit of patient care)

Working with colleagues and in teams
(This competency is working effectively with other professionals to ensure patient care, including the sharing of information with colleagues)

Community orientation
(This competency is about the management of the health and social care of the practice population and local community)

Maintaining performance, learning and teaching
(This competency is about maintaining the performance and effective continuing professional development of oneself and others)

11.
Maintaining an ethical approach to practise (This competency is about practising ethically with integrity and a respect for diversity)

Fitness to practice
(This competency is about the doctor's awareness of when his/her own performance, conduct or health, or that of others might put patients at risk and the action taken to protect patients) are undertaken and so potentially mitigate future risks [25][26][27][28]. Against this background, we aimed to identify and prioritise the most safety-critical issues to be addressed by educational supervisors and trainees in the first 12-weeks of specialty training in the general practice environmentthe training period judged to be specifically high risk and beyond which trainees are given greater clinical freedom. In doing this we may help maximize early opportunities to address safety-critical issues proactively via a checklist reminder, which may lead to a reduced risk of patients being unintentionally harmed during and after the training period.

Methods
We used a six-stage mixed methods approach to develop and content validate the checklist for educational supervisors (Table 3).

Preliminary theme and item content generation
As a first step we held three 90-minute workshops in April 2010 with a total of 72 experienced GP educational supervisors (known previously as 'trainers') at the south east of Scotland annual GP training conference. They were asked to prioritise their perceptions of the six most critical threats to patient safety in the GP specialty training (GPST) environment (beyond specific clinical topics). In each workshop 4 or 5 groups of supervisors reflected on this task for around 45 minutes, provided feedback on their findings (via a flip chart) and engaged in open discussion about how this information could be of value in the training environment. The workshop facilitator (PB) made contemporaneous field notes and retained all the flip chart data with consent from participants. The authors then independently reviewed each chart and coded the text before generating a preliminary list of safety-critical themes and items (specific educational and training issues) of relevance to early GPST. Differences in theme formulation and item nesting were jointly negotiated until agreement was reached. We then identified and examined a small number of published reviews and grey literature on the key threats to patient safety in primary care internationally [5][6][7][8][9][29][30][31][32][33]. The information collated was evaluated by the authors and a small number of safety-critical themes and related items of specific relevance to GPST were highlighted. We augmented our aforementioned list with these data to generate an updated preliminary checklist consisting of 15 safety-critical themes and 67 related items.
We invited and recruited a convenience sample of six highly experienced GP specialty training programme directors and course organisers based in the west of Scotland to participate in a modified-Delphi process (three rounds by electronic mail) to gain consensus on the checklist domains and items that were judged to be 'essential' to the first 12-weeks of specialty training in the general practice setting. This further iterative refinement resulted in the checklist being reduced to 14 safety-critical themes with 55 related items.

Further input and validation from educational supervisors and specialty trainees
We invited a random sample of 25 West of Scotland regional GP educational supervisors by electronic mail to reflect on the relevance of each of the 55 items to the first 12-weeks of training in the GP environment and then attend and participate in a follow-up 3-hours workshop to further review and endorse the checklist content. A total of 21/25 (84%) completed both tasks leading to further refinement and updating of content themes (n = 14) and items (n = 47). Discussion with this group also resulted in the proposed final checklist being adapted as a self-assessment tool for specialty trainees.
To further triangulate our findings, we sent the checklist content for review to nine volunteer specialty trainees who had recently completed their initial training period in the GP environment. We held a follow-up 2-hour workshop (n = 3) and telephone discussions (n = 6) with the trainees who further validated the content. Additional issues were raised by them which were judged by the authors to be relevant to the quality of the learning environment trainees were based in, rather than being deemed essential safety-critical issues in the first 12-weeks of training.

Content validity index (CVI) exercise
Finally, a further 24 GP educational supervisors from a single geographical district in the west of Scotland undertook a CVI exercise. The relevance of each retained checklist item was assessed by asking the supervisors to rate them using a validated [34] 4-point scale: (1 = not relevant, 2 = somewhat relevant, 3-quite relevant and 4 = very relevant). Twenty out of 24 supervisors were required to endorse each item by assigning a rating of at least 3 out of 4, to establish content validity beyond the 0.05 level of significance. This was determined to provide sufficient evidence for inclusion of each item as part of the final checklist. Supervisors were also asked to identify any missing items that they deemed important for inclusion, but after reflective consideration and open discussions no more items were added, deleted or amended.

Ethical review
The study proposal was pre-screened by the west of Scotland research ethics committee but judged to be a service development not requiring ethical approval.  2. Controlled Drugs (e.g. knowledge of storage, dose adjustment, prescription format) [5,12] 3. Awareness of Health Board/Formulary Prescribing Guidance [9] 4. Knowledge of practice repeat prescribing system [7] 5. Risks associated with signing repeat & special requests without consulting records [5,6] 6. Monitoring drug side-effects (e.g. Myalgia with Statins) [5,6] DEALING WITH MEDICAL EMERGENCY 13. Knowledge of practice system for results handling [7] PATIENT REFERRALS 14. Identifying the need for referral (i.e. recognition of condition requiring further investigation and/or treatment) [3] 15. Referral system (e.g. how and when to refer 'urgently' and 'routinely' [7,9] 16. Clinical appropriateness of referral (e.g. ensure correct clinical priority and correct specialty) [9] 17. Quality of acute referral letter (e.g. past medical history, medication status, social circumstances) [7] EFFECTIVE & SAFE COMMUNICATION 18. Knowledge of internal communication processes within the practice (e.g. e-mail, message systems, practice meetings. . .) [7] 19. How to liaise with and understand the roles of team members: who, purpose, how, where, when? [8] 20. Safe communication with patients and relatives (e.g. consultations, phone calls and letters). [4] CONSULTING SAFELY 21. How to safety-net (face-to-face) [1] 22. How to safety-net (when providing telephone advice) [1] 23. Awareness of guidelines for use of Chaperones [11,12] ENSURING CONFIDENTIALITY 24. Avoiding breaches of confidentiality [11] 25. Appropriate disclosure of medical and personal information [11] AWARENESS OF THE IMPLICATIONS OF POOR RECORD KEEPING 26. Failing to keep records [12] 27. Failing to keep accurate records [12] 28. Failing to confirm patient identify [12] 29. Failing to document all patient contacts [12] 30. Knowledge of related legal issues [12] RAISING AWARENESS OF PERSONAL RESPONSIBILITY 31. Awareness of professional accountability [12] 32. Recognising the limits of own clinical competence [12]

Results
The study identified 14 safety-critical domains and 47 related items that were judged to be essential components of specialty training in general practice to be addressed by educational supervisors and trainees in the first 12-week period of training in the general practice setting. Each item was also aligned with one or more of the 12 RCGP curriculum competencies to assist and guide supervisors in delivering the curriculum and collecting supporting evidence (Table 4). All checklist items were endorsed by a minimum of 20 of 24 GP educational supervisors who rated each item ≥3 on the 4point scale. The overall content validity index ratio for the tool was 0.98 (Table 5).

Discussion
We identified and prioritised a whole range of safetycritical issues that were judged to be 'essential' as part of the high risk 12-week phase of specialty training in the general practice environment. We demonstrated that the content validity of the checklist tool is adequate, providing the first steps towards validating this approach as a method for educational supervisors to guide and support safety-critical education interventions at this phase of training and potentially in future training periods (Additional file 1). At the suggestion of supervisors, the checklist was also adapted to enable trainees to selfassess safety-related learning needs early in their specialty training and monitor future progress (Additional file 2). At a fundamental level, the domains and related items generated in this study have the potential to be used by GP Educational Supervisors and General Practice Managers to modernize existing Induction Packs, or help inform the development of new induction processes. More specifically, the tool can be applied by supervisors as a checklist prompt or reminder to ensure that the most safety-critical educational tasks are actually carried out and are done so timeously, efficiently, and without ambiguity. The potential to improve the reliability of educational provision in these important areas of specialty training and health care practice should be evident. Completion of the checklist could also be used as supporting evidence to the postgraduate deanery (e.g. during training accreditation visits) that the core patient safety element of the training curriculum is being proactively considered and delivered.
All of these factors are of prime importance because of the potential medico-legal implications for the GP educational supervisor if a trainee is inadequately tutored and supervised and patient safety is subsequently compromised. GP educators need to have mechanisms in place to make an early assessment of a trainee's competencies and to undertake regular performance reviews and feedback sessions as part of an overall strategy to minimise risks. It should be noted that as trainees' Table 4 Validated safety checklist themes and related items mapped against 12 RCGP curriculum competencies (Continued) 33. How and when to seek help [12] 34. Personal organisation and effectiveness [12] DEALING WITH CHILD PROTECTION ISSUES 35. Recognition of harm and the potential for harm in children [2] 36. How to liaise with other agencies [8] 37. Breaching confidentiality [11,12] ENHANCING PERSONAL SAFETY 38. How to access emergency alarms/panic button for personal safety [12] 39. Dealing with aggressive & violent patients [12] 40. Ensuring personal safety and security on home visits [12] EMPHASISNG THE IMPORTANCE OF THE LEARNING ENVIRONMENT 41. Ensure rapid access to supervisory advice, feedback and support [10] 42. Raise awareness of practice team contribution and support [10] 43. Ensure reflective learning recorded in E-Portfolio [10] 44. Knowledge of clinical audit and significant event analysis [10] SAFE USE OF PRACTICE COMPUTERISED SYSTEMS 45. Ensure proficiency in using practice computer system [7] 46. How to prioritise computer system safety alerts (e.g. Yellow and Red Traffic lights) [7] 47. The need to avoid common pitfalls (e.g. leaving notes open and writing up the wrong patient) [7] progress through training they are exposed to a more 'positive' and less 'reductionist' view of care quality in general practice via engagement with a range of improvement methods and exposure to the Quality & Outcomes Framework as part of routine clinical practice. Theory suggests that because human memory and attention are imperfect [24], one way to mitigate errors or lapses in the execution of tasks is through the use of procedural checklists to help us perform more reliably. Compliance checklists to improve the safety or reliability of processes are standard practice in highreliability industries [35,36], most notably in commercial aviation and petro-chemical organisations. In health care checklist use is well documented, particularly in surgical [37], obstetric [38] and intensive care [39] settings where their implementation is associated with improved clinical outcomes, team working and productivity efficiencies.
However, despite these positive reports there is some scepticism over whether the introduction of a checklist on its own is the single greatest factor attributable to improvement success. Bosk et al. [40] suggest that the widespread deployment of a checklist may not be successful without a clear understanding that it is a 'technical' solution being introduced to a complex dynamic working environment where pre-existing 'socio-cultural' issues may need to be contained or resolved. Clinicians may resist or feel threatened by checklist implementation because they perceive it to impinge on their expertise, interfere with their decision-making or over-simplify the working environment. We also know that there is variation in perceptions of local safety climate within and between general practice teams and that some may unintentionally inflate this measure because of inexperience with, or lack of knowledge of, local safety concerns [41]. Similar variation exists in the strength of team working [42], the maturity of the learning environment [21] and commitment to quality improvement [43]. Attending to these types of socio-cultural issues is, therefore, as equally important as supporting efforts into checklist development and implementation if momentum is to be gained and a successful impact made. Adaptation of the checklist as a self-assessment tool for trainees was an unexpected study development. Selfassessment is a method of measuring and interpreting one's own performance and is a well-established educational intervention amongst all health care professions and as part of specialty training [44]. The use of selfassessment tools by specialty trainees to identify learning needs is routine in medicine [45,46], but does not appear to be well developed in terms of highlighting specific educational interventions related to patient safety. The activity takes on a greater significance when the performance focus is related to the identification of learning need associated with safety-critical education and training. Although the limitations of self-assessmentis well established [44], it is still promoted as a valuable educational activity because it can be used as a baseline measure which prompts joint discussion and monitoring of performance over time by the trainee and supervisor.

Study strengths and limitations
A key strength is that the study idea and subsequent tool development were driven by the identification of a need for a patient safety education intervention by experienced frontline medical educators. We used mixed methods to explore and capture in-depth qualitative feedback and quantitative measures to further confirm validation of tool content which strengthened the rigour of this methodological process. For instance, the final CVI ratio was very high which may be a reflection of the deep consideration of checklist items by a large number of informed educators.
Limitations included the potential for bias through the pragmatic use of convenience samples of volunteer participants. Issues around multi-ethnicity were not raised by our participants but we recognise that in other areas of the UK and beyond, particularly in larger cities, this may be recognised as a potential safety-critical problem. Demographic data on participants was not collected as we judged that this would not necessarily add value to study findings and may hinder recruitment. Our key interest was in tapping into trainers' knowledge, experiences and perceptions of the safety-critical elements of training.
We only recruited a small number of trainees to validate both the checklist and self-rating scale (which they would use) as it was important to triangulate our findings and get their perspective on safety-critical issues. This was useful in terms of confirming rather than adding to our findings, but exploring these issues with more trainees may have resulted in deeper insights.
Although content validation processes were robust, the tools have yet to be adequately tested and evaluated for evidence of acceptability, feasibility and educational gain. Determining how the successful implementation and wide-spread use of the checklist would directly impact on safety outcomes may not be possible, which means we may need to define proxy measures of success.