Accuracy of telephone triage in suspected transient ischaemic attack or stroke: A cross-sectional study

Background The Netherlands Triage Standard (NTS) is a widely used decision support tool for telephone triage at Dutch out-of-hours primary care services (OHS-PC), which, however, has never been validated against clinical outcomes. We aimed to determine the accuracy of the NTS urgency allocation for patients with neurological symptoms suggestive of a transient ischaemic attack (TIA) or stroke, with the clinical outcomes TIA, stroke, and other (neurologic) life-threatening events (LTEs) as the reference. Method A cross-sectional study of telephone triage recordings of patients with neurological symptoms calling the OHS-PC between 2014 and 2016.The allocated NTS urgencies were derived from the electronic medical records of the OHS-PC. The clinical outcomes were retrieved from the electronic medical records of the patients’ own general practitioners. The accuracy of a high NTS urgency allocation (medical help within three hours) was calculated in terms of sensitivity, specicity, positive and negative predictive values (PPV and NPV) with the clinical outcomes TIA/stroke/other LTEs as the reference.


Background
Prompt recognition of patients with a transient ischaemic attack (TIA) or ischaemic stroke is crucial for timely initiation of therapeutic interventions to minimise the risk of (permanent) brain injury and recurrent stroke. (1)(2)(3)(4)(5)(6) Previous studies showed that urgent diagnostic assessment of TIA and minor stroke patients followed by a timely start of stroke preventive treatment resulted in a tremendous decrease of the early stroke risk (1,5,7) with a reduction of recurrent stroke up to 80% within three months.(1) However, the detection of TIA, and to a lesser extent stroke, may be challenging because multiple other diseases like migraine with aura, seizures or syncope can mimic TIA or stroke.(8-10) Moreover, symptoms may be non-speci c in TIA or stroke, notably vertebrobasilar insu ciency, and in the case of TIA, symptoms are often short lasting and already resolved by the time a patient seeks medical help. (8,11) Patients with symptoms suggestive of TIA or stroke often contact the general practitioner (GP) rst. (12)(13)(14)(15) During evenings, nights and weekends such care is provided by the out-of-hours services in primary care (OHS-PC). At the OHS-PC, the initial contact is by telephone, and nurses perform triage while supervised by GPs. (16) The goal of telephone triage is to assess the severity of patients' complaints and to link this to an adequate urgency allocation with corresponding response time to medical care. Telephone triage in the Netherlands is supported by a semi-automatic decision support tool called the 'Netherlands Triage Standard' (NTS). The NTS is a ve-level triage tool, which was developed by an expert panel and derived from existing Dutch national telephone guidelines for primary care o ce hours, and the Manchester Triage System (MTS).(17, 18) Based on the annual incidence of 0.006% of serious adverse events (SAEs) in the Dutch OHS-PC setting, the NTS is considered to be safe (19). However, questions have been raised about the e ciency. (16) There was a clear increase in high urgency allocations since the implementation of the NTS in 2011 onwards, suggesting a low e ciency. (20) This was supported by the results of a national survey among GPs in 2016, showing that the vast majority believed telephone triage with the NTS resulted in unnecessary consultations and home visits. (16,21) Most previous studies assessed the overall accuracy of triage decision support tools in emergency department (ED) settings, and to a lesser extent at the OHS-PC. (22,23) Few studies focused on speci c domains of patients (e.g. chest pain), some of which included clinical outcomes as the reference (e.g. acute coronary syndrome), yet, only in ED settings. (24)(25)(26)(27)(28)(29)(30)(31) Comparable accuracy studies in primary care settings are limited; one study that assessed the overall accuracy of a telephone triage tool in primary care used a 'surrogate' reference created by the researchers themselves (e.g. hospital referrals or costs). (18,22,(32)(33)(34) The NTS urgency allocation, or the urgency allocation of other decision support tools for telephone triage in primary care settings, were never evaluated against the nal clinical outcomes of patients as the reference.
We aimed to determine the accuracy of the NTS urgency allocation in patients calling the OHS-PC with symptoms suggestive of TIA or stroke, with presence or absence of the nal clinical outcomes TIA, stroke and other (neurologic) life-threatening events (LTEs) as the reference.

Methods
Design and setting We conducted a cross-sectional study in which we analysed real-life telephone triage recordings of nine OHS-PC locations in the vicinity of Utrecht, the Netherlands between 2014-2016. These OHS-PCs provide out-of-hours primary care for approximately 1,5 million people, handling 400,000 triage calls per year.

Data collection
We evaluated patients with symptoms suggestive of TIA or stroke. The accuracy of NTS urgency allocation was assessed with the nal clinical outcomes as the reference, that is, TIA, stroke and other (neurologic) life-threatening events (LTEs), e.g. subarachnoid haemorrhage. The triage recordings were selected in a two-step inclusion procedure, i.e. (i) selection based on the International Classi cation of Primary Care (ICPC) codes that are linked to the call and re ected our study domain (i.e. K89, K90, N17, N18, N19, N29, N89, N91), and (ii) keywords in the OHS-PC electronic medical records suggesting TIA/stroke (e.g. neurological de cit, arm or leg weakness, face drooping, communication problem, visual problem, sensory disturbances and common synonyms). (35) A detailed description of the ICPC codes, medical keywords, inclusion and exclusion criteria has been published elsewhere. (36) We selected and relistened a random sample of 2,209 calls. Patient and call characteristics, and assigned NTS urgencies were collected. From the patients' own GPs we retrieved the nal diagnosis, which was based on the discharge letter from the neurologist or the ED if the patient was referred for additional investigations. For patients who were not referred to the hospital we used follow-up data from the electronic medical records of GPs for up to one month to capture possible recurrence of TIA/stroke. NTS urgency allocation in day-to-day practice Telephone triage with the NTS starts with a mandatory 'ABCD' check (i.e. airway, breathing, circulation, disability). In case of direct life-threatening situations, an ambulance will be sent immediately. (37) If there is no life-threatening situation, the triage nurse continues by choosing one out of the 56 main complaints within the NTS. Every main complaint consists of an algorithm composed of hierarchically ordered questions. (18). One of these 56 main complaints is 'neurological de cit'. After lling out the patient's responses, the NTS will automatically generate an urgency level ranging from U0 to U5 which is linked to the response time within which a patient should receive medical help (see Table 1). (18,38) The NTS urgency may be scaled up or down by the triage nurse, often after rst consulting the supervising GP. (21) The reason for overruling should be registered, but this is not a mandatory step to complete the NTS triage process.

Difference between NTS urgency and nal urgency
Besides the NTS urgency, which is automatically generated, we also evaluated the nal urgency, which was de ned as either the NTS urgency (if not changed) or the overruled NTS urgency.
In around 20% of all triage calls, the nal urgency was unclear after re-listening the recordings in which it was evident that the triage nurse overruled the NTS urgency. This because the triage nurse did not notify the actual allocated urgency after overruling the NTS; e.g. the NTS urgency was U3, but in the audio recording the triage nurse tells the caller "I will sent an ambulance immediately" (U1)). Nevertheless, the urgency in the NTS system remained U3. A panel of three experienced GPs assessed calls in which the nal urgency was unclear, blinded to the nal diagnosis, and determined the nal urgency (unanimously, or majority of votes after group discussion).

Data analyses
The patients were dichotomised into a high (U1 and U2) and low (U3, U4 and U5) urgency group, and differences in characteristics between these groups were compared. We calculated the accuracy in terms of sensitivity, speci city, positive and negative predictive values of (i) the NTS urgency allocation and (ii) the nal urgency allocation (including overruled NTS urgencies), with the clinical outcomes TIA/stroke/LTEs as the reference. For the accuracy calculations we considered for TIA/minor stroke case the urgencies U1, U2 and U3 as adequate, and for major stroke and other LTEs the urgencies U1 and U2. Finally, we compared the baseline characteristics of patients in whom we could retrieve the nal diagnosis with those in whom we could not, to assess potential selection bias. Statistical analyses were performed using SPSS version 25.0 (IBM Corp., Armonk, NY, USA).

Group characteristics
We included 1,269 patients of whom a nal diagnosis could be obtained (see Figure 1). The median age was 72.0 (IQR 57.0-83.0) years, and 56.9% were female. The NTS allocation of high (U1 and U2) and low (U3, U4 and U5) urgencies was equally distributed between men and women (see Table 2).
The characteristics of patients with a known nal diagnosis were comparable with those for whom the GP did not provide the nal diagnosis (see Supplementary data Table S1).

Final diagnoses
In 434 (34.2%) patients the nal diagnosis was a TIA or minor stroke, and in 201 (15.8%) a major stroke.
Sixty-one (4.8%) patients had other LTEs, i.e. intracerebral haemorrhage or subarachnoid haemorrhage. The remaining 573 patients (45.2%) were diagnosed with other neurological disorders (e.g. migraine, epilepsy) or other disorders (e.g. peripheral vestibular syndromes or psychogenic syndromes). See Table   3 for a complete overview of nal diagnoses.  patients had other (neurologic) LTEs. The urgency allocation of the NTS tool was poor to moderate regarding sensitivity and speci city with TIA/stroke/other LTEs as the reference. In 42.6% the NTS urgency was overruled by the triage nurse. The nal urgency allocation (including overruled NTS urgencies) showed modestly improved sensitivity (safety) whereas the speci city remained equally poor (e ciency). The positive predictive value did not change after overruling of the NTS, but the negative predictive value increased. This suggests that overruling by the triage nurses leads to safer telephone triage without compromising e ciency (i.e. overlapping con dence intervals of the NTS and nal urgencies' speci cities).

Strengths and limitations
This is the rst study to report accuracy ndings of the NTS tool for telephone triage at the OHS-PC with clinical outcomes as the reference. Because researchers were blinded to the nal clinical outcome during data collection, the effect of hindsight bias was limited.
A limitation was missing data on the nal clinical outcome (25% of all re-listened recordings). However, a detailed comparison in patient characteristics between those with a nal outcome and those without showed that these groups were comparable (no indication of selection bias). Therefore, we believe our results are generalizable to similar OHS-PC settings.
Comparison with existing literature As described previously, many studies assessed the accuracy of other triage systems (22), and some of these also used clinical outcomes as the reference. (24)(25)(26)(27)(28)(29)(30)(31) One study assessed the Manchester Triage System (MTS) in the domain of patients suspected for neurological disease seen at the ED.(32) The accuracy of a high urgency allocation was calculated with neurological disease (not otherwise speci ed) as the reference; a c-statistic of 0.73 was reported. High MTS urgency allocation was signi cantly associated with neurological disease (odds ratio 3.0, 95%CI 2.4-3.8, p<0.001).(32) Unfortunately, sensitivity or speci city was not calculated. Comparison to our study is also hampered, because in the primary care setting the prevalence of emergent cerebrovascular events is lower, and on average includes less severe cases. This may be re ected in less evident clinical presentations.
In our study, we considered different urgency levels as adequate; for TIA/minor stroke U1-U3, and for major stroke/other LTEs U1-U2. The rationale for high urgency allocations in suspected stroke patients is mainly because of available treatment options, and not because TIA/stroke may result in ABCD instability (i.e. airway, breathing, circulation, disability). Assigning high urgency levels to patients with acute stroke enables early initiation of (invasive) prognostically bene cial treatment. (39)(40)(41) In patients with TIA/minor stroke early initiation of antiplatelets for secondary stroke prevention is key, given the substantial risk of major stroke in the rst hours to days after a TIA (5,6,42). Current treatment guidelines on TIA/stroke recommend that patients suspected of TIA should be seen within 24 hours after symptom onset at a TIA outpatient clinic for a neurological assessment, while secondary stroke prevention should be started as soon as possible after a con rmed diagnosis of TIA/minor stroke (43,44) or directly if the patient cannot be assessed by a neurologist the same day.(45) Therefore, we considered U3 (patient seen within three hours) as su cient in patients who nally showed to have had a TIA/minor stroke.

Implications for research and/or practice
Our study indicated that the accuracy of the NTS was poor to moderate, yet safety improved after overruling by the triage nurse. Apparently, triage nurses and/or their GP supervisors capture some vital patient information that is not yet incorporated in the NTS. Further improvement of safety, as well as improving e ciency of telephone triage in the domain of patients calling with neurological symptoms is necessary. Improving the accuracy of already existing triage systems such as the NTS should be the rst step. In order to do so, prediction models are needed based on multivariable analyses to provide an evidence-based basis for which triage questions are helpful, and which are not.

Conclusions
The NTS decision support tool used in Dutch OHS-PC performed poor to moderately regarding safety (sensitivity) and e ciency (speci city) in allocating adequate urgencies to patients with and without TIA/stroke/other LTEs. There are indications that overruling the NTS by triage nurses improves safety, without compromising e ciency.

Declarations
Ethics approval and consent to participate The Medical Ethics Review Committee Utrecht, the Netherlands approved this study (National Trial Register identi cation number: NTR7331, reference number WAG/mb/16/003208). In addition, a waiver of informed consent was granted as our study involved minimal risk to subjects and this study would not have been practicable without the waiver. All personal and research data were handled and stored according to the European General Data Protection Regulation.

Consent for publication
Not applicable as all personal and research data were made unidenti able

Availability of data and materials
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors declare that they have no competing interests.

Funding
This work was supported by (i) the department of general practice of the University Medical Center Utrecht, (ii) Associate Professorship-promotion grant of D.L. Zwart, MD, PhD, (iii) the foundation 'Netherlands Triage Standard' and (iv) the 'Stoffels-Hornstra' foundation. The views expressed are those of the authors and not necessarily those of the foundations. This research was conducted without direct involvement from both funding foundations.

Authors' Contributions
DLZ and FHR conceived the idea for the study and gained funding. All authors designed the study. DCE and LTW collected the data, and DCE analysed the data. DCE prepared the manuscript and wrote the rst draft, supervised by DLZ. All authors provided intellectual input, critically reviewed the manuscript and read and approved the nal manuscript.       Table S2.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.