Can emergency care practitioners differentiate between an avoided emergency department attendance and an avoided admission?

Background After a 999 call to the ambulance service, there is no ‘gold standard’ for determining whether the actions of an emergency care practitioner (ECP) result in a patient avoiding attendance at an emergency department (ED) or avoiding an admission to hospital. Within the Great Western Ambulance Service NHS Trust this outcome has previously been measured using an audit form completed by the ECP. However, the accuracy of the ECP's opinion has not been assessed. Aim To evaluate the accuracy of the ECP's opinion when deciding whether their actions resulted in a patient avoiding attendance at an ED or avoiding hospital admission. Methods Over a 10-week-period in 2009, quantitative data were collected using a case review approach. Anonymised patient consultation records were independently reviewed by an ED consultant and a general practitioner. The decision as to whether the actions of the ECP resulted in the patient avoiding ED attendance or hospital admission was compared between the three healthcare professionals using descriptive statistics and κ values to assess inter-rater agreement. Results Overall inter-rater agreement between the three healthcare professionals was κ=0.385 (fair agreement). The complete agreement rate on a case by case basis for all three healthcare professionals was 80.2% (138/172). Conclusion This study provides some evidence that ECPs can accurately report on whether their actions, at the time of that care episode, result in a patient avoiding attendance at an ED or avoiding a hospital admission.


INTRODUCTION
The traditional UK ambulance service response to a 999 call has been to send a double-crewed paramedic ambulance to the patient, provide any necessary life support to stabilise the patient and transport to an emergency department (ED). 1 Once in the ED, a patient will either be assessed, treated accordingly and discharged, or admitted to hospital for further assessment and inpatient care. 2 The emergency care practitioner (ECP) is a relatively new alternative to this traditional approach. When working for the ambulance service, one of the roles of the ECP is to attend patients who call 999 and, if appropriate, treat and discharge the patient at scene. 3 When an ambulance service ECP attends a patient as a result of a 999 call and completes that care episode at the scene, their actions result in the patient either avoiding attendance at an ED or avoiding hospital admission. The difference between these two outcomes is important because they have very different implications for both the patient and the healthcare system, particularly in relation to resource use and cost. 4 What appears to be unknown is the most appropriate and valid way to collect data which differentiates between these two possible outcomes. It has been suggested that there is no national definition of an avoided admission or how this can be applied in practice. 5 Within the Great Western Ambulance Service (GWAS), an audit form has previously been used to record which of these two outcomes the ECP attributes to their actions. However, the accuracy of an ECP's opinion has not been validated.
Studies that have compared the decision-making of ECPs with that of other healthcare professionals have found favourable results when the ECP has decided whether a patient can be treated at home or requires either ED attendance or hospital admission. 6e8 However, locally, ECPs have expressed concerns that they are not sure which cases should be considered hospital admission avoidance. It also seems likely that this method of data collection is highly subjective. 5 The aim of this study was to evaluate the accuracy of the ECP's opinion when deciding whether their actions resulted in a patient avoiding attendance at an ED or avoiding a hospital admission and, thereby, assess the reliability of this part of the ECP audit process. Because this study was designed to evaluate the ECP's opinion for that care episode, the outcome of any subsequent attendance after a 999 call was not investigated. However, this would be a useful area for further research.

METHODS
This study compared the opinion of the participating ECPs with that of two senior healthcare professionals by reviewing anonymised written records of patients treated at the scene by ECPs.
The reason for seeking the opinion of an ED consultant is that they would assess and determine the treatment plan for these patients if they were conveyed to the ED and not treated at the scene by an ECP. The reason for including a general practitioner (GP) is because the role of the ECP has been likened to a 'space between GP, nurse and paramedic'. 5

Research questions
1. When an ECP treats a patient at the scene, does the ECP consider that their actions avoided an ED attendance or avoided a hospital admission?
2. When an ECP treats a patient at the scene, does an ED consultant consider that the actions of the ECP avoided an ED attendance or avoided a hospital admission? 3. When an ECP treats a patient at the scene, does a GP consider that the actions of the ECP avoided an ED attendance or avoided a hospital admission?

Participants
The opportunity to participate was advertised at two ambulance stations. Student ECPs were excluded from participation. Although participation was voluntary, all seven ECPs based at the two stations agreed to participate (table 1). Wider participation was prohibited by logistical limitations in the documentation retrieval process. ECPs attended all categories of 999 calls as coded by the Advanced Medical Priority Dispatch System. There was no selective dispatching of ECPs.

Data generation
Data were collected using a case review approach over a 10-week period from 1 September 2009. For patients treated at scene, the ECP's opinion as to whether their actions resulted in ED avoidance or hospital admission avoidance was recorded on the GWAS audit form. For each case the ECP completed a GWAS patient care record (PCR), detailing the history of the presenting complaint, previous medical history, drug treatments, their clinical assessment, diagnosis and care plan. Each case PCR was anonymised and photocopied.
An ED consultant and GP independently reviewed the copied PCR and recorded on a separate form whether they believed the actions of the ECP resulted in either an avoided ED attendance or an avoided hospital admission, for that care episode. The ED consultant and GP were provided with no additional case information. The limitations of this methodology are considered later.
Each case was allocated a unique identifying number to enable the outcome opinion of each healthcare professional to be input into an electronic spreadsheet. Additional demographic information was also recorded.

Sample
Patients were included if they were treated at scene by an ECP and not conveyed to hospital during that care episode. However, there are a number of clinical situations which, if attended by a traditional ambulance response, would not result in conveyance to hospital. These situations therefore constituted exclusion criteria. < recognition of life extinct; < successful treatment of hypoglycaemia; < home assistance only; < patient refusal of treatment/transport.
To reduce resource implications, each ECP was responsible for excluding any case that met the exclusion criteria.
No patient-identifying information was presented to the ED consultant or GP. There was no change in the assessment and treatment of patients by ECPs. Submission to an NHS ethics committee (North Somerset and South Bristol Research Ethics Committee) confirmed that patient consent was not necessary for this study.
A review of previous audit data suggested a sample size of 200 patients treated at the scene by the participating ECPs within the available study timeframe. This sample size is similar to other studies of ECP practice. 5 7 Data analysis Descriptive statistics were used to summarise quantitative analysis of data, such as case by case agreement.
To measure the inter-rater agreement between two healthcare professionals Cohen's k was calculated using SPSS V.17.0. Fleiss k was used to calculate the reliability of agreement between the ECP, ED consultant and GP. Fleiss k values were calculated using an online resource (http://www.ccitonline.org/jking/homepage/ kappa1.xls) and interpreted using the values advocated by Landis and Koch. 9

RESULTS
One hundred and seventy-six cases were submitted over the 10week study period. Participating ECPs had not excluded four cases which met the exclusion criteria ( figure 1). Data analysis was carried out on172 cases which met the sample criteria. The mean patient age was 49 years (range 1e96). Over a third of patients treated at the scene were 65 years or older. Analysis of patient age for the 20 cases that ECPs considered admission avoidance found a mean age of 63 years (SD 22), with 12 of these patients aged over 70 years.

Original article
ECPs considered that 11.6% (20/172) of patients treated at the scene avoided hospital admission as a result of their intervention. The ED consultant reported an admission avoidance rate of 15.1% and the GP 9.9%. Figure 2 shows the admission avoidance rates decided upon by each healthcare professional for each ECP. The mean overall admission avoidance estimate was 12.2%.
In 138 of the 172 cases (80.2%) all three healthcare professionals agreed that the outcome was either an ED attendance avoidance or a hospital admission avoidance ( figure 3). Removal of the two outlying cases (ECP B: 100% and ECP D: 65.2%) did not alter the mean significantly.
In addition to full agreement in 138 cases, the ED consultant agreed with the ECP opinion in a further 15 cases. This equates to a case by case agreement rate between ECP and ED consultant of 89%.

DISCUSSION
This study suggests that the opinion of ECPs as to whether their actions had avoided ED attendance or hospital admission has at least some degree of validity. The overall admission avoidance rate appears to be between 10% and 15% of patients treated at the scene by ECPs. However, the study was undertaken over 10 consecutive weeks in early autumn and, consequently, the hospital admission avoidance proportion does not necessarily reflect an annual admission avoidance rate. This study found that over a third of all patients treated at scene, and the majority of those where admission avoidance occurred, were 65 years or older, which corresponds to similar previous studies. 10 The Healthcare Commission Patient Survey Report published in 2004 states that 53% of patients attended by all ambulance trusts are 66 years or older. 11 Once transported to hospital, older patients are more likely to be admitted as a result of uncertainty about their social circumstances and support. This becomes particularly problematic outside normal office hours, and where community services are not well developed. Therefore ECPs may have a particular role in the avoidance of hospital admission in the older adult population. With the number of people in this group predicted to increase by around 53% between 2001 and 2031, 12 the targeting of ECPs to older patients may prove a particularly cost-effective strategy.
Four ECPs (A, C, E and G) reported exactly the same hospital admission avoidance rate as the ED consultant. Interestingly, these participants were the most experienced of the ECPs, suggesting that with increasing experience the opinion of an ECP may also have increasing validity. Similarly, the two ECPs with the least experience compared least well with the outcome decision of the ED consultant.
Disagreements between the three clinical professionals may have a number of explanations. Perhaps most obviously, an ECP may lack knowledge of the hospital assessment process and likelihood of admission after ED assessment. This may be a significant problem for relatively inexperienced ECPs, as the results for this group showed the lowest level of inter-rater agreement. Similarly, the less experienced ECPs may have failed to appreciate the severity of their patient's condition and therefore the desirability of hospital assessment and/or admission.
The overall inter-rater agreement was considered 'fair agreement', with the more experienced ECPs achieving a k value representing 'substantial agreement' or 'moderate agreement'. Limitations such as decision-making from consultation records  and an individual's acceptance of risk make 100% agreement highly unlikely. In practice, an agreement rate between ECP and ED consultant of 89% and between all three healthcare professionals of 80% provides evidence to substantiate local reporting of admission avoidance rates based on the opinion of the ECP, with an acceptable tolerance of 10e20%.

Study limitations
This small-scale study evaluated the validity of a local method of determining whether the actions of ECPs result in patients avoiding ED attendance or avoiding hospital admission. The study design did not allow conclusions to be drawn about the costs and benefits of ECPs as a professional group. Although generalisability is limited by the small number of ECP participants, the range of experience and professional background of the ECPs, coupled with the overall consistency of the results, suggests a level of transferability within GWAS. The educational and clinical development of ECPs in different ambulance trusts means that transferability to other organisations should be approached with some caution.
Unfortunately, the proposed sample size was not achieved (86% of target). The study period could not be extended owing to changes in deployment of ECPs and the impact of pandemic flu.
The study was reliant on the participant ECPs submitting consecutive case records, and excluding those which met the exclusion criteria. Although it is possible that some eligible case records were not submitted, the aim of the study is unlikely to have been adversely affected by 'missed' cases.
It is also important to note that the ED consultant and GP were required to base their opinion exclusively on the ECP's written case record. This approach is known to have weaknesses. 7 An alternate to the written case review would be direct observation of the ECP/patient consultation by an independent clinician and this may be an area for further research.
It has been suggested that k values may be low, even though there are high levels of agreement. 13 This study found that overall inter-rater agreement appears low when compared with the percentage agreement figures. This finding may inform the debate as to the appropriateness of using k values when investigating inter-rater agreement between healthcare professionals.
The age of patients who were conveyed to hospital was not obtained in this study and, consequently, the results cannot provide an indication as to whether the age of patient correlates to the likelihood of treatment at the scene.

CONCLUSION
This study provides some evidence that ECPs can accurately report whether their actions result in a patient avoiding attendance at an ED or avoiding hospital admission, thereby validating this part of a local audit process.
Transferability of the results to other ambulance services employing ECPs should take account of differences in education, experience and methods of deployment. However, this methodology for testing the accuracy of ECP opinion would be feasible in other ambulance trusts.