Concordance of care processes between medical records and patient self-administered questionnaires

Background Despite the increasing use of medical records to measure quality of care, studies have shown that their validity is suboptimal. The objective of this study is to assess the concordance of cardiovascular care processes evaluated through medical record review and patient self-administered questionnaires (SAQs) using ten quality indicators (TRANSIT indicators). These indicators were developed as part of a participatory research program (TRANSIT study) dedicated to TRANSforming InTerprofessional clinical practices to improve cardiovascular disease (CVD) prevention in primary care. Methods For every patient participating in the TRANSIT study, the compliance to each indicator (individual scores) as well as the mean compliance to all indicators of a category (subscale scores) and to the complete set of ten indicators (overall scale score) were established. Concordance between results obtained using medical records and patient SAQs was assessed by prevalence-adjusted bias-adjusted kappa (PABAK) coefficients as well as intraclass correlation coefficients (ICCs) and 95% confidence intervals (95% CI). Generalized linear mixed models (GLMM) were used to identify patients’ sociodemographic and clinical characteristics associated with agreement between the two data sources. Results The TRANSIT study was conducted in a primary care setting among patients (n = 759) with multimorbidity, at moderate (16%) and high risk (83%) of cardiovascular diseases. Quality of care, as measured by the TRANSIT indicators, varied substantially between medical records and patient SAQ. Concordance between the two data sources, as measured by ICCs (95% CI), was poor for the subscale (0.18 [0.08–0.27] to 0.46 [0.40–0.52]) and overall (0.46 [0.40–0.53]) compliance scale scores. GLMM showed that agreement was not affected by patients’ characteristics. Conclusions In quality improvement strategies, researchers must acknowledge that care processes may not be consistently recorded in medical records. They must also be aware that the evaluation of the quality of care may vary depending on the source of information, the clinician responsible of documenting the interventions, and the domain of care. Electronic supplementary material The online version of this article (10.1186/s12875-019-0979-7) contains supplementary material, which is available to authorized users.


Introduction
In recent years, measuring and improving quality of care has been a thriving area of interest both in North America and internationally. High-quality data which are a result of good documentation practices are essential not only for patient care but for monitoring quality of care and implementing quality improvement strategies. Despite the increasing use of medical records to measure quality of care, studies have shown that the validity of medical records is suboptimal [1][2][3].
Medical records are a key source of information about care processes (e.g. diagnosis, treatment, referral, prescribing) and clinical outcomes (e.g. morbidity, mortality, health-related quality of life) [4,5]. However, their accuracy may be limited by time pressures and delayed recording which can lead to poor documentation [1,3,4,6]. Other potential drawbacks are the lack of documentation standards [7,8] and the fact that some of the data recorded in medical records depend on patient selfreported information such as symptoms and lifestyle behaviors [1,3]. Self-reported data sources such as patient self-administered questionnaires (SAQs) have some advantages over medical records. They are developed in line with research objectives and data generated by SAQs provide insight into the patient's point of view [9]. However, patient SAQs are dependent on the amount of information the patient remembers, considers relevant, and is willing to share [10,11].
Several studies have assessed concordance between information collected through SAQs and medical records, but have focused on diagnoses [12], health services utilization [13], medication [14], and specific symptoms [15] or diseases [7]. In a primary care setting, particularly in the context of an intervention program aimed at improving quality of care, little is known on the concordance of care processes evaluated using medical records and patient SAQs. As part of a participatory research program (TRANSIT study) dedicated to TRANSforming InTerprofessional clinical practices to improve cardiovascular disease (CVD) prevention in primary care, members of the primary care community developed a set of 81 process indicators (TRANSIT indicators). Among those, 10 indicators were evaluated through both medical record review and patient SAQs. The objectives of this study are to assess the concordance of care processes evaluated through medical record review and patient SAQs using the TRAN-SIT quality indicators and identify patients' sociodemographic and clinical characteristics associated with agreement between the two data sources.

Study design
The TRANSIT study is a three-phase participatory research program aimed at supporting continuous improvement of CVD prevention in primary care. Details on the study design of each phase are described elsewhere (Phase I [16], Phase II [17], and Phase III [18]). The current study is a secondary analysis of the data collected in the cluster randomized controlled trial (Phase III) and has been approved by the research ethics board of the Centre de santé et de services sociaux de Laval (CSSSL) (2013-2014/04-02). Every participating clinician and patient signed an informed consent form.

Study population
Family medicine groups (FMGs) were recruited in the large urban area of Laval, Quebec, Canada. To be eligible, each FMG had to build an interprofessional facilitation team (IFT) composed of voluntary participating clinicians. The role of the IFT was to promote the implementation of the program among colleagues inside and outside the FMG. It was required that each IFT include at least one nurse, two physicians, and one community pharmacist, as well as one nutritionist, kinesiologist, or psychologist. Some FMGs invited their administrative staff to join the IFT, but this remained optional. Pharmacists, nutritionists, kinesiologists, and psychologists could be recruited from outside the FMG. Each FMG was asked to recruit 100 participating patients. To be eligible patients had to meet the following criteria: 1) have 18 years old and older; 2) be at moderate or high risk of CVD as evaluated by the Framingham CVD risk score [19]; 3) have hypertension, dyslipidemia or diabetes that was either uncontrolled or for which pharmacotherapy had initiated in the past 12 months; and 4) have at least two other chronic diseases, excluding CVD and CVD risk factors. Patients were excluded if they did not meet the 4 inclusion criteria or if they received home care services.
A total of eight FMGs, 98 clinicians and 759 patients participated in Phase III of the TRANSIT study. Six FMGs including 590 patients were randomized into the "facilitation" group while two FMGs including 169 patients were randomized into the "passive diffusion" group. Four FMGs used paper medical records while the other four used paper and electronic medical records (EMRs).

Quality indicators
The TRANSIT indicators were developed through a rigorous participatory process involving researchers, health care managers, clinicians, patients, and family members (see Additional file 1). The full set includes 81 process indicators (see Additional file 2: Table S1) that are evaluated using medical records (n = 60), pharmacy renewal charts (n = 4), and patient SAQs (n = 17). Of the 17 indicators documented through patient SAQs, ten were also documented through medical record review. These indicators read as follows: 1) instructions for  [20,21]. Testretest reliability was almost perfect while inter-rater reliably was substantial to almost perfect [22].

Data collection
Five research assistants attended a two-week training session described in the Additional file 1. Once the training was completed, the research assistants documented the TRANSIT indicators retrospectively over the 14 months prior to the end of the TRANSIT study. For each patient, they assessed whether the care processes were in compliance (yes/no) with the indicators or were not applicable.
A patient SAQ was mailed to patients at baseline and at the end of the study (t 14 ) with a postage-paid envelope for returning the completed version of the questionnaire. Clear instructions on how to complete the patient SAQ were found on the first page of the questionnaire. Patients could communicate with a research assistant by phone if they had any questions concerning the questionnaire. The patient SAQ sent at baseline (n = 101 items) was used to assess quality indicators and collect patient characteristics while the one sent at t 14 (n = 97 items) was used to collect data on quality indicators. Items included in the questionnaires were mainly closeended multiple choice and short open-ended questions.

Statistical analyses
Baseline characteristics of participants were described using means and standard deviations for continuous variables as well as numbers and proportions for categorical variables. At t 14 , the compliance (yes/no) to each indicator (individual compliance scores) as well as the mean compliance (number of compliant indicators/number of applicable indicators) to all indicators of a category (subscale compliance scores) and to the complete set of indicators (overall compliance scale score) were established.
The concordance of care processes was determined by assessing correlations between results obtained using medical records and those obtained using patient SAQs. Patients with missing data for care processes in either data source during the 14 months preceding t 14 were not included in the respective statistical analyses. Percentage of concordant evaluations and prevalence-adjusted biasadjusted kappa (PABAK) coefficients [23] were computed for each pair of indicators while intraclass correlation coefficients (ICCs) and 95% confidence intervals (CI) were computed for subscale and overall compliance scale scores. PABAK coefficients were used because, unlike kappa coefficients, they are not influenced by prevalence and bias between raters [23]. PABAK values less than 0, between 0.00 and 0.20, between 0.21 and 0.40, between 0.41 and 0.60, between 0.61 and 0.80, and greater than 0.81 indicate poor, slight, fair, moderate, substantial, and almost perfect agreement, respectively [22]. ICCs values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 indicate poor, moderate, good, and excellent agreement, respectively [24].
To better understand if patient characteristics could influence self-report, an explanatory analysis was conducted. This analysis was conducted only among patients with compliant care processes in the medical record and no missing data in the patient SAQ. For each corresponding pair of indicators, generalized linear mixed models (GLMM) (binary logistic regression) with a random (cluster specific) intercept were used to identify patients' sociodemographic and clinical characteristics associated with agreement of the indicator pair [25]. The sociodemographic characteristics included in this analysis were: age, sex, highest level of education completed, current work, and annual gross family income at baseline. The clinical characteristics included as covariates in the models were: intervention program group, CVD risk category estimated by the Framingham CVD risk score [19], smoking status, body mass index (BMI), diagnosis of hypertension, dyslipidemia, and diabetes as well as the achievement of therapeutic targets for hypertension, dyslipidemia, and diabetes at baseline. An odds ratio (OR) and 95% CI were computed for each pair of indicators. A two-sided alpha level of 0.05 was used to test statistical significance.

Results
Of the 1024 eligible patients invited to participate to the TRANSIT study, 759 (74%) accepted. The patient SAQ response rate was 100% at baseline and 79% at t 14 .
As reported in Table 2, in the "hypertension management" category, 50% of patients received instructions regarding blood pressure monitoring based on patient SAQs (IND29) compared to 15% based on medical records (IND30). In the "diabetes management" and the "interprofessional collaboration" categories, the percentages of patients reporting compliant care processes were also higher compared to the levels of compliance obtained using medical records. For the category "motivational interviewing and support for healthy lifestyle change", levels of compliance based on patient SAQ and medical records were similar for the indicators "meeting with the nurse to discuss lifestyle habit" (77 and 78%) and "use of the TRANSIT health booklet" (41 and 46%). However, compared to medical records, compliance assessed by patient SAQ was lower for the indicator "impact of chronic health conditions" (18 and 34%) and higher for the indicator "personal lifestyle changes target" (76 and 55%). Overall, for seven out of ten indicators, the compliance as assessed by patient SAQ was higher than the compliance assessed by medical records. The percentage of concordant evaluations for each pair of indicators varied from 47 to 83%, with PABAK coefficients ranging from − 0.06 to 0.66. As reported in Table 3, subscale compliance scores (95% CI) obtained using patient SAQs varied from 41% (38-44) to 78% (74-81) while subscale compliance scores obtained using medical records varied from 15% (12)(13)(14)(15)(16)(17)(18) to 53% (50-56). Overall compliance scale scores obtained using patient SAQs and medical records were equal to 51% (49-54) and 37% (35-39), respectively. Concordance between patient SAQs and medical records, as measured by ICCs (95% CI), varied from 0.18 (0.08-0.27) to 0.46 (0.40-0.52) for subscale compliance scores and was equal to 0.46 (0.40-0.53) for the overall compliance scale score.
As reported in Table 4, to examine the association between patients' characteristics and the reporting of compliant care processes by the patient among those with compliant care processes based on the medical record, GLMM were used. Indicators related to

Discussion
To our knowledge, our study is the first to assess the concordance of care processes evaluated using medical records and patient SAQs in the context of an intervention program aimed at improving quality of care. Quality of care, as measured by the TRANSIT indicators, varied substantially between medical records and patient SAQ. Concordance between the two data sources was poor for the subscale and overall compliance scale scores. Seven out of ten pairs of indicators had lower individual compliance scores when evaluated through medical record review. Three pairs of indicators related to care processes delivered exclusively by the nurse had higher individual compliance scores when assessed using medical records. Except for two clinical characteristics, (i.e. smoking status and diagnosis of hypertension), GLMM showed that, among patients who received care based on the medical record, agreement between the two data sources were not affected by patients' sociodemographic and clinical characteristics.
Agreement varied substantially between the ten pairs of indicators and was poor for subscales and overall compliance scale scores. These findings are supported by another study evaluating the concordance between medical records and patient self-reports for diagnoses, clinical services delivered, counselling and referrals as well as medication use [14]. As in our study, authors found substantial variations both across and within domains of medical care. However, compared to the overall compliance scale score obtained in our study (ICC [95% CI]: 0.46 [0.40-0.53]), Tisnado and al. concluded that the total concordance between the two data sources was fair to good (kappa = 0.5 [0.5-0.6] and total agreement = 80%). This could be explained by the coefficients used to evaluate concordance (ICC versus kappa) and the domains being assessed in each study.
Compared to the indicators evaluated using patient SAQs, most indicators evaluated through medical record review had lower individual compliance scores, suggesting that clinicians may not consistently record care processes in the medical record. When carefully examining the ten pairs of TRANSIT indicators used in this study, we noticed they were all related to counseling and referrals to some extent. Other studies have reported that these interventions are poorly documented in medical records compared to patient surveys. An observational study including 4454 patients treated by 138 physicians showed that the sensitivity of medical records was low for measuring health habit counseling and moderate for physical examination, laboratory testing, and immunization [2]. Another study conducted in an academic family practice clinic also concluded that the most common physician-patient disagreements concerned counseling or treatment procedure [26]. Finally, a study including 1270 patients sampled from 39 medical organizations found that, at the domain-level, counseling and referrals had the worst concordance [14].
Three indicators in the category "motivational interviewing and support for healthy lifestyle change" had higher individual compliance scores when they were documented using medical records. These three   [27]. Indeed, in this study, nurses were 2.2 times more likely to document pain assessment after therapy than physicians (30% versus 16% p < 0.001) [27]. Another study comparing nurses' and physicians' documentation of functional abilities of older patient in acute care showed that nurses took more responsibilities in the documentation of the impairment of both Activities of Daily Living and Instrumental Activities of Daily Living [28]. In our study, sociodemographic characteristics were not associated with agreement between medical records and patient SAQ. Some studies found comparable results [9] while others found differences in the odds of agreement by patient demographics such as age, sex, and level of education [11,12,[29][30][31][32]. In terms of clinical characteristics, smoking was associated with reduced agreement between medical records and patient SAQs while a diagnosis of hypertension was associated with increased odds of concordance between the two data sources. Patient motivation, which is influenced by a patient's physical and psychological state, has been found to affect reporting [33]. In our study, hypertensive patients might have been motivated to change their lifestyle habits. This might explain why they were more likely to accurately report a personal target for lifestyle change. Alternatively, sensitive topics such as weight have been found to affect agreement between patients' surveys and medical records [14,34]. In our study, smokers might have omitted to report meeting with the nurse to discuss lifestyle habits to avoid exposing their non-adherence to recommendations related to smoking cessation.

Strengths and limitations
This study was conducted as part of an intervention program aimed at improving CVD prevention in primary care. It focuses on care processes that consider priorities for action and challenges such as the implementation of collaborative practices and the provision of appropriate support for lifestyle changes [16,17]. The reliability of the indicators documented through medical record review were previously assessed in a psychometric analysis [20,21]. Despite these strengths, our study is subject to potential limitations. Only ten indicators in the complete set of TRANSIT indicators were evaluated through both medical record review and patient SAQs. Missing data in patient SAQs (i.e. up to 33% for IND29) reduced the number of patients included in our analyses. Since neither data source could be considered as a gold standard, it was impossible to determine whether it was the patient (overreporting) or the clinician (underrecording) who was responsible for the discrepancy between the patient SAQ and the medical record. Thus, the way in which indicators are rated (yes/no/not applicable) made it impossible to know if care processes did not comply with the indicators or if the information needed to evaluate the compliance was not found in the medical record during the study period. While this study was conducted in a complex population of multimorbid patients at moderate or high risk of CVD with at least one uncontrolled CVD risk factor, the findings might not be generalizable to the population treated in primary care.

Conclusions
To our knowledge, this is the first study evaluating the concordance of care processes based on medical records and patient SAQs in the context of an intervention program aimed at improving CDV prevention in primary care. Our results suggest that medical record and patient SAQ overall agreement is poor. Therefore, when developing and evaluating the effectiveness of quality improvement strategies, researchers must acknowledge that care processes may not be consistently recorded in medical records. They must also be aware that the evaluation of the quality of care may vary depending on the source of information, the clinician responsible of documenting the interventions, and the domain of care (e.g. treatment, diagnosis, management, medication, counseling, referrals, etc.). Our results also suggest that patients' sociodemographic and clinical characteristics do not affect the reporting of care processes.