Development of the Health Literacy of Caregivers Scale - Cancer (HLCS-C): item generation and content validity testing

Background Health literacy refers to an individual’s ability to engage with health information and services. Cancer caregivers play a vital role in the care of people with cancer, and their capacity to find, understand, appraise and use health information and services influences how effectively they are able to undertake this role. The aim of this study was to develop an instrument to measure health literacy of cancer caregivers. Method Content areas for the new instrument were identified from a conceptual model of cancer caregiver health literacy. Item content was guided by statements provided by key stakeholders during consultation activities and selected to be representative across the range of cancer caregiver experiences. Content validity of items was assessed through expert review (n = 7) and cognitive interviews with caregivers (n = 16). Results An initial pool of 82 items was generated across 10 domains. Two categories of response options were developed for these items: agreement with statements, and difficulty undertaking presented tasks. Expert review revealed that the majority of items were relevant and clear (Content Validity Index > 0.78). Cognitive interviews with caregivers suggested that all except three items were well understood. Conclusion A resultant 88 item questionnaire was developed to assess cancer caregiver health literacy. Further work is required to assess the construct validity and reliability of the new measure, and to remove poorly performing and redundant items, which will result in a shorter, final measure. The new measure has the potential to inform the development and evaluation of interventions and the improvement of health service delivery to cancer caregivers.


Background
A diagnosis of cancer impacts not only the person diagnosed, but also their family members and friends. These social supports are often called upon to provide informal care and assistance managing the disease [1] and to provide practical, emotional and physical support [2]. Individuals who provide informal care and support, often referred as caregivers [3], also play a significant role in health-related decision-making [4], are involved in communications with healthcare providers [5], and assist with sourcing and interpreting health information [6]. These caregiving responsibilities are often undertaken unexpectedly, and caregivers are often provided limited information and support [1]. Recognition of the challenges of the caregiving role has led to development of interventions designed to meet the informational, practical, and psychosocial needs of caregivers [7][8][9]. Although information provision is included in the majority of these interventions [10], few studies have examined improvements in the level of caregivers' knowledge and skills [7,10]. This may, in part, be due to the lack of measurement tools that assess caregiver knowledge and skills [2].
Consistent with broad definitions of health literacy [11][12][13], caregiver health literacy is defined here as the personal characteristics and social resources needed for caregivers to access, understand, appraise and use information and services to participate in decisions relating to the health and care of the care recipient. This includes the capacity to communicate, assert and enact these decisions. Whilst evidence suggests an association between poor health literacy and poorer health outcomes [14], worse physical functioning and reduced quality of life [14][15][16][17][18], little is understood about the relationship between caregiver health literacy and the health outcomes of care recipients.
To accurately identify the health literacy needs of cancer caregivers, and understand the impact of caregiver health literacy on care recipient health outcomes, it is essential to measure the construct effectively. Previous studies of caregiver health literacy [19][20][21][22] have used measures that assess a subset of health literacy constructs. Measures such as the Test of Functional Health Literacy in Adults (TOFHLA [23]) or its short form [24], the Rapid Estimate of Adult Literacy in Medicine [25], and the Newest Vital Sign [26] assess an individual's reading, numeracy, and comprehension skills in relation to healthcare. Reviews of health literacy measurement instruments increasingly call for the development of tools that capture the full range of health literacy constructs [27][28][29], such as critical thinking, interaction and communication, and confidence [11].
In response to this gap in the literature, health literacy measurement tools are now emerging that capture the multidimensional nature of health literacy [30,31]. However, these tools are grounded in the perspectives of the potential care recipient, and have limited utility for the identification of the needs of caregivers. Similarly, caregiver health literacy measures designed to assess health literacy of parents of infants [32,33] cover domains not relevant to the role of caregiving for an adult recipient.
The aim of the current study was to develop a measure of health literacy specifically for caregivers of people with cancer. Best practice guidelines for questionnaire development require a detailed conceptual basis to guide development [34,35]. The conceptual model of caregiver health literacy developed by the authors (Yuen, Dodson, Batterham, Knight, Chirgwin, & Livingston, in press) was used as the basis for the development of the Health Literacy of Caregivers Scale -Cancer (HLCS-C). The model, as shown in Figure 1, proposes six major themes and 17 sub-themes associated with caregiver health literacy.

Methods
A validity-driven approach [36] was employed in the development of the HLCS-C. The steps undertaken are outlined in Figure 2. The study was approved by the Eastern Health Human Research Ethics Committee (E41-1011) and Deakin University Human Research and Ethics Committee (2011-115), in Melbourne, Australia.

Content area specification
The content areas for inclusion in the questionnaire were drawn directly from 17 sub-themes in the conceptual model of cancer caregiver health literacy (see Figure 1). The following considerations were used to determine whether (and how) themes should be represented in the questionnaire: 1) the questionnaire should capture the experiences of caregivers caring for recipients with a wide range of cancer types, stages, treatments, and potential outcomes; 2) the questionnaire should capture the experience of caregivers providing differing forms and levels of support; 3) the questionnaire should be consistent with the broad definition of caregiver health literacy, and encompass factors associated with accessing, understanding, appraising and using health information to promote and maintain the health of the care recipient; 4) the questionnaire should be presented as a list of items/statements accompanied by an appropriate set of response options; and 5) the questionnaire should contain the fewest number of domains as possible to reduce length and administration burden.
Another consideration when identifying content areas for inclusion was whether representative statements generated by participants during consultation activities captured caregiver experiences or whether they captured broader contextual factors that influenced caregiver health literacy. In addition, content areas were examined to determine whether statements representative of a sub-theme could be combined to form a scale; previous scale development studies that used similar processes to derive a conceptual model, have found that although statements within some sub-themes were conceptually related, could not be summed to form a scale score, and required deletion on psychometric grounds [30]. Further, to assist cross-referencing of the new measure against other measurement tools that assess related constructs, the included content areas were also aligned with a recently developed taxonomy that identified 12 dimensions of health literacy [11]: literacy; interaction; comprehension; numeracy; information seeking; application/function; decision making/critical thinking; evaluation; responsibility; confidence; navigation; and maintaining and promoting health ( Table 1).

Generation of items and response scale
Statements and words provided by participants during consultation activities associated with the development of the cancer caregiver health literacy conceptual model (see Table 1) were used as the starting point for questionnaire items [37] to maximize content validity. For each content area, item selection and refinement was guided by two vignettes developed to describe an individual with a high degree of capacity in that area, and the other with low levels [30]. Where the proposed content areas for the new measure were similar to domains included in the Health Literacy Questionnaire (HLQ; [30]), a validated measure of health literacy derived using similar approaches, the HLQ items were used as the basis and revised to accommodate the caregiver audience. Response scales for each content area were developed to match the nature of the associated items and vignettes. Refinements to how content areas and vignettes were framed were undertaken to ensure consistency in response scales across the content areas of the proposed questionnaire. Items were also examined against a structured item  development criteria [38] (see Table 2). Readability of items was assessed using Flesch Reading Ease [39] and Flesch-Kincaid Grade Level [40] formulas available through Microsoft Word. Item difficulty was included in the item development criteria to ensure that the final items formed a scale that could distinguish between low, moderate, and high levels of health literacy (i.e., scale sensitivity). The revised Bloom's taxonomy, which includes two dimensions (knowledge and cognitive process; [41,42]) was used to guide the selection of set of items for each content area to ensure they captured a range of difficulty. The first Bloom dimension describes levels of knowledge acquired (factual, conceptual, procedural, or metacognitive) whilst the second dimension describes cognitive processes that occur during learning (remembering, understanding, applying, analyzing, evaluating, and creating; [41,42]). It is posited that items that address higher level cognitive tasks (e.g., decisionmaking) would elicit fewer maximum ratings compared to items that addressed lower level cognitive tasks (e.g., access to information). The taxonomy has been previously used to guide the development of health literacy measures [30,43,44].

Expert review
Expert review of items was undertaken to establish the content validity of the proposed items [45]. In a judgmentquantification process [46], items within each proposed scale were assessed by seven experts for relevance and clarity. Participants included two oncologists, a general practitioner, an oncology social worker, a general medical nurse, a health researcher, a policy advisor for a state-wide caregiver organization, and a retired executive member of a cancer information and support service. The content validity of the tool as a representation of its intended purpose was also qualitatively assessed. Experts were identified and recruited from the research team's existing professional networks. Between 5 and 10 experts have been suggested as a number sufficient for establishing content validity using expert review [46].
Experts were asked to assess each item for relevance and clarity using a 3-point scale ("low, moderate, high" and "unclear, neutral, clear" respectively). To determine content validity, expert ratings for relevance and clarity were quantified using the Content Validity Index (CVI) calculated as the percentage of experts who indicated 2 or 3 on the scale. It has been recommended that when six or more experts have evaluated the instrument, items with a CVI less than 0.78 should be considered for revision or deletion [46].
Experts were also asked to consider all items within individual scales and respond to two open-ended questions, "Do you suggest including any other ideas to represent the scale", and "Do you suggest changing any words for any of the above items". Experts were also asked to provide feedback on whether any major concepts or ideas were omitted in the questionnaire and to make suggestions on how to improve the instrument. To

Cognitive interviews
Cognitive interviews are frequently used in questionnaire development to determine whether respondents interpret and respond to items in the way the researchers intended [47]. The think-aloud approach [47,48] was the predominant method used in the current study. A convenience sample of participants was recruited from a not-for-profit government funded caregiver organization. Ninety-nine caregivers who identified themselves as providing care to a family member or friend with cancer were invited to participate. Nineteen caregivers (19%) who returned the questionnaire were then contacted via telephone about taking part in a telephone interview. Three respondents completed the questionnaire; however, they declined to participate in the cognitive interview because of personal circumstances. Of the 16 caregivers who participated, the majority were female (94%), and ranged in age between 42 and 80 years (Mdn = 61.5; see Table 3).
To minimize respondent burden, a sampling scheme was applied to allow each participant to be interviewed on items from approximately 6, rather than all 10, constructs in the questionnaire. Participants were randomly assigned an item set that included items from complete constructs. Using this method, each item in the questionnaire was reviewed by at least 8 participants (range = 8 -11; Mdn = 9). Although participants did not complete the full set of items, the sampling scheme was sufficient as the purpose of the cognitive interviews was to test the items across a range of individuals to inform decision making [47].
Responses from the cognitive interviews were analyzed using a systematic evaluation of participant responses for each item [49]. Each item was assessed using three criteria: whether the participant interpreted the question as the researchers had intended; whether the item was applicable to the participant; and whether the participant found it difficult to respond to the item. In cases where responses had problems with an item, common themes and issues were noted.

Selection of content areas
Inspection of the 17 sub-themes outlined in the cancer caregiver health literacy model against the considerations for inclusion of content areas led to the identification of 10 constructs for the new questionnaire (see Table 1). Several sub-themes were subsumed under the encompassing scale titles: Adequate information about cancer and cancer management, and Understanding the healthcare system. Two sub-themes were considered broader contextual factors that influenced caregiver health literacy, and thus were excluded from the questionnaire. For example, statements in the Financial and Legal Support subtheme related to availability of support from Government services, which was considered a broader contextual factor that influenced a caregiver's capacity to effectively engage with the caregiving role. Two additional sub-themes were excluded because their representative statements, although conceptually related, were considered unable to be summed to form a scale.

Item generation and response options
Eighty-two items were developed for expert review, with 7 to 12 items for each construct (see Table 1). An item pool 50% larger than that intended for the final scale was drafted to enable identification of items with adequate internal consistency as determined through psychometric analyses (Phase 3; see Figure 2) [45]. For eight content areas, an 'agree/disagree' Likert scale was suitable. For the remaining two content areas (Processing health information, and Active engagement with healthcare providers) a 'cannot do/very easy' Likert scale was more suitable. Readability analysis of the items showed a Flesch-Kincaid reading level of grade 6.7, with a Flesch reading ease of 80.6 (out of a possible 100, with higher scores indicating greater ease).

Expert review
The range of content validity indices for relevance and clarity for the ten scales as assessed by 7 experts are provided in Table 4. Although 8 experts responded, one participant provided general comments about including additional content areas rather than assessing all individual items, thus, was excluded from the content validity analysis. The participant's comments were considered when determining the inclusion of additional content areas. Items were considered relevant by experts (CVI > 0.78) for all but one item related to processing health information (#70, "Find out if the health information that I have received is suitable for the person I am caring for"). Item #70 was considered invalid both for relevance and clarity (CVI < 0.78), and thus was revised (see Table 5) after considering expert comments, and reviewing participant statements generated during concept mapping workshops.
Using the content validity equation, five items although deemed relevant by experts, were considered to lack clarity (CVI < 0.78; see Table 4). These five items were revised (see Table 5). An additional item (#81, "I know which healthcare providers look after the health of the person I care for"), although deemed relevant and clear, was deleted in response to expert comments about its similarity to another item in the scale.
Twelve items, although demonstrated adequate relevance and clarity (CVI > 0.78), underwent minor revisions in response to suggested improvements from experts (See Table 6). Item #12 ("I have strong support from at least one friend" was combined with item #66 ("I have strong support from at least one family member) following feedback about the similarity of items, and suggestions from experts to merge the two items.
Nine new items were included in the questionnaire in response to comments from experts (see Table 7). Revision of the item pool resulted in 89 items for testing through cognitive interviews. Experts identified three main areas that were missing from the questionnaire: understanding of healthcare services, palliative care, and sexuality issues. However, only the concept of understanding of healthcare services was captured in newly generated items.

Cognitive interviews
Overall, participants interpreted and responded to the majority of the questionnaire items in ways intended. However, three items (#18, #74 and #1) emerged as having common issues. For item #18 ("I have all the information I need to help make decisions about treatments") two participants reported that they did not help make decisions about treatments, thus the item was not personally relevant to them (e.g. "I'm not a doctor and I wouldn't know of other treatments, so I trusted what doctors told me" [Participant 1]). For item #74 ("Find out if health information from various resources is suitable for the person I am caring for") participants interpreted the word 'resource' as being internet-specific (e.g. "Yeah I think it is, you just borrow the kids internet and have a look" [Participant #11]). Further, for item #1 ("I spend a lot of time looking for information about the cancer") two caregivers reported that although they spent time looking for information when their care recipient was first diagnosed with cancer, it was no longer relevant after many years of providing care (e.g. "My husband has had cancer now for years. At the beginning I spent a lot of time researching but now only when you feel up to it" [Participant #10]).

Discussion
The current study describes item generation and content validity testing of a new questionnaire to assess the selfreported health literacy of caregivers of people with cancer, the Health Literacy of Caregivers Scale-Cancer (HLCS-C). As a result of the expert review and cognitive interviews, the HLCS-C now contains 88 items across 10 scales: proactivity and determination to seek information; The scales included in the HLCS-C covered a broad range of themes that assessed individual, interpersonal as well as healthcare provider and healthcare system factors that may be relevant to caregiver health literacy. Many of these themes are currently not included in widely-used measures of health literacy. For example, some scales in the HLCS-C assess an individual's comprehension (e.g., Adequate information about cancer and cancer management, and Understanding the healthcare system), or their critical thinking skills (e.g., Processing health information), while other scales assess a caregiver's interpersonal relationship with the care recipient (e.g., Communication with the care recipient, and Understanding the care recipient). Caregivers' capacity to effectively engage with healthcare providers was also included (Active engagement with healthcare providers). Further extending dimensions of health literacy measures, the HLCS-C assesses external influences on an individual's health literacy. Similar to the Health Literacy Questionnaire [30], the HLCS-C contains a scale that assesses the caregivers' perspectives of healthcare provider provision of services and information in ways that enable them to adequately navigate the caregiving role and the healthcare system (e.g., Supported by healthcare providers to understand information). Unlike the existing unidimensional measures of health literacy, the multidimensional nature of the HLCS-C allows identification of specific strengths and difficulties and therefore the identification of opportunities to improve caregiver health literacy and the health literacy responsiveness of the healthcare system.
As part of the expert review, experts suggested including items related to sexuality issues. However, the authors made the decision to not include items related to sexuality issues as this topic was not identified by stakeholders during the concept mapping workshops. Concept mapping workshop participants included caregivers providing care for, and people with cancer, across a range of cancer types and stages. It is possible that issues of sexuality were not their primary concern when identifying health literacy needs. Further, it is possible that given the workshop setting, participants may have felt uncomfortable discussing the topic of sexuality. Moreover, studies have shown that caregivers of people with gender-specific cancers (e.g., breast or prostate) were more likely to report additional information needs related to sexual and physical intimacy [50]. Further revisions of the questionnaire could consider sub-sets of items relevant to specific cancer types.
Similarly, experts commented on the inclusion of items related to palliative care. However, the questionnaire was   To address the three items identified as having common issues following cognitive interviews, the decision was made to revise two items and delete one item. To ensure included items were relevant to all cancer caregivers, item #18 was revised to "I have enough information to understand the potential side effects of cancer treatment", which still captured the concept of adequate cancer information. To improve clarity for item #74, the word 'resources' was replaced with 'places' , as participants frequently used this word during cognitive interviews to describe sources of information. Further, as item #1 was not relevant for all caregivers across the caregiving trajectory, the item was deleted. Cognitive testing of the revised items is suggested to ensure items are understood as intended.
Two of the 16 participants responded with 'disagree/ very difficult' on five items, which suggested that they had difficulty, or were unable to complete that task. However, during the cognitive interviews, it was revealed that these participants had provided care to someone who had deliberately avoided conventional cancer treatments for exclusive use of complementary and alternative therapies to manage the cancer. Thus, in responding to specific items, these participants were not conveying difficulty or inability to complete the task; rather their intention was to convey that the item was 'not applicable' to their circumstance. Item writing was guided by statements generated by participants during concept mapping workshops who were recipients of, or caregivers of people who received, conventional cancer treatments. It is therefore recommended that future studies be conducted with caregivers of people who solely receive complementary and alternative therapies to manage their cancer to ensure a sub-set of items that address the health literacy needs of this caregiver population.
Limitations of the study included the low response rate for expert reviews (29%). Although low response rates may potentially affect generalizability of the results, the sample size for the expert review analysis was in line with recommendations [51]. Participation rate was also low for the cognitive interviews (19%); however, between 8 to 11 interviews were conducted for each item, which met the recommended sample size of 5 to 15 participants to identify problems with items [47]. Further, participants for cognitive interviews were predominantly female (94%), which limits generalizability of the findings. Further, reporting error may occur due to the selfreport nature of the questionnaire, in which respondents may report differently depending on their social experiences [52].

Conclusion
Using systematic grounded approaches, a new measure of cancer caregiver health literacy is being developed that contains 10 key constructs hypothesized to represent a caregiver's capacity to find, understand, appraise, and use health information to provide optimal care. The next step in the development of this measure is to assess the reliability and validate the questionnaire in a large sample of Australian cancer caregivers, and reduce the number of items it contains.

Practice implications
The current study represents the first attempt to establish an instrument to measure the health literacy of caregivers of people with cancer. Assessment and understanding of the health literacy needs of caregivers has the potential to enable the evaluation and development of interventions designed to improve caregiver knowledge and skills.

Consent
Written informed consent was obtained from caregivers for the publication of this report.