Identification of adults with symptoms suggestive of obstructive airways disease: Validation of a postal respiratory questionnaire
© Frank et al; licensee BioMed Central Ltd. 2003
Received: 15 November 2002
Accepted: 25 April 2003
Published: 25 April 2003
Two simples scoring systems for a self-completed postal respiratory questionnaire were developed to identify adults who may have obstructive airways disease. The objective of this study was to validate these scoring systems.
A two-stage design was used. All adults in two practice populations were sent the questionnaire and a stratified random sample of respondents was selected to undergo full clinical evaluation. Three respiratory physicians reviewed the results of each evaluation. A majority decision was reached as to whether the subject merited a trial of obstructive airways disease medication. This clinical decision was compared with two scoring systems based on the questionnaire in order to determine their positive predictive value, sensitivity and specificity.
The PPV (positive predictive value) of the first scoring system was 75.1% (95% CI 68.6–82.3), whilst that of the second system was 82.3% (95% CI 75.9–89.2). The more stringent second system had the greater specificity, 97.1% (95% CI 96.0–98.2) versus 95.3% (95% CI 94.0–96.7), but poorer sensitivity 46.9% (95% CI 33.0–66.8) versus 50.3% (95% CI 35.3–71.6).
This scoring system based on the number of symptoms/risk factors reported via a postal questionnaire could be used to identify adults who would benefit from a trial of treatment for obstructive airways disease.
The aim of the study was to validate two simple scoring systems for a self-completed postal questionnaire designed to identify adult patients likely to have asthma/COPD (obstructive airways disease).
It has been reported that asthma is under-diagnosed and under-treated in adults [1, 2]. In the short term this probably leads to increased morbidity for sufferers, and in the long term it may have a detrimental effect on their lung function and clinical state . Chronic obstructive pulmonary disease is also thought to be under-diagnosed in adults . This may lead to increased morbidity and the loss of an effective opportunity for giving smoking cessation advice. There may be clinical benefits to the individual and health economic benefits to society from identifying and treating patients who have asthma or COPD and who are unknown to the medical services. Before this hypothesis can be tested, however, mechanisms for identifying such patients need to be established. Full clinical review of everyone in a community is neither cost efficient nor practical. An alternative approach is to use a screening questionnaire, designed to identify individuals most likely to have the condition, and therefore most likely to benefit from clinical review.
The Wythenshawe Community Asthma Project (WYCAP) is a long term prospective study of the natural history of respiratory symptoms in two general practice populations in Manchester, UK. One of its aims is to develop a method for identifying patients in the community with asthma and COPD as the first stage in evaluating the benefits and costs of treating such individuals. In 1993 and 1995, postal respiratory questionnaire surveys were carried out in both practice populations. We did not feel that it would be possible to differentiate asthma from COPD using a simple postal questionnaire and so two simple scoring systems were developed, based on the number of symptoms/risk factors reported from key questions, both having the aim of identifying adults with either asthma or COPD. This paper examines the validity of these scoring systems, using data relating to respondents to the 1995 postal survey. A similar exercise has been performed on children in the two practices . This study had ethical approval from the local research ethics committee and informed consent was obtained from all subjects attending for clinical review.
Six key questions from the respiratory questionnaire and two markers of disease severity
Six key questions:
1) Have you had wheezing or whistling in your chest at any time*?
2) Have you woken up with a feeling of tightness in your chest*?
3) Have you been woken by an attack of shortness of breath at any time*?
4) Have you been woken by an attack of coughing at any time*?
5) Has any person in your family (parents, grandparents, sisters, brothers, or your children) had asthma?
6) Have you ever had hay fever or eczema?
*(in the last 12 months)
Markers of severity:
a) Breathlessness when wheezing was present.
b) Wheezing or whistling without a cold.
Response categories of respondents, those invited for and those attending clinical review
Number of symptoms/risk factors
Number invited for clinical review
Number who attended review (% of those invited)
The clinical assessment (carried out by a clinician, TLF) involved a structured medical history and physical examination. A research assistant performed the investigations, supervised and assisted by the clinician. Neither individual had access to the postal questionnaire results for the subject being examined.
The investigations performed have been described in a previous study  and included forced expiratory volume in one second (FEV1), forced vital capacity (FVC), FEV1 / FVC, a peak expiratory flow diary, skin prick testing using the skin prick/puncture technique  and reversibility testing to β2 agonists . A bronchial challenge test to histamine was also performed by the YAN technique . A positive histamine challenge was defined as a fall in FEV1 of 20% from the pre challenge FEV1.
For each subject, three independent consultant chest physicians were sent full details of the clinical assessment but excluding the details of the postal questionnaire. The consultants were asked to decide whether the subject merited a trial of asthma/COPD (obstructive airways disease) medication. Patients warranting a trial of treatment were said to have a diagnosis of possible obstructive airways disease.
Four or more symptoms/risk factors reported from the six key questions.
Four or more symptoms/risk factors reported from the six key questions plus at least one marker of severity (Table 1).
The positive predictive value, sensitivity, specificity and their confidence intervals were calculated for each scoring system using the majority consultant opinion as the "gold standard". The method used was appropriate for the two-stage process . The prevalence of subjects who would merit a trial of obstructive airways disease medication was also estimated using the two-stage process.
Of 10,429 questionnaires sent out in 1995, 7,580 were returned after three mailings. After a 5.5% adjustment for adults no longer at the mailing address , this represented a 78% response rate. 6947 responses contained sufficient information for the present analyses. 420 adults were invited for clinical review of whom 202 (48%) attended.
Level of consultant agreement Number of adults categorised according to whether they merit a trial of obstructive airways disease medication by the level of consultant agreement.
Meriting a trial of obstructive airways disease medication
2 out of 3 agreed
Comparison between majority expert opinions and questionnaire scoring systems
Categories of score
"Meriting a trial of medication"
"Not meriting a trial of medication"
Four or more symptoms/risk factors
Fewer than four or more symptoms/risk factors
Four or more symptoms/risk factors plus one marker of severity
Positive predictive value, sensitivity and specificity of the two scoring systems Discriminative properties of two scoring systems for questionnaire responses compared with the majority expert opinion about whether the patients merit a trial of obstructive airways disease therapy.
1. Four or more symptoms/risk factors
2. Four or more symptoms/risk factors plus one marker of severity
PPV % [95% CI]
Sensitivity % [95% CI]
Specificity % [95% CI]
The prevalence of adults "meriting a trial of obstructive airways disease therapy" (a surrogate for obstructive airways disease) in our survey was calculated  to be 22% (95% CI 15.5–31.4).
Two simple scoring systems, (based on the number of symptoms/risk factors reported on a respiratory questionnaire) produced high positive predictive values when compared with majority opinion of three respiratory physicians regarding possible obstructive airways disease.
Two important sources of selection bias could have occurred. First, non-respondents to the questionnaire survey could have been materially different from respondents. This was examined after the first questionnaire survey in 1993, when a comparison was made between the practice medical records of a random sample of 100 respondents and 100 non-respondents. No important differences were found with respect to age, gender, and total number of consultations and consultations for respiratory problems in the previous year. There is no reason to suspect that the situation was different in 1995. Second, the adults attending for clinical review may have been different from those who were invited but who did not attend. These two groups were compared with respect to their age, gender and the number of cigarettes smoked as reported on the questionnaire. No significant differences were found between the two groups.
The purpose of a screening test is to identify individuals with a good chance of having disease and who require further clinical assessment to confirm or refute the diagnosis. When defining the threshold at which a screening test is deemed to be positive, consideration has to be taken of the balance between false positive results (which can lead to extra distress because of unnecessary further investigations) and false negatives (which result in some cases of disease being missed). The positive predictive value of a test reflects the frequency of disease in those with a positive screening test. The cost effectiveness of a screening programme will depend on the cut-off values chosen for the screening procedure as these determine the number of new cases detected and requiring treatment, unnecessary investigations undertaken etc.
When assessing a screening test it can be difficult to know what diagnostic standard to use, particularly for conditions such as obstructive airways disease which do not have a universally accepted clinical definition. In our study, we defined adults as having possible obstructive airways disease if the majority opinion of three consultant physicians was that a trial of treatment for obstructive airways disease was merited (majority decisions of specialists has been used in previous studies to diagnose asthma) [1, 5]. Many clinicians would probably agree that it is reasonable to assess an adult who might merit a trial of obstructive airways disease treatment. Comparing the two scoring systems against this "standard", we were able to determine the test characteristics (sensitivity, specificity and positive predictive value) of each system. The more stringent system (four or more symptoms/risk factors plus a marker of severity) had a greater positive predictive value and greater specificity (so gave fewer false positive results) than the less stringent system (four or more symptoms/risk factors alone), but at the cost of reduced sensitivity (more cases were missed).
Any system, which increased sensitivity, would almost certainly have reduced positive predictive value probably resulting in and unacceptable rate of false positives. In contrast with screening for malignant conditions when detection of all cases is of prime importance (i.e. high sensitivity), it could be argued that in population screening for diseases such as asthma/COPD the PPV carries more weight than sensitivity.
A recent article by Grimes and Schultz  illustrates this point:
"Although sensitivity and specificity are of interest to public-health policymakers, they are of little use to the clinician. Stated alternatively, sensitivity and specificity (population measures) look backward (at results gathered over time). Clinicians have to interpret test results to those tested. Thus what clinicians need to know are the predictive values of the test (individual measures, which look forward)".
In a previous paper , 1112 patients (13.8% of responders) were identified positively by the screening questionnaire. The PPV would indicate that approximately 834 of these had obstructive airways disease. Further, 529 of those who screened positively had no recorded diagnosis of obstructive airways disease or received inhaled medication in the previous 12 months. Thus despite the disadvantage of a relatively low sensitivity for the scoring system significant numbers of patients with obstructive airways disease would be identified. It should be noted that the prevalence of adults "meriting a trial of obstructive airways disease therapy" in our survey was calculated  to be 22% (95% CI 15.5–31.4). This may be an overestimate of the true prevalence of obstructive airways disease as it will include patients who would not benefit from a trial of treatment.
The three specialists reviewing the clinical information relied on detailed written data supplied to them by the research team; they did not have direct contact with patient. Neither the experts nor the examining team were aware of the results of the postal questionnaire at the time of their involvement in the study. Total agreement by the consultants on whether a subject warranted a trial of medication was reached in 59.2% of adults. Difficulties in making a diagnosis without personally seeing the patient may also have contributed to disagreement between consultants although it is likely that some difference of opinion would have persisted even with personal examination. Expert opinion concerning asthma diagnosis has been used to define asthma in previous epidemiological studies [1, 11, 12]., whilst others have used patient recall of asthma diagnosis or treatment when measuring prevalence of the disease or its underdiagnosis . The majority of diagnostic decisions are made in primary care and we acknowledge that choosing three consultant respiratory physicians as the diagnostic gold standard therefore has limitations. They were however chosen to reflect a spread of secondary and tertiary respiratory opinions.
An advantage of the scoring systems used was their simplicity. Techniques such as discriminant analysis and logistic regression could have been used to assess the predictive value of each question and a different scoring system developed from them. This would have the disadvantage that validation would then have been based on the same data as was used to derive the scoring system and therefore would tend to be over optimistic. In addition the scoring system might be more difficult to implement.
In clinical practice, a simple scoring system to identify patients requiring further review is attractive. Choices about which system to use will depend on a number of factors; the balance between positive predictive value and sensitivity of each system and available health care resources. It is important to remember that if this questionnaire were to be used in another setting the positive predictive value would have to be recalculated as this value is dependent on the prevalence of disease in the population studied.
When assessed against the majority expert opinion of "meriting a trial of obstructive airways disease medication", two simple scoring systems based on responses to a brief postal questionnaire provided a good method for identifying adults likely to benefit from a trial of treatment for obstructive airways disease and so warranting clinical review.
The authors would like to thank the three consultants, Dr J Miles, Dr S Burge and Dr J Williams who provided our expert majority diagnosis and Tamsin Wright who helped with the clinical evaluations; the staff of Bowland Medical Practice and Tregenna Group Practice and the North West Lung Research Centre; and the subjects who answered the postal questionnaire and who attended for clinical review. We would like to thank the Royal College of General Practitioners (RCGP) Manchester Research Unit (now relocated to Aberdeen). This study was supported by Glaxo-Wellcome, Zeneca Pharmaceuticals, Manchester Airport, The Royal College of General Practitioners and The North West Lung Centre.
- De Marco R, Cerveri I, Bugiani M, Ferrari M, Verlato G: An undetected burden of asthma in Italy: the relationship between clinical and epidemiological diagnosis of asthma. Eur Respir J. 1998, 11: 599-605.PubMed
- Frank PI, Ferry S, Moorhead T, Hannaford P: Use of a postal questionnaire to estimate the likely under-diagnosis of asthma-like illness in adults. Br J Gen Pract. 1996, 46: 295-297.PubMed CentralPubMed
- Agertoft L, Pedersen S: Effects of long-term treatment with an inhaled corticosteroid on growth and pulmonary function in asthmatic children. Respir Med. 1994, 88: 373-381.View ArticlePubMed
- van den Boom G, Rutten-van Molken MP, Folgering H, van Weel C, van Schayck CP: The economic effects of screening for obstructive airway disease: an economic analysis of the DIMCA program. Preventive Medicine. 2000, 30: 302-308. 10.1006/pmed.2000.0633.View ArticlePubMed
- Frank TL, Frank PI, McNamee R, Wright T, Hannaford P, Morrison J, et al: Assessment of a simple scoring system applied to a screening questionnaire for asthma in children aged 5–15 yrs. Eur Respir J. 1999, 14: 1190-1197. 10.1183/09031936.99.14511909.View ArticlePubMed
- Basomba A, Sastre A, Peleaz A, Romar A, Campos A, Garcia-Villalmanzo A: Standardisation of the prick test. Allergy. 1985, 40: 395-399.View ArticlePubMed
- American Thoracic Society.: Standards for the diagnosis and care of patients with chronic obstructive pulmonary disease (COPD) and asthma. Am Rev Respir Dis. 1987, 225-244.
- Yan K, Salome C, Woolcock AJ.: Rapid method for measurement of bronchial responsiveness. Thorax. 1983, 38: 760-765.PubMed CentralView ArticlePubMed
- McNamee R: Optimal designs of two-stage studies for estimation of sensitivity, specificity and positive predictive value. Statistics in Medicine. 2002, 21: 3609-3625. 10.1002/sim.1318.View ArticlePubMed
- Grimes DA, Schulz KF: Uses and abuses of screening tests. Lancet. 2002, 359: 881-884. 10.1016/S0140-6736(02)07948-5.View ArticlePubMed
- Jenkins MA, Clarke JR, Carlin JB, Robertson CF, Hopper JL, Dalton MF, et al: Validation of questionnaire and bronchial hyperresponsiveness against respiratory physician assessment in the diagnosis of asthma. Int J Epidemiol. 1996, 25: 609-616.View ArticlePubMed
- Burr ML: Editorial: Diagnosing asthma by questionnaire in epidemiological surveys. Clin Exp Allergy. 1992, 22: 509-510.View ArticlePubMed
- Kaur B, Anderson HR, Austin J, Burr M, Harkins LS, Strachan DP, et al: Prevalence of asthma symptoms, diagnosis, and treatment in 12–14 year old children across Great Britain (international study of asthma and allergies in childhood, ISAAC UK). BMJ. 1998, 316: 118-124.PubMed CentralView ArticlePubMed
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2296/4/5/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.