- Research article
- Open Access
- Open Peer Review
A small unconditional non-financial incentive suggests an increase in survey response rates amongst older general practitioners (GPs): a randomised controlled trial study
BMC Family Practicevolume 14, Article number: 108 (2013)
Few studies have investigated the effect of small unconditional non-monetary incentives on survey response rates amongst GPs or medical practitioners. This study assessed the effectiveness of offering a small unconditional non-financial incentive to increase survey response rates amongst general practitioners within a randomised controlled trial (RCT).
An RCT was conducted within a general practice survey that investigated how to prolong working lives amongst ageing GPs in Australia. GPs (n = 125) were randomised to receive an attractive pen or no pen during their first invitation for participation in a survey. GPs could elect to complete the survey online or via mail. Two follow up reminders were sent without a pen to both groups. The main outcome measure was response rates.
The response rate for GPs who received a pen was higher in the intervention group (61.9%) compared to the control group (46.8%). This study did not find a statistically significant effect of a small unconditional non-financial incentive (in the form of a pen) on survey response rates amongst GPs (Odds ratio, 95% confidence interval: 1.85 (0.91 to 3.77). No GPs completed the online version.
A small unconditional non-financial incentives, in the form of a pen, may improve response rates for GPs.
Low response rates in general practice research are common and lead to loss of power, selection bias ,unexpected budgetary constraints and time delays in research projects. Kellerman and colleagues  conducted a systematic review looking at how to increase response rates amongst physicians. Effective strategies were: monetary incentives, use of stamps on both outgoing and reply paid envelopes and shorter surveys. Non-effective strategies were: pre-notification, personalised mailouts and non-monetary incentives. VanGeest et al.  also performed a systematic review in 2007 amongst physicians and looked at two strategies amongst physicians: incentives and design based approaches. Contrary to expectations, small incentives of $1 were effective whereas non-monetary incentives were much less effective. Additionally, first-class stamps on reply paid envelopes, short surveys, personalised and studies endorsed by professional organisations were more effective. VanGeest et al. also found that postal and telephone strategies were more effective than fax and web based surveys, and mixed methods also appeared to be effective. In 2010, the US National Cancer Institute conducted a large-scale workshop to investigate how best to improve the quality of surveys of physicians and medical groups. One of the key issues discussed was that more research was required into a range of response incentives, including non-monetary incentives .
To further advance the current knowledge base, our study intended to test the effectiveness of an unconditional incentive to increase response rates amongst GPs. One study in the literature reported that the inclusion of an attractive pencil increased the response rate to a second mail-out amongst physicians . Hence, this study examined the effectiveness of offering a small unconditional non-financial incentive on survey response rates amongst GPs. It was hypothesised that an unconditional non-financial incentive in the form of an attractive pen would lead to a higher response rate compared to using nothing.
This study was part of a larger study investigating how to improve the sustainable employment of ageing GPs in rural Australia. The study incorporated both an anonymous survey and optional follow up interviews with participating GPs about sustainable employability. The survey questions pertained to socio-demographic and practice characteristics, retirement intentions, work ability, burnout, work satisfaction and health. The sub-study reported here was conducted to improve the response rates as part of the larger pilot-study.
The following strategies that have demonstrated an effect on physicians’ response rates in experimental studies were used to increase response rates for both groups: use of multiple stamps on reply paid envelopes [1, 5], short survey [1, 5], endorsement by a professional organisation , a topic relevant to the participants , multiple reminders , university based research , guaranteed confidentiality .
The study was a randomised controlled trial.
Participants and setting
Eligible GPs were those who practiced in the Northern Rivers region of NSW, Australia, were aged 45 years and above, and included those doing any related GP work. Participants were drawn from a database held by the local GP Network which is routinely updated and includes all GPs in the region. In October 2011, all eligible participants were sent an invitation package consisting of a letter of support from the local GP Network, a study information letter, a three page quantitative survey, an on-line survey web-address, and an optional invitation to take part in an interview. Because the surveys were returned anonymously, completed surveys were taken as consent. Two reminders were sent two and four weeks after the initial invitation.
The intervention group was sent a non-conditional non-financial reward in the form of a nice looking pen, equating to a monetary value of $2, in the first mail-out. The pen had a mountain view with clouds, the University’s name, and a statement “Doctors working on” printed on it. No further incentives were sent. The control group did not receive the pen.
The primary outcome measure was the response rate of returned surveys.
The GP Network’s database had 125 eligible members aged 45 years and over. Given the small sample, all eligible study participants were included in this sub-study. A similar study surveying early retirement amongst GPs who were recruited from a similar organisation in Australia, achieved a response rate of 59% . Assuming a power of 80%, significance level of 5%, and a response rate of 60% in the intervention group and 35% in the control group, the estimated sample size required to detect a 25% difference was 122 or 61 in each group.
A random number generator in SAS  was used to create random numbers for the two groups. No restriction was made. The list of GPs in the GP Network database were numbered consecutively and GPs were randomly allocated to the intervention or control group. In consultation with GP advisors and based on the literature , it was anticipated that higher response rates would be achieved if the survey was anonymous to allow GPs to take part and answer sensitive questions about their health. Therefore, the surveys did not include an identifying number but only one ‘letter’ to identify which group the returned surveys belonged to.
Participants were blinded because they were not made aware in advance that the trial was carried out. The sequence allocation was not concealed from the project manager or the GP Network staff. However, participants were unaware of treatment allocation.
The odds ratio for response rates was calculated for the first mail out, second reminder and last reminder. Chi-square analyses were used to compare categorical variables between groups. T-tests were used to compare continuous variables.
Evaluating nonresponse bias
In line with Johnson and Wislar , nonresponse bias was examined by:
comparing the survey responses between early responders (initial mail-out) and late responders (2nd and 3rd reminders), under the assumption that later responders are most similar to non-responders; and
Comparing of survey sample with other data sources: the Australian Institute of Health and Welfare, National Health Workforce Data Set 2011 .
The total number of participants aged 45 years and over in the trial was 125. No GPs completed the survey online. Table 1 displays the number of participants randomised and the number responding based on first, second and third mail out. Invitations were sent between October to December 2011. The overall response rate was 54.4%. The final response rate for the intervention and control group were 61.9% and 46.8% respectively and was not statistically significant (Chi-square = 2.89, P = 0.089). The odds ratio for the final response rates for the intervention versus the control group was 1.85 (95 CI% 0.91, 3.77).
Based on survey responses, there were no statistically significant differences between the intervention and control group in relation to socio-demographic, practice and perceived work issues (Table 2).
Evaluation of nonresponse bias demonstrated that there was no difference between early responders and late responders on survey responses (Table 3) and the distribution of gender and age was very similar compared to the rural general practice population on national level (Table 4).
Discussion and conclusions
The response rate for GPs who received a pen was higher in the intervention group (61.9%) compared to the control group (46.8%). The incentive group resulted in a 15.1% higher response rate compared to the control but this did not reach statistical significance.
Strengths and limitations
A strength of this study is that, to our knowledge, this is the first study that investigated the use of a nice pen to improve response rates amongst GPs compared to a simple control group. Another strength is that a lot of the techniques that have been demonstrated to improve response rates were implemented in this study. Also, nonresponse bias analyses demonstrated there was little nonresponse bias and age and gender distribution was similar to the rural general practice population. The major limitation of the study is the small sample size, which led to the study being underpowered; Assuming a power of 80%, significance level of 5%, and given our response rate of 62% in the intervention group and 47% in the control group, the estimated sample size required to detect a 15% difference was 340 or 170 in each group. This needs to be taken into account when interpreting the results. It is likely that a larger sample size would have led to statistically significant results. The study formed part of a larger pilot study and the sample size was therefore confined to the current sample. Another limitation of the study is not being able to link survey data to responders. However, it was felt that identifiable information would have limited response rates and GPs ability to answer the questions truthfully. Another limitation of the study is that the results may only be generalizable to GPs.
Comparison with existing literature
Edwards and colleagues  found that unconditional incentives were an effective strategy to increase postal survey response rates in the general population. The odds of responses also increased slightly when using non-monetary incentives. Non-monetary incentives also increased the odds of higher responses for electronic surveys. However, a systematic review found that the majority of physician surveys confirm that non-financial incentives (such as stickers, risk disks, pens or candy) are not effective . Three trials reported in this review used a pen or pencil as an incentive. Two of those studies found no effect. One Australian study compared four different methods of recruitment rather than using a control group , while another by Clark et al.  surveyed US obstetricians rather than family physicians. In line with our study, the third trial compared a pencil with a control group and found that the response rate was higher in the intervention group (32%) compared to the control group (17%) , whereas the authors also conducted a 2nd study comparing a pen versus a telephone reminder and did not find a significant effect. Recently a study amongst health care providers  showed that lottery tickets or non-financial conditional incentives did not lead to response rates above 60% but this included only three studies for each group. The authors call for improved reporting standards for health care provider surveys.
Despite the majority of general practices being computerised in Australia, no GPs completed the survey online. It is not yet well understood why physicians are less likely to complete online surveys . It may be that we had zero responses because the GPs had to enter the URL link themselves. However, other studies confirm the extremely low response rates for online surveys among general practitioners [2, 13, 14]. For example, in an Australian study, GPs were invited via an email newsletter and found that less than 0.1% opted to complete the online survey  while 12.4% completed the postal survey. VanGeest et al. conducted a systematic review of 66 studies  amongst physicians and also found that postal and telephone surveys were more effective than web based surveys. Additionally, 81% of Australian GPs have nominated mails surveys as their preferred mode of administration .
Implications for GP surveys
In conclusion, a small unconditional non-financial incentive, in the form of a pen, may improve response rates for GPs by an important margin.
Implications for GP recruitment research
Further study is required to focus on the efficacy of non-financial incentives in the GP population, given the rather limited and somewhat conflicting results in this area to date. The acceptability and uptake of electronic communication is rapidly evolving and the preference for paper-based surveys may change. Recruitment of GPs for primary care research will remain a challenge and the research community will need to continually update the evidence for participant recruitment. We recommend that researchers who survey GPs build in a randomised controlled trial to test different recruitment methods in their research planning and budget to advance this field.
The study was approved by the University of Sydney Human Research Ethics Committee (ID 14112).
Kellerman SE, Herold J: Physician response to surveys. A review of the literature. Am J Prev Med. 2001, 20 (1): 61-67. 10.1016/S0749-3797(00)00258-0.
VanGeest JB, Johnson TP, Welch VL: Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Prof. 2007, 30 (4): 303-321. 10.1177/0163278707307899.
Klabunde CN, Willis GB, McLeod CC, Dillman DA, Johnson TP, Greene SM, Brown ML: Improving the Quality of Surveys of Physicians and Medical Groups: A Research Agenda. Eval Health Prof. 2012, 35 (4): 477-506. 10.1177/0163278712458283.
Sallis JF, Fortmann SP, Solomon DS, Farquhar JW: Increasing returns of physician surveys. Am J Public Health. 1984, 74 (9): 1043-10.2105/AJPH.74.9.1043.
Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S: Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009, 3: MR000008
Brett TD, Arnold-Reed DE, Hince DA, Wood IK, Moorhead RG: Retirement intentions of general practitioners aged 45–65 years. Med J Australia. 2009, 191 (2): 75-77.
SAS Institute Inc: SAS. Version 9.2. 2010, Cary, NC, USA: SAS Institute Inc
Johnson TP, Wislar JS: Response rates and nonresponse errors in surveys. JAMA. 2012, 307 (17): 1805-1806. 10.1001/jama.2012.3532.
AIHW: Medical workforce 2011. National health workforce series. Cat. no. HWL 49. 2013, Canberra: AIHW, http://www.aihw.gov.au/publication-detail/?id?=?60129542627&tab?=?3. Date accessed: 10 July 2013
Ward J, Bruce T, Holt P, D'Este K, Sladden M: Labour-saving strategies to maintain survey response rates: a randomised trial. Aust N Z J Public Health. 1998, 22 (3 Suppl): 394-396.
Clark TJ, Khan KS, Gupta JK: Provision of pen along with questionnaire does not increase the response rate to a postal survey: a randomised controlled trial. J Epidemiol Community Health. 2001, 55 (8): 595-596. 10.1136/jech.55.8.595.
McLeod CC, Klabunde CN, Willis GB, Stark D: Health Care Provider Surveys in the United States, 2000–2010: A Review. Eval Health Prof. 2013, 36 (1): 106-126. 10.1177/0163278712474001.
Crouch S, Robinson P, Pitts M: A comparison of general practitioner response rates to electronic and postal surveys in the setting of the National STI Prevention Program. Aust N Z J Public Health. 2011, 35 (2): 187-189. 10.1111/j.1753-6405.2011.00687.x.
Bonevski B, Magin P, Horton G, Foster M, Girgis A: Response rates in GP surveys - trialling two recruitment strategies. Aust Fam Physician. 2011, 40 (6): 427-430.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2296/14/108/prepub
We thank the GPs who took part in the study and our Advisory Committee members: Dr. Christine Ahern. Dr. Nathan Kesteven, Prof. Lesley Barclay and Dr. Peter Honeyman. We also thank the Northern Rivers General Practice Network for collaboration.
This work is funded by a Sydney Medical School Early Career Researcher grant and the University Centre for Rural Health. Dr Sabrina Pit was funded by a NHMRC Research Training Fellowship (511998).
The authors declare that they have no competing interests.
SWP designed the study, collected data, analysed and drafted the manuscript. VH collected data, and participated in study design. DE participated in the design of the study and gave advice. All authors read and approved the final manuscript.