How To Make Your Survey Count

Research during residency often seems like a daunting task, given the limited amount of time we have to accomplish it. For this reason, many of us choose to create surveys.

The method used to conduct data collection, as well as the quality of questions asked, will determine the accuracy and reliability of the data.^1,2 This article aims to give you practical guidance on ways to improve the quality of the data and make your research more fruitful.

When to Use a Survey
While a survey seems like an obvious choice for research during residency, it is not applicable to every situation. Surveys are best utilized when researching human phenomena such as emotions or opinions.¹ Surveys can also be used when the data you want to analyze is not directly observable or available in documents.¹ Additionally, surveys may be used when directly obtaining the data is not practical.¹

Type of Survey to Use
The three primary types of data-gathering methods described here are phone surveys, face-to-face interviews, and written surveys.

Phone surveys have a higher completion rate and some inherent quality control because the interviewer can clarify survey questions that the responder does not understand.³

Face-to-face interviews are best for complex topics.⁴ They have the highest response rate but are also the most expensive.³ With both phone surveys and face-to-face interviews, data might be less reliable because participants may be biased to choose a more socially desirable response.³

Written surveys have lower response rates but are the easiest to standardize because they avoid differences between interviewing techniques.⁴ Written surveys are also less expensive and more convenient because they do not require an interviewer.

Writing Questions
Once you have decided that a survey is the best method to obtain the data you’re looking for, and you have determined which type of survey to use, the next step is to write the questions.

When you start writing questions, it is important to keep the answering process in mind. Typically, responders first interpret the question and try to determine what is being asked. Responders then search their memory for relevant information and try to fit it into a single response. Lastly, if response choices are offered, they try to find their response among these options.⁵ The less effort required by the respondent, the more accurate the answers will be.⁶Furthermore, accuracy of responses decreases with increased task difficulty, the respondent’s ability to answer a specific question, and lower motivation level.

Response choice order has important implications for the way respondents answer. For instance, in written surveys, respondents are more likely to choose the first choices, whereas in verbal surveys, they are more likely to choose the last option.⁶ One way to try to combat the bias of either choosing the first or the last response based on the survey technique is to change the order of the responses when giving individuals the same survey.^2,5

Consider using “filter” questions, so that if a question does not pertain to the respondent, they can skip to the next question. This way, respondents won’t answer questions that don’t pertain to them; if they did, it could affect the accuracy of your data.⁵

Surveys that include “I don’t know” as a possible response may limit the reliability of the data. While this option encourages people who “do not know” to avoid choosing a random answer and skewing the data, it also is an easier choice for less motivated respondents. Furthermore, when an “I don’t know” answer choice is added, data quality does not improve.⁵ Therefore, it may be best to avoid this option altogether.

Phone surveys should be limited to 15 minutes or less to avoid participant dropout. Question-and-answer choices must be simple and clear because the responder does not have the opportunity to read the questions.³

Question Order
The order of questions can help increase participant engagement and accuracy of responses.

Initial questions should be strongly tied to the survey’s overall topic and purpose, thereby piquing participant interest.^5,6 Start with easy questions to help build rapport, and then transition to more sensitive or difficult questions.^2,4,5 To decrease respondents’ cognitive burden, organize your survey so that questions belonging to the same category are grouped together. Also group together “how often” and “how much” types of questions, especially within categories.^5,7

Consider respondents’ motivation for the survey. Participants are more likely to be motivated if they benefit from survey results, have a desire for self-expression, perceive the survey as an intellectual challenge, and/or wish to help the survey sponsor. Respondents may be less motivated if the survey is required. Other participants may be motivated at the start but become tired or disinterested and therefore less motivated as they continue the survey.⁵ If respondents are less motivated, their most accurate answers will be at the beginning of a survey, so it might be best to place less pertinent demographic questions at the end.^5,6 On the other hand, highly motivated participants may actually increase the accuracy of their responses with more questions.⁶

Testing Your Survey
Prior to administering a survey for research data collection, it is important to review the survey for comprehensibility, participant burden, and validity.³ This can be accomplished by having an expert in the field you are researching review your survey to identify confusing questions or answers and determine which, if any, terms or phrases require definitions or explanations.⁷ Software programs are also available to evaluate questionnaires and surveys for common errors.⁵

The next step would be to test your survey on a small sample of the relevant population under conditions that are close to or identical to those of the actual survey.⁵ For verbal surveys, consider cognitive interviewing where the respondent is asked to think out loud about how they would answer the question, and verify that the respondent’s comprehension of the question matches your intention.^5,7 After the survey has been completed, debrief with respondents to evaluate the time it took them to complete the survey, identify questions that arose during completion, and note other ways to improve the survey.⁷

Recruiting Subjects
Recruiting subjects for surveys can be difficult. It is important to tell potential subjects why you are doing the survey, why they were chosen to be part of the study, what is expected of them, and what they will gain from the experience.⁴

Increasing Response Rates
To increase the response rate, make sure that instructions are clear and that the survey is as brief as possible to obtain the appropriate meaningful data.³ Choosing a topic that is highly relevant to your sample population will increase motivation.³ If ethically and financially feasible, consider giving incentives and rewards for participation.^3,7Lastly, consider sending out repeated reminders for non-responders. ^3,4

Data Interpretation
Because the average response of online surveys is 20-30%, it is important to consider how non-responders will affect the results (validity, generalizability, etc).^3,4,9 To reduce or eliminate this non-responder bias, consider trying to sample a small group of non-responders using vigorous efforts (for example, messaging or calling them), to see if their responses are different from the responder results.⁴ Take into account the demographics of the non-responder population compared to the responder population to determine if that could potentially affect your data from the survey versus the true population data.⁴

Finally, when reporting and presenting the data obtained in a survey, be certain to include details related to the persons asked to participate in the study, including how they were contacted, response rate, and any strategies used to increase response rate. A description of the survey and a copy of the actual survey should be included, if applicable and if space allows.^3,9

Surveys can be great research tools to obtain data when carefully designed and correctly administered. When writing a survey, consider the best method to administer it, take into account participant motivation, and develop questions that maximize participation and generate valid and meaningful responses.

References

1. Phillips AW. Proper Applications for Surveys as a Study Methodology. West J Emerg Med. 2017;18(1):8-11.
2. Bite-Size Guide to Patient Insight: Writing an Effective Questionnaire. NHS England. 2018. Accessed 19 May 2020.
3. Safdar N, Abbo L, Knobloch M, Seo S. Research Methods in Healthcare Epidemiology: Survey and Qualitative Research. Infect Control Hosp Epidemiol. 2016;37(11):1272-1277.
4. Coggon D, Barker D, Rose G. Epidemiology for the Uninitiated. 4th ed. 1997.
5. Krosnick J, Presser S. Question and Questionnaire Design. Web.stanford.edu. 2009. Accessed 17 October 2020.
6. Duke Global Health Institute. Five Tips for Designing an Effective Survey. 2018. Accessed 19 May 2020.
7. Proeschold-Bell R. How to Write a Good Survey Item [video]. 2017. Accessed 27 May 2022.
8. Gehlbach H. Seven Survey Sins. The Journal of Early Adolescence. 2015;35(5-6):883-897.
9. Bennett C, Khangura S, Brehaut J, et al. Reporting Guidelines for Survey Research: An Analysis of Published Guidance and Reporting Practices. PLoS Med. 2011;8(8):e1001069.