Pretest your questionnaire with enough respondents per segment to ensure you find and fix any problems with the instrument. Proper pretesting involves interviewing respondents to find out what questions mean to them, how they came up with their answers, and to uncover any problems with the questionnaire. You would use techniques such as those listed below:
- Ask respondents to tell you in their own words what they think each question is asking. This way you will find out if the questions are working as intended.
- Ask respondents how they came up with their answers to specific questions. This will help you see if they struggled with the question, or if there was ambiguity about how to answer it.
- Find out what comes to mind when respondents think about certain terms and phrases in the questionnaire. For example, ask: “What does the term ‘family member’ mean to you?” Often we find that respondents have different definitions of terms and phrases than we do.
- Probe to find out if questions were hard to answer, or why they took a little longer than expected to answer a question.
These sorts of questions reveal all sorts of problems with questions that we had no idea existed.
The “Think Aloud” Method for Pretesting
The think-aloud method for pretesting asks respondents to verbalize their thought process as they try to answer each question.
In this example, the interviewer reads the question, and the respondent verbalized his thought process as he tried to answer the question. This example reveals the importance of pretesting.
Interviewer: “How many cars do you currently own?”
Respondent: “We have three cars, in that I have a car, my wife has a car, and we have a car for our nineteen-year-old son, but he is away at college, so I am not sure if you want me to count that one. We only own one of them, though, and that would be the one that my son drives. The car that I drive is actually a truck, but I assume you want to know how many vehicles we drive, right? The car my wife drives is actually a company car, but we use it as a family car as well, so I am not sure if you want me to count that one. So I guess I’ll just say two, but I don’t know if that is the right answer.”
Think-aloud is a useful method for peeking into mental processes. This example reveals several problems with the question. Both the concepts of car and ownership are ambiguous and need clarification. The term, you, is also unclear to this respondent. He doesn’t know whether to count his son’s car. Clearly, pretesting revealed a host of problems with a seemingly simple question.
The problem we face in survey research is that many companies do not properly pretest their questionnaires. Their version of pretesting involves sending questionnaires out and seeing if they come back with answers. That is not properly pretesting the questionnaire.