Qualitative research is essential for organizations to understand their customers, and reach their potential. There are many ways to improve the quality of qualitative research—from how we design topic guides to how we conduct interviews and focus groups. One step in the quality process that is often overlooked and taken for granted is screening.
Writing a screener that gets you the right respondents is no simple task. The purpose of this article is to present seven guidelines for writing effective screeners. These guidelines are adapted from our book, The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions, by David Harris & Ron Tatham (in press).
1. Determine the information needed before writing the questions. Too often people jump to write questions without carefully thinking through what information they really want. We saw the following question, “Is your dog a vegetarian?” and had to wonder what information they really needed. It turned out they wanted to talk to people who do not feed their dog meat.
We saw the following question, “How interested are you in Product Y?” and asked the clients what they really wanted. They said they wanted to know how likely respondents were to purchase the product. Well then why not ask, “How likely are you to purchase Product Y?” Interest and likelihood to purchase are different things, and thinking through the information we want before deciding how to ask the question is good practice.
The information you need, and how to ask the question, are two different thought processes. Jumping straight to writing questions often gets us information that really isn’t the information we want. Determining the information needed before writing the questions also results in questionnaires and screeners that are, on average, half as long.
2. Make sure your question is really only one question. Consider the following question: “Do you think the government should provide tax incentives and reduce regulations to encourage small business growth?” One respondent thinks government should produce tax incentives but not reduce regulation, yet another respondent is against tax incentives but favors regulation. Some will say yes when they agree with only one of the items, while others will say no. This is what we call a “double barreled” question, and it can easily be made into two questions.
Here’s another example:
Please indicate whether you agree or disagree with the following statement:
I am concerned about my health, so I often take vitamins.
__ agree
__ disagree
One respondent might think, “I am concerned about my health, but I don’t take vitamins.” Another might think, “I am not all that concerned about my health, but I do take vitamins.” Either of these two respondents might agree or disagree with the statement because it contains two ideas.
Double-barreled questions are all too common in rating scales, where people are asked to rate items such as:
“How important are each of the following issues to you at our company? Please use the scale below where 1 = not important and 5 = very important.”
Commitment to learning and continuous improvement 1 2 3 4 5
Commitment to learning and commitment to continuous improvement are two different things! Just make them two different questions, and respondents will know what to do, and you will know what they think.
If your question is really two or more questions, respondents do not know how to respond, and you won’t know how to interpret the answer. Make sure each question is really only one question. This will help ensure your screener is getting the right answers from people, so you can then have the right respondents for research.
3. Describe concepts clearly and concretely. All too often what seems like a straightforward question is ambiguous. Consider the question, ‘How many customers do you have?” Do we mean people who visited the store, people who purchased something, or regular customers? And what time frame do we mean? Are we asking about this month, the past year, or since the business started? It isn’t clear. Determining the information you really want first will help you write a better question.
What about the question, “In the past 30 days, have you experienced a computer crash?” The respondent wonders if we mean at work or at home? What constitutes a computer crash anyway? Try this instead: “In the past 30 days, has your home computer had any problems that resulted in the need for a repair?”
Or what about this question for physicians: “How many times per year does the typical depression patient ask for a specific antidepressant? What is a “typical depression patient” anyway?”
Defining concepts clearly and concretely will help respondents understand what you mean, and you will end up with more of the right respondents in research.
4. Be careful about asking for percentages. Many screeners want people who spend a certain amount of time engaging in an activity. Here is a question where the client wanted doctors who spend at least fifty percent of their time in clinical practice. “What percent of your time is spent in clinical practice?” One doctor calculated the percentage by using 8 hours of his working day as his denominator, while another doctor calculated the percentage by using all 24 hours of the entire day.
If you are going to ask for a percentage, make sure the denominator is clear. Maybe ask, “What percent of your professional time is spent in clinical practice?” For many other questions, we suggest asking for a unit of measurement instead of a percentage, and let the computer calculate the percentage (e.g., “In past 5 business days, how many hours have you…”).
5. Use the vocabulary of the people you want to recruit. Avoid industry jargon and technical terms. Just because we know what we are saying doesn’t mean anyone else does. Instead of asking people if they have hypertension, ask if they have high blood pressure. I had a client who wanted us to ask doctors about the last time they talked to a Therapy Area Specialist. That was their name for a sales representative. No doctor had ever talked to a Therapy Area Specialist, but almost all of them recalled talking to a sales representative!
Use the vocabulary of the people you want to recruit. By having a collaborative relationship with your recruiting team, they can explore how to ask questions in their first few calls. Just make sure they know what information you are seeking, not just how you want to ask the question.
6. Keep questions under 25 words, and make the questions conversational. Longer questions are much harder, if not impossible, to process in one reading. To make questions shorter, try saying them out loud as if you were talking to someone. We find that helps us make questions less wordy and technical.
Instead of this: How much of the household shopping, if any, do you yourself, usually do? By household shopping, we mean all of the food, cleaning, paper, personal care (like health products, cosmetics, etc.) pet products, and other household products you usually buy?
Say this: How much of the household shopping do you yourself do for products such as food, toiletries, cleaning items, health products, and cosmetics?
Most questions can be under 25 words if you know what you want, and you work on simpler more conversational way of asking for it.
7. Write short screeners. Sometimes screeners are too long. Potential respondents get frustrated and wonder if we really know what we are doing. If you carefully determine what information you need, and create questions according to principles of questionnaire design, you will end up with shorter screeners that get the right respondents. Most screeners should have fewer than ten questions.
Writing an effective screener is challenging. It requires questionnaire design expertise, and knowledge of how the target audience thinks about the subject matter. Learning the principles of questionnaire design will help. We also suggest collaborating with your fielding agency on who you want to recruit, and the information you need before writing questions. Attention to these issues will improve the quality of recruits, and therefore the quality of your research.