One of the most common tasks in survey research is to ask respondents to select from a list of numeric ranges. For example, we might ask how many hours per day do they watch TV, how much time do they spend on your computer, or how many hours per day they study. Several authors have shown through experimentation that the range of response options we offer respondents has a substantial impact on their answers.
For example, J. D. Smyth, D. A. Dillman, and L. M. Christian, 2007*, conducted an experiment where students were randomly assigned to receive the following question:
“How many hours per day do you typically study?”
One group had a Low Range of response options; another had a High Range of response options; while another was given an Answer Box to write in their response. The response options are listed below.
| Low Range | High Range | Answer Box |
| ___ Less than .5 hours per day ___ .5–1 hour ___ 1–1.5 hours ___ 1–2 hours___ 2–2.5 hours ___ More than 2.5 hours |
___ Less than 2.5 hours ___ 2.5–3 hours ___ 3–3.5 hours ___ 3.5–4 hours ___ 4–4.5 hours ___ More than 4.5 hours |
 |
If all that mattered were the question stem, results from the three groups would be the same. However, as a number of researchers have shown, respondents look to the answer choices to gauge their responses.
Below is a table showing some of the results.
| Low Range | High Range | Answer Box | |
| Up to 2 ½ hours | 70% | 29% | 42% |
| Over 2 ½ hours | 30% | 71% | 58% |
So how many hours per day do students typically study? If you want to know what percent study up to 2 ½ hours per day, it is either 70%, 29%, or 42%, depending on the range of response options you offer. If you want to know what percent study over 2 ½ hours per day, it is either 30%, 71%, or 58%, depending on the range of response options you offer. Clearly, the range of answer choices offered to respondents affects responses. This issue has been studied with other activities, such as computer usage and watching TV.
Several authors have suggested ways to address this effect. You should always have the range of response options reflect the distribution of the real answers in the population. If you don’t know, then start with an answer box to eliminate influencing answers by the range of response options you offer respondents. You might start with an answer box the first time you ask this question then use those answers to determine the range of responses for subsequent studies.
This is one of many experimental effects that all of us survey researchers should know. I am working on a paper on the topic of the top ten experimental effects all survey researchers should know. If you have ideas of other experiments that should be on the list, please let me know.
* “Context Effects in Web Surveys: New Issues and Evidence,” by J. D. Smyth, D. A. Dillman, and L. M. Christian, 2007, in A. Joinson, K. McKenna (pp. 427-443), New York, NY: Oxford University Press.