In his book, Method and Meaning in Polls and Surveys, 2008, Howard Schuman discusses the distinction between open and closed-ended questions. Many people think that the open-ended format is more valid because it allows respondents to say whatever is on their minds. However, the question stem in the open format may establish a frame of reference that is narrow and might precludes legitimate responses. Additionally, media and other influences may make some responses come to mind more readily in the open format.
To illustrate, let’s look an experiment conducted 1986 (Schuman, 2008). Respondents were randomly assigned to get either the open or closed-ended version of this question: “There have been a lot of national and world events and changes over the past 50 years—say from 1930 right up until today. Would you mention/choose the event or change that seems to have been the most important?” **
The responses are summarized below.
In the closed version of the question, 30 percent of respondents selected “Invention of the computer, “ while in the open version of the question, only 1 percent of respondents mentioned invention of the computer. Schuman argues that the wording of the question in the open format encourages respondents to think within a relatively narrow frame of reference. For this question, the idea of “national and world events” may have led respondents to think of specific political and dramatic events, not issues that are slower to evolve, such as the invention of the computer. Yet note that when given “invention of the computer” as an answer choice in the closed format, 30 percent of respondents chose it.
In another example, Schuman discusses an experiment where respondents are asked to indicate “the most important problem facing this country at present” in either an open or closed-ended format. In the open format, only 16 percent of respondents indicated “crime and violence” as the most important problem facing the country, whereas in the closed format 35 percent select “crime and violence” as the most important problem facing the country. Schuman argues that crime and violence are usually seen as local, not national issues, and “the closed question informs respondents that ‘crime’ can be a legitimate answer to the question.”
The open-ended format is not necessarily better or more valid than the closed-ended format. As we’ve seen in both examples, the wording of the question will call forth a particular mindset that may be too narrow for our purposes. One way to let respondents know the breadth of issues we want them to consider is to ask the question in a closed format and let the list broaden the range of issues to consider.
Open and closed-ended questions are different tasks for respondents. In the open-ended question, respondents write down what is readily available in their minds, whereas in the closed-ended question we have respondents focus their “attention on specific responses chosen by the investigator,” (Converse and Presser, 1986). It is important to recognize the strengths and limitations of each format when we design surveys.
** I have revised this question slightly to make this blog post brief.
Converse, Jean M., and Stanley Presser. 1986. Survey Questions: Handcrafting the Standardized Questionnaire, Beverly Hills, CA: Sage.
Schuman, Howard. 2008. Method and Meaning in Polls and Surveys. Cambridge, MA: Harvard University Press.
