Why concern yourself?

A PLEA FOR THE IMPORTANCE OF ASKING GOOD QUESTIONS

What is the need for this first book on question wording? No one else has considered it necessary to devote a whole book to the subject. A chapter or two has always seemed enough before. Articles of several pages frequently appear in professional journals here and there. What more can a book do for question wording? If it all boils down to the familiar platitudes about using simple, understandable, biasfree, non-irritating wordings, all of us recognize these obvious requirements anyway. Why say more?

Oblivious of the obvious

One reason for elaborating on the subject is that all of us, from time to time, forget these requirements. Like some church-goers, we appear to worship the great truths only one day a week and to ignore them on working days. Or we remember a certain example, but fail to see how it applies to other situations. In combatting our very human frailty, a more provocative set of examples and a detailed list of points to consider may be more helpful than the isolated examples and the broad generalities which we now so often disregard.

The import IS important

Another reason for separate treatment is to help us realize how basic the phrasing of questions is to worthwhile research. As it is, we may look upon the few available examples of differences wrought by wording as mere freak occurrences. There is no danger of our making such weird mistakes, we think. Besides, if the question "works," it must be a good question.

By now, we should realize that the fact that something "works" does not mean that it works correctly. The Literary Digest poll seemed to "work" all right until 1936. The methods of subsequent election polls seemed to "work" all right until 1948. In both of these cases, attention has concentrated on sampling difficulties as causes of the wide margins of error. In the case of the Digest the faulty sample was no doubt the prime contributor to the error. At least one expert had predicted how and why this faulty sample would lead to error (1). In the more recent case, sampling is only one of the many possible causes, according to the Social Science Research Council committee (2).

Be that as it may, people — laymen and qualified experts alike — are impressed that sampling is an important feature of the survey method. Even if we agree that they are right, however, let us not forget that the Gallup Poll in 1948 overestimated the Dewey vote by only 4.4 percentage points, and the Digest poll in its 1936 prediction of the Landon vote was only 19 percentage points too high.

I use the word "only" advisedly because, as we shall see, survey differences resulting from changes in words or phrases sometimes amount to considerably more than a 19 point error! Perhaps, if we grant that by emphasizing the importance of sampling the election upsets have contributed to advances in sampling techniques, we should deplore the fact that nothing similar has happened to concentrate attention on the importance of question wording.

Tens versus tenths

At the present stage of development of the survey method, improvements in question wording and in other phases can contribute far more to accuracy than further improvements in sampling methods can. I don't mean that the sampling experts should stop seeking further improvements, trying to knock a few more tenths of a percent off the statistical error. But, while they are laboring with tenths of a per cent, the rest of us are letting tens of per cents slip through our fingers. As Frederick Stephan of Princeton University has remarked, "It's like using a surgeon's scalpel in a butcher shop."

The experts said so

Even as far back as 1936, however, when the Literary Digest was riding to its fall, a group of experts called question wording the Number One problem. Howard T. Hovde asked a sample of researchers what they saw as the principal defects of commercial research (3). Here are their most frequently mentioned criticisms:

Improperly worded questionnaires 74%
Faulty interpretations 58
Inadequacy of samples 52
Improper statistical methods 44
Presentation of results without
supporting data 41

Since three experts in every four pointed their fingers at question wording, it seems that the subject should have been worthy of concentrated attention. The specialists in statistics and sample theory certainly didn't let these expert opinions dampen their efforts. It's too bad that question worders weren't more stimulated by Hovde's report.

Samuel A. Stouffer and his collaborators in their recent monumental work on the American soldier (4) came to a similar conclusion: "To many who worked in the Research Branch it soon became evident that error or bias attributable to sampling and to methods of questionnaire administration were relatively small as compared with other types of variation — especially variation attributable to different ways of wording questions."

Jack-of-all-trades

Probably the reason that the question worder hasn't done more to advance his phase of research is that he just doesn't exist, at least not as a specialist. The statistician is the only one among us who has a specialty. All the rest of the work comes under the jurisdiction of a jack-of-all-trades. This man's job is to develop the questionnaire, pretest and revise it, have it printed, select, train, and supervise the interviewers, conduct the survey, analyze the results, write the report, and present the findings. His attention is necessarily divided. Question phrasing is but one part of the complex machinery which he must put together and operate. Small wonder that this all-around type of researcher has not had much time to formulate detailed statements on question wording.

Not special pleading

Having said that sample theory has moved ahead through specialization, I do not mean to imply that question wording needs also to be treated as a specialty. In actual practice, wording cannot be thought of as being in a vacuum apart from other phases of the survey method. It would be extremely naive to expect a single question to provide "some magic way of reducing a complex matter of people's attitudes, wishes, and aspirations to some simple wording which will not bias the returns." Most researchers will agree with Daniel Katz (5) that the solution usually requires "an integrated questionnaire which explores the problem comprehensively from many angles."

The fact that this book deals with the subject of question wording by itself and usually in terms of improving a single question does not mean that the other phases of the work can be slighted. The steel square is a useful tool to the carpenter and much can be said about its proper use without denying his need for jack-plane, saw, hammer, nails, and so forth. Several books are available on the use of the steel square. It is in much the same way that I advocate paying serious attention to question framing. For people who are also interested in the more general features of the questionnaire technique, I might suggest Albert Blankenship's first book, Consumer and Opinion Research (6).

All types of surveys

The importance of wording is not restricted to any single type of survey. The mail questionnaire presents much the same problems of wording as the personal interview. The so-called "factual" survey is in need of careful wording just as the attitudinal or opinion survey is. Although opinion surveys present the greatest variety of examples, experience in factual or census-type enumerations also furnishes enough examples to show that facts as reported in answer to questions are not always the facts that exist. We shall draw upon experience with all these types of surveys here, but most of the discussion will be based upon opinion surveys conducted by personal interview. First, then, let us look at some opinion questions.

Implied alternatives

Sometimes the questioner assumes that the negative side of the question is so obvious that it need not be stated. He may simply ask:

Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year?

63% said companies could avoid layoffs,
22% said they couldn't, and
15% had no opinion.

The alternative here seems to be so implicit in the question that it need not be stated. Either companies could avoid layoffs — or they couldn't. No other interpretation seems possible. But what happens when we take the trouble to state an alternative to another carefully matched cross section of respondents?

Do you think most manufacturing companies that lay off workers in slack periods could avoid layoffs and provide steady work right through the year, or do you think layoffs are unavoidable?

35% said companies could avoid layoffs,
41% said layoffs are unavoidable, and
24% expressed no choice.

So, a few words changed here and there and explicit statement of the other side of the picture results in a 28 per cent falling off from the affirmative side of this question! This suggests the need for definite expression of the alternatives. Most of the discussion about alternatives, however, can be postponed to later chapters where we will not be so intent upon demonstrating the basic importance of wording.

Three little words

When we see the three words — might, could and should — together, we realize that they have somewhat different connotations. Yet when it comes to stating questions, we may sometimes use these words as synonyms. The trouble with this assumption is that the public actually does see distinctions among these words and changes its replies to fit. Here are the results of an experiment in which all things other than these words were kept equal. A casual reader might not detect the differences among these three questions:

Do you think anything should be done to make it easier for people to pay doctor or hospital bills?

Do you think anything could be done to make it easier for people to pay doctor or hospital bills?

Do you think anything might be done to make it easier for people to pay doctor or hospital bills?

But enough respondents understand the distinctive feature of each connotation that significant differences show up in the replies of three matched samples of people: 82% said something should be done, 77% said something could be done, and 63% said something might be done. The two extremes, should and might, come out 19 per cent apart. This is the same amount by which the Literary Digest poll missed the 1936 election. Again we see that wording is as important as sampling, at least in opinion polls.

The issue at question

Question wording involves more than toying around with this word and that to see what may happen, however. It is more than a mere matter of manipulation of words to produce surprising illusions. The most critical need for attention to wording is to make sure that the particular issue which the questioner has in mind is the particular issue on which the respondent gives his answers.

If we refer again to the three questions using might, could, and should, we observe that they pose different issues. The should wording brings up the moral issue of need in the sense of, "It's a crying shame! Something should be done about it." The could wording poses the issue of possibility, "Yes, but could anything be done?" The might wording moves to the issue of probability, "Maybe it could, but it might or might not be done." The results show that more people see the moral need for an easier payment method than grant the possibility of doing anything about it and that even fewer people think such a method is likely to be put into practice. These are three distinct sets of basic content. Yet any one of them might inadvertently have been used alone to cover any or all of the three contents.

To assure that the intended issue is understood, then, is a fundamental function of question wording. A large share of our discussion will be given over to ways and means of reproducing in the minds of respondents the same issues that are in our thinking.

Census "facts"

Now, for some examples of the importance of question phrasing in factual surveys. The U.S. Bureau of the Census no doubt produces more facts than any other organization anywhere. Its experts consider question wording an important feature of securing this wealth of information. They know that such facts as the number of employed workers, the average number of bedrooms in homes, and even the size of the population itself are susceptible to different interpretation according to the skill used in devising the questioning techniques.

Baby counting

Would you believe that in the decennial Census there is danger of undercounting the population because some parents neglect to report their infant children? This very real problem arises not so much because of any lack of pride in the new offspring, but apparently because the parents are not yet used to thinking of the new little "it" as a person. Consequently, the question was asked at every dwelling: Have we missed anyone away traveling? Babies? Lodgers? Other persons staying here who have no home anywhere else? The enumerator was also instructed to "be sure to include ... all children, even the very youngest." In addition, he received an extra 10 cents for each Infant Card, which was to be filled for every child born in January, February, or March 1950. These precautions probably helped in the recording of thousands of infants who might otherwise have been overlooked (7).

Found: 1,400,000 workers

Take the problem of estimating the millions of persons at work in the nation, as done in the Monthly Report on the Labor Force of the Census Bureau. Gertrude Bancroft and Emmett Welch have explained how a change in the questioning technique affected these estimates (8). Prior to July 1945 a single question was used. It asked, Was this person at work on a private or government job last week? Beginning in that month, two questions were substituted. The first of these merely asked what the person's major activity was during the preceding week. If the major activity was something other than working, the enumerator then asked whether in addition the person did any work for pay or profit during that week.

The upshot of this change in questions was that in the trial when both versions were used, the new questioning showed an increase of 1,400,000 employed persons over the old wording. About half of these additional workers had worked 35 or more hours during the week under consideration!

Found: 500,000 unused bedrooms

This next example is more one of change of definition than of the wording of the actual question, but in factual surveys the two so often go hand-in-hand that they may well be taken together. During World War II the Census Bureau conducted hundreds of surveys for the National Housing Agency. Among these were some for the Homes Use Service, which was interested in inducing home occupants in critical areas of housing shortage to make rooms available to war workers. The delicate problem posed to the Census Bureau was to determine how many bedrooms were not in use as sleeping rooms in these areas.

This required asking a representative sample of home dwellers how many bedrooms they had and how many they were using for sleeping. It was necessary to explain that for the purpose a bedroom was any room except the living room, dining room, kitchen or bath, which contained a bed or which with the addition of a bed could be used as a sleeping room. When it came to asking how many of these bedrooms were in use, it was necessary to emphasize use by the occupants for sleeping. Excluded from this use were guest rooms; bedrooms used only as studies, sewing rooms, storage rooms, or playrooms; and bedrooms reserved for the use of absent members of the family such as students away at school.

Surveys in 83 critical areas showed a total of 500,000 bedrooms not in current use for sleeping. Only a sixth of these had previously been offered for rent. Publication of these results, area by area, helped greatly to spur room registrations (9).

This was a case, often duplicated in other surveys, where the respondent was asked to consider a situation in somewhat unfamiliar terms. That playroom or sewing room had never been used as a bedroom! Yet for this survey it was called a bedroom. This survey was different from many in that the researchers recognized that they were inverting normal thinking and went out of their way to define just what was wanted. In other studies we sometimes seem to forget that we are dealing with an unfamiliar situation or in unusual terms.