The Harris Poll® #113 , November 13, 2007

"Margin of Error", When Used by Pollsters, Is Widely Misunderstood and Confuses Most People

Many media and pollsters, when releasing new poll results, include statements such as "the margin of error for this survey is +/- 3 percent". A new Harris Poll was developed to measure the public’s understanding, or misunderstanding, of the phrase "margin of error" when used to describe opinion polls. It found that these words are misunderstood by most people. Arguably they confuse more people than they enlighten, and they suggest a level of accuracy that no statistician could justify.

These conclusions are based on a Harris Poll of 1,052 U.S. adults surveyed by telephone between October 16 and 23, 2007 by Harris Interactive®.

This number is actually a purely theoretical calculation of what the likely maximum error (at a 95% confidence level) would be if the survey had used a pure probability sample with a response rate of 100% and there were no other possible sources of error. In the real world of polling there are several other sources of error that may sometimes be larger than this theoretical calculation of sampling error, and there is no good way to calculate them. However, a new Harris Poll shows that most people do not understand this.

There are a number of other possible sources of error in any poll which include:

• Non-response errors - Pollsters often do not complete interviews with most of the people they intend to survey because they are not available or refuse to be interviewed;
• Errors due to question wording or question order. The answers to questions are sometimes influenced by such things as how the questions are posed, what questions were asked earlier in the survey, or which responses are presented to the respondent, among other things;
• Errors due to interviewers. Interviewers sometimes influence, often unconsciously, the answers given by the people they survey (e.g. social desirability, acquiescence bias, researcher expectancy effects, etc.);
• Weighting errors - Most polls are "weighted" statistically to compensate for demographic and other biases in the survey sample; this is an imperfect process. Weighting the data can cause errors in the results.

The magnitude of these sources of error is impossible to calculate and they are therefore not included in the so-called "margin of error" calculations reported by polling organizations.

Some of the key findings of this Harris Poll are:

• A 52 percent majority of all adults believes wrongly that statements about "the margin of error being plus or minus 3 percent" means "that all of the results of the survey are accurate to within a maximum of 3 percent given all types of error";
• A 66 percent majority of adults believes wrongly that the words "margin of error" includes calculation of errors caused by "how the questions are worded";
• Large minorities believe wrongly that the calculation of the "margin of error" includes "errors in developing a representative base or weighting errors (45%), mistakes made by interviewers (45%), and errors because of where the questions are placed in the survey (40%);
• Only a very small 12 percent of the public agrees that the words "margin of error" should only address one specific source of error, sampling error — as they almost always do;
• A 56 percent majority believes that statements about "margin of error" do not make it clear that this calculation excludes all sources of error except for sampling error for a pure random sample.

At the end of this survey, the adults interviewed were asked if pollsters should use the phrase "margin of error" given the impossibility of calculating most possible sources of error. Surprisingly, in light of their other responses, a 52 percent to 40 percent majority thinks that they should. This may reflect their thinking that it is important to point out that polls do not claim to be infallible.

There are several important conclusions that can be drawn from this Harris Poll:

1. The use of words such as "margin of error" is controversial because they are often used when reporting telephone polls even though it is not possible to calculate a real margin of error.
2. Pollsters need to do a much better job of explaining all the possible sources of error in their polls not just a theoretical sampling error, which does not take into account of other, potentially substantial, sources of error;
3. The accuracy of opinion polls should be judged empirically by the accuracy and reliability of their findings, not on a theoretical basis when there is no way to calculate a real margin of error;
4. Pre-election polls should continue to be trusted only so long as their final forecasts are reasonably accurate, not because they are theoretically "scientific" (since there is no means to establish that they are);
5. The words "margin of error" should probably not be used at all in conjunction with polling results.

The Harris Poll has not used the phrase "margin of error" for many years. In our standard methodology (see below) we explain why not and we seek to educate our readers about the different sources of error which may make our surveys inaccurate.

"When you see the statement about "margin of error of a survey being plus or minus 3 percent" do you think this statement means all of the results of the survey are accurate to within a maximum of 3 percent given all types of error or something else?"

Base: All Adults

Note: Percentages do not add up to 100% due to rounding

"When you see the phrase "margin of error" what types of possible sources of error do you believe it includes?"

Base: All Adults

Multiple responses allowed

"Other sources of error in opinion polls include things like the wording of questions, the order the questions are in, interviewer bias and weighting. If the words "margin of error" are used, should they …?"

Base: All Adults

"In fact, pollsters can only calculate the probability of error due to one source of error – that a randomly selected sample may not be representative. Statements about "margin of error" do not take into account any of the other sources of error which may cause a poll to be inaccurate.

Do you think the use of the phrase "margin of error" makes this clear?

Base: All Adults

"Given that it is impossible to calculate the possible error due to most factors, do you think pollsters should or should not use the phrase "margin of error"?

Base: All Adults

The Harris Poll^® was conducted by telephone within the United States between October 16 and 23, 2007 among 1,052 adults (aged 18 and over). Figures for age, sex, race/ethnicity, education, region, number of adults in the household, size of place (urbanicity) and number of phone lines in the household were weighted where necessary to bring them into line with their actual proportions in the population.

All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words "margin of error" as they are misleading. All that can be calculated are different possible sampling errors with different probabilities for pure, unweighted, random samples with 100% response rates. These are only theoretical because no published polls come close to this ideal. These statements conform to the principles of disclosure of the National Council on Public Polls.

J31986

Q650, 655, 660, 665, 670