Conducting a Survey – What To Do and What NOT To Do

The results of a survey about conspiracy theory beliefs are making the rounds as a fluff piece in the media lately.  The survey was conducted by Public Policy Polling and asks 20 questions about beliefs and six questions at the end about political and demographic classifications.  The press release contains several paragraphs based on crosstabulated findings along with words like “bizarre” and “crazy.”  http://bit.ly/XGVrM2.  Although surveys can certainly be used for entertainment purposes, this project can provide some lessons for what not to do when conducting a survey.

AVOID AMBIVALENCE.

The first 20 questions each begin with “Do you believe…”  The only possible answers (with one exception) are “Do,” “Do not,” and “Not sure.”  It is not clear whether “Not sure” means you are taking a middle view (“maybe/maybe not”) or have no knowledge on the subject (“don’t know anything about it”)?  All scale elements should be clearly defined and align appropriately with each question.  These 20 questions could have been set up as a battery (which also allows the items to be easily rotated to avoid any order-effect).  A single lead-in instruction could then define how the scale would be applied to each assertion.

MAKE SURE THE INTENDED MEANING IS CLEAR.

Question 1 asks, “Do you believe global warming is a hoax, or not?”  What answer should a respondent give if he or she suspects that academics have jumped to a conclusion, but have not done so deceptively?  Technically, they should have answered “Do not.”  Many of those people may have chosen “Do” as the closest answer to their beliefs.  This is especially true given that this is the first question and the respondent has not yet understood the nature of the survey or how the word “hoax” should be interpreted.  The results to this question may be measuring one’s acceptance or skepticism of the theory itself and not academic dishonesty.

BE PRECISE.

Question 10 asks, “Do you believe aliens exist, or not?”  Two different cohorts could legitimately answer “Do” to this question.  The first cohort is comprised of people who believe that aliens visit the earth in space ships.  The second cohort is comprised of biologists who speculate that simple alien life forms could exist throughout the universe.  A more precise question is “How likely is it that the earth is being visited by an intelligent alien race or races?”

AVOID LOADED TERMINOLOGY.

The phrase ‘conspiracy theory’ itself is a negative term since it is always used in the context of dismissing someone’s ideas.  No one would ever call Osama bin Laden’s conspiracy to destroy the World Trade Center a ‘conspiracy theory.’  If the CIA is suspected to have conspired in overthrowing a government, that is not called a ‘conspiracy theory’ either.  Unfortunately, there is no accepted neutral term.  The questionnaire asks ‘Do you believe…”  Beliefs and theories are not synonymous.  ‘Conspiracy belief’ would at least have been an attempt to provide some neutrality.

DO NOT CONVOLUTE FACT AND OPINION.

The set of belief questions covered a wide range of topics.  The possibility of Bigfoot’s existence does not constitute a conspiracy.  Several other questions also do not involve conspiracies unless the respondent has inferred a cover-up.  Other questions try to measure beliefs in existence and events (facts).  Still other questions try to measure beliefs in the interpretation of events (opinions).  Lumping these topics together may make it difficult for respondents to provide their honest opinions on each item.

BE SERIOUS.

If you ask your respondents about “shape-shifting reptilian people,” you are presenting them with a light-hearted survey and should not expect the same level of careful thought and concern for the research objectives.

RESPECT YOUR RESPONDENTS.

You have taken up someone’s time, typically without compensation.  You have asked them to share their views with you.  Do not thereafter refer to any of those views as “the wackier ideas out there.”

THERE’S MORE THAN JUST CROSSTABS.

Here is some additional information I would like to know.  On average, how many conspiracy theories does an American voter believe in?  Which beliefs correlate positively or negatively with one another?  Can ‘conspiracy theorist’ voters be segmented in a few cohorts based on similar beliefs?

SUMMARY

This survey, its questionnaire, its results, and the conclusions have made for some good laughs.  Unfortunately, this may be all the exposure to social science some people will ever get.  And that could hurt the various social science professions and their ability to gather, analyze, and use survey data.

This entry was posted in Uncategorized. Bookmark the permalink.