How to avoid bias in your survey

You want respondents to tell you the truth. 

Hearing the truth might hurt sometimes. It might take your marketing, product or sales in a completely new direction. But you’ll be thankful later when you have a product or service that is EXACTLY what consumers want.

It’s therefore super important that you remove any chance of biasing your respondents’ answers. 

Here are a few simple and effective ways to avoid bias in your surveys.

And here’s the TL;DR list to kick you off:

• Avoid yes/no questions
• Don’t ask leading questions
• Order questions and answers to avoid bias
• Be anonymous wherever possible
• Don’t feed answers to your respondents
• Clarify your language
• Look out for (and remove) assumptions

Avoid yes/no questions

Try to avoid questions with yes/no answers. This is because respondents have a tendency to want to say yes—it’s the positive, aspirational answer. This is called acquiescence bias. 

For example, instead of “do you find beautiful packaging essential when purchasing dark chocolate?” Ask, “which of the following do you find essential when purchasing dark chocolate?” 

Don’t ask leading questions

Look out for leading questions, which push respondents to respond in the way you want them to. This sometimes results in confirmation bias. You can do this by avoiding unnecessary adjectives. 

For example, don’t say “What do you think of [brand’s] delicious dark chocolate”. Instead, say “[brand’s] dark chocolate.” 

And don’t forget that scale question answers can also be leading. Make sure your answers for scale questions include positive and negative options, not just all positive options. 

Order questions and answers to avoid bias

The way you order your questions and answers has an effect on how people respond. This is called order bias. 

Order bias in questions

Order bias can happen when a question you’ve asked (or a media asset you’ve presented) has an impact on how your respondents think about the rest of the survey. 

For example, if you’re presenting a series of adverts, all subsequent adverts will likely be compared to the earlier ones in the respondent’s mind, even subconsciously. 

Here’s what you can do to avoid order bias in questions:

1. Don’t give away answers to later questions with earlier ones. Think about the order in which you ask questions to make sure you move from the most general at the start to the most specific towards the end of the survey.
2. Randomize questions wherever possible. Some platforms—like Attest—have this feature built in, allowing you to shuffle the order of your questions. 

Order bias in answers

The answers people give can also be influenced by the order in which you present them. People are more likely to select the answer(s) towards the top of the list. If this happens, you won’t have a true representation of your audience’s views.

Any good survey platform will allow you to randomize answer options. It makes sense to do this for all answers to all questions—except when you’re using scale answers (e.g. very likely, likely, neither likely or unlikely…).

Be anonymous wherever possible

Don’t give away who you are! If you do, this can cause sponsor bias—when respondents feel they need to give positive answers about the brand asking the questions. 

And it also stops people from letting their perceptions of your brand cloud their answers. Yes, you need to know what consumers think about your brand—that’s why we go on about brand tracking!—but there are times when you want to know what people think regardless of your brand.

For example, if you’re running new product development research, you want to know what people think about that exciting new product or service you’re considering. You don’t want their opinions about your brand—positive or negative—to influence the product insights.

Don’t feed answers to your respondents

Surveys are often designed to flow from the most general topics to the most specific. This is to make sure certain ideas or feelings aren’t brought to the front of mind because they were mentioned in previous questions. 

If possible, try to avoid emotionally-loaded language (words and phrases with positive or negative descriptors beyond their literal meaning) entirely. Be as neutral as possible.

Clarify your language

Avoid confusing respondents, or leaving room for interpretation in answers by objectifying your language. 

For example, instead of using ‘recently’, ‘often’ or ‘infrequently’, specify the time frame as ‘daily’, ‘weekly’, or ‘once per year’.

And don’t assume that respondents understand your business jargon, acronyms or slang. Make sure you know how your ideal customer speaks—this should help you use the right language.

Look out for (and remove) assumptions

Avoid making an assumption that forces respondents to answer untruthfully. You can correct this easily if you consider whether or not you should include the options ‘None’, ‘N/A’ and ‘Other’ in your list of answers. 

For example, asking ‘Which type of laptop do you own?’ assumes that the respondent owns a laptop, so you’ll want to make sure they can answer truthfully by adding options for ‘None’ or ‘N/A’.

And look out for places where you’ve assumed a positive or negative response, by wording your questions to allow for either option. 

Instead of ‘What did you enjoy about your last holiday?’, reword this as two distinct questions: ‘What did you think about your last holiday?’ ‘Very enjoyable’, ‘Quite enjoyable’ etc. And remember to add an option for the response ‘I haven’t taken an international holiday [in the last year].’

Then follow this up with a free text question like ‘Why do you feel this way?’