9 Top Tips for Building a Survey with Optimum Data Quality

On-demand surveys are changing the face of research. They’re easy to build, you get results quickly, and, if you use the Attest platform, you can connect to an audience of 100M+ from all over the world. What’s not to love?

Sometimes diving into the world of market research surveys can feel a little overwhelming, but the ultimate goal is the same – you don’t just want data, you want quality data. So how do you make sure the data you’re collecting is accurate and isn’t just telling you what you want to hear? We’ve put together a list of top tips and things to avoid when you’re building a survey. 

Want to be an expert in surveys? Of course you do. Get your free copy of our Complete Guide to Survey Creation: 

What causes bad quality survey data?

There are lots of things that can cause survey data to lose quality. Here are a few biases and mistakes that can result in bad quality data: 

Selection Bias

If you send a survey out to your target audience, can you be sure that the responses you get back are a representative sample of that whole audience? Selection bias can happen when the group of respondents you get doesn’t reflect an accurate representation of the audience you’re trying to survey. 

It’s important to remember that though the size of your audience needs to be big enough to be representative, sample source is more important than sample size. Make sure all subgroups of a population are represented in your survey data.

Question Bias

It’s possible that the way you ask your questions will bias the results. This is called question bias, and it happens when questions are leading, or angled towards a particular outcome – usually without the researcher realising. Often an emotional connection to the subject matter of the survey can cause the person writing it to accidentally frame them in a way that biases the survey respondents towards the answer they expect or want them to choose. It’s important to make sure that whoever creates the survey has a limited emotional connection to the subject matter.

Question bias can also occur more with certain question types, like ranking questions or questions with lots of answer options. Often, if a respondent feels overwhelmed by the information, they’ll go for the middle, ‘neutral’ option in a ranking question, or the top or bottom answer in a question with lots of answer options. In these cases, it’s essential to randomise the answer order to negate this bias.

Irrelevant Questions

Some questions won’t be relevant to every respondent. Say if you ask how often someone drives, and they answer ‘never’, and the very next question is about how much they enjoy driving, that question is irrelevant to them and will produce an answer that is inaccurate.

Online survey platforms like Attest offer routing to avoid this. This way, you can send groups of respondents down different routes based on their answers. This stops them having to answer irrelevant questions – for example, if someone said they never drive, you can route them away from the question that asks how much they enjoy driving, and even ask something specific to why they don’t drive.

Duplicate Responses

If the survey you create offers incentives for respondents, there’s a risk that those respondents will attempt to take the survey more than once to double up on the rewards. There are simple ways to avoid this, such as vote protection. 

9 Top Tips for Building a Survey with Optimum Data Quality

Now we know some of the root causes of bad quality survey data, let’s look at some of the simple ways you can make sure your survey has the best possible data quality: 

1. Avoid questions that ask respondents to ‘select all that apply’

Information overload is real. When you present respondents with lots of information and ask them to choose all that apply to them, they’re likely to overclaim. Instead, try limiting these multiple-choice questions to ‘choose your top 3’ – that way, respondents will prioritise and you’ll get the most accurate answers. 

2. Don’t give too many answer options

Similarly to tip number one, this is all about reducing information overload and streamlining your survey. When building out your survey, you’ll need to prioritise the information you’d like to include – for example when asking which brands people have heard of in a certain category, you can’t include every brand that exists in that category. You’ll need to prioritise the ones that matter most to you. When building an Attest survey, you’ll get 10 answer options, so you’ll never need to worry about whether you’ve overloaded your respondents.

3. Randomise answer options

One of the best ways to avoid question bias is to randomise the order of your answer options. That way, the answer that was first on the list for one respondent won’t be first on the list for another, cancelling out that propensity for picking the answer that appears first. With Attest, randomising answer options is as simple as clicking a button. 

4. Make sure your questions are detailed

I know, I know, we just said not to overload your respondents with information – but there’s a difference between providing too many options and making sure your question is clear. If you make your question too vague, you’ll compromise your data quality. For example, asking ‘have you driven very far recently?’ will yield far poorer results than asking ‘in the last 3 months, have you driven a distance of over 100 miles in a single journey?’. Consider things like time frames when writing your questions, and make sure your questions can’t be interpreted in lots of different ways.

5. Include ‘none’ and ‘other’ as answer options

In case your respondent comes across a question that’s irrelevant to them, they need to have an option that specifies that – otherwise they’ll be forced to pick an answer that doesn’t accurately reflect what they really think, and your data quality will suffer.

6. Add red herrings

Throw in a red herring answer to one of your questions – something that is obviously incorrect or fake – that will alert you to any respondents who aren’t fully comprehending the meaning of the question. If anyone chooses the answer that is clearly false, you can screen them out of the survey. 

7. Never ask yes or no questions

Avoid asking questions that have only two answer options – ‘yes’ and ‘no’. These are not only too simplistic, but can cause a kind of question bias; respondents will often bias towards answering ‘yes’ because they feel it’s what you want to hear.

8. Use qualifying questions

Using a survey platform like Attest, you can target audiences based on demographics, such as age, profession, or even pet ownership. To get even more specific (and make sure you’re speaking to exactly the people you want to be), add a qualifying question at the beginning of your survey. You can select which answers will qualify respondents in or out – if they’re qualified in they can continue with the survey, and if they’re qualified out they’ll be removed.

9. Avoid leading questions

The best way to avoid accidentally leading your respondents in one direction or another is to make sure whoever is writing the survey has little to no emotional connection to the subject, or doesn’t have a preference for which answer is chosen. After all, we run surveys to get accurate answers to our questions – and those aren’t always the answers we necessarily want to hear. 

–

Ready to dive into survey creation, but unsure where to start? You don’t have to be a market research expert to start collecting quality market data. At Attest, we want to make it easy – you can start getting results in minutes with our intuitive survey platform. Book a demo with our team and we’ll be happy to show you around. 

Want all of the survey knowledge with none of the human interaction? We’ve got you covered. Grab your free copy of our Complete Guide to Survey Creation and start buffing up on online research: