Surveys – a Force for Good or Evil?

The Internet is a wonderful thing.  With little effort, we can connect to hundreds (or millions!) of people.  That access makes it really easy to conduct surveys.  So easy, in fact, that we no longer have to spend much time thinking about it.  And it’s obvious that many companies don’t.

While a proper survey can teach you about your customers, poor surveys lead you down the wrong path, sacrificing development dollars on delivering something your customers just don’t want.

Survey problems show up in three ways:

  • Asking for opinions instead of using readily available data
  • Outsourcing your thinking to your customers, asking them what you should develop
  • Piling on “just one more question”

Surveys vs. Data

It has now become easier to ask a survey than to do actual research.  Just because you can ask a survey, though, doesn’t mean that you should.

About 18 months ago the financial management website Mint conducted a survey that has been used by a host of speakers purporting to show the huge impact the economy has had on spending habits. One often-used slide:

You can see one presentation using this data at http://slideshare.net/MirrenBizDev/2010-new-business-conference-mintel. This slide is used to show how consumers are abandoning credit cards – 12% discontinued their use in 2009!

Not likely. Can you imagine the ripple effect if one out of every eight consumers completely discontinued the use of credit cards? The fallout would be massive!

The biggest problem is the question Mint asked.  Just because respondents said they discontinued credit cards does not mean they actually did it. Worse, the real data is only a short search away. What was the real change in credit card usage in 2009? According to the Fed, credit cards did decline – but by 0.2%! Yes, this is a dramatic change from the growth of previous years – but nothing like the impact that the Mint survey suggests.

Predicting the Future with Surveys

Survey data are frequently used as input to business decisions. Asking customers what we should develop feels right – but doesn’t work.  Consumers are notoriously bad at predicting what they want. Take this survey by the Consumer Electronics Association. While it’s dated, I saw the waste it generated at a consumer electronics retailer firsthand.

In this survey the CEA asked consumers what content they wanted to watch on their HDTVs. 47% said they wanted to watch home videos, while 44% wanted to view digital photos. This survey was cited in numerous business cases, and the retailer developed dozens of endcaps showing customers how they could do this through adding a computer to their home theater or connecting their Xbox 360 to the computers on their home network. We invested hundreds of thousands of dollars in these displays – likely millions when inventory is considered – yet sold very few.  What went wrong?

You can’t ask customers to predict the future – even their own behavior.  When asked whether they wanted to see their home videos on their computer, almost half the respondents clicked yes. Clicking a Yes box is a far cry from actually purchasing a thousand dollars of equipment and installing it into your home theater.  When it came to actually installing a computer to the home theater, very few were willing to take that step in order to watch their videos of photos.  Predicting the future is always risky business – this survey is just asking for trouble.

Yet, there is some truth to this data. Consumers clearly did want a better way of viewing their home photos. But when compared with the daunting task of getting computer content onto their TV, most took the sensible path of a digital photo frame – much easier, with almost the same result. Surveys are a great way to learn about your customer – but not a great way to learn what they will do.

Question 21.1.2.1

There is also the issue of the rapidly growing survey. Since it’s easy to ask 5 questions, why not 10? 20? Or, in my favorite “Bad Survey” example, why not 45?

This survey is by one of my favorite retailers. But it is a poster child for bad survey design, featuring a total of 45 questions, 40 of which are required. There’s even a question “21.1.2.1!”

When you’re writing a survey, it’s tempting to include everybody’s input. And that’s a good idea. But every question you add results in a few more customers dropping out.  Surveys require discipline:  prune the non-critical items to be sure customers will give you good data on what is left.

————

Does this mean that you don’t need surveys? Of course not – well-designed surveys provide critical input. But you need to spend the time to do surveys right. Some tips to success:

  1. Start with the end in mind. What is the #1 thing you need to learn? Is the rest critical? If you need to accomplish two very different things, consider a second survey.
  2. Decide whether a survey is really the right tool. If you want to understand behavior, observational data or behavioral analytics will typically give you much better results.  Surveys are best if you want to compare data over time, or compare results from two different groups.  Just keep in mind that the specific numbers (47% want to watch videos on their TV) are almost certainly wrong.  It’s not about predicting the future – it’s about understanding customer needs.
  3. If a survey is the right tool, determine how much patience your target market has. If it’s a free survey, keep it to no more than 5-7 minutes in length.  This is especially true for satisfaction or NPS surveys – keep these focused on this specific outcome, and use other surveys for market research.
  4. Consider using an expert to help you design the questions. Poorly-phrased questions will give you data – but sometimes customers answer different questions than you think you’re asking. If you cannot afford an expert, at least use an outsider to review what you develop.
  5. Test out the survey first. Have people outside the development team take the survey, and talk to them as they go through it – make sure their understanding of the question matches yours.

While the Internet makes it cheap and easy to do a survey, it also makes it cheap and easy to do crappy work. But if you take the time to do them right, surveys can be an excellent view into the Heart of Your Customer!

– Jim Tincher, Heart of the Customer

2 replies
  1. Avatar
    Barry Kukulka says:

    Some excellent points about surveys. I think one of the common fumbles in the creation of CEM surveys is that the creator(s) are too broad in their scope. As you put it “start with the end mind….is the rest critical?” I’ve done some other studies in CEM related to customer verbatims on surveys and the number one complaint that resonates throughout is “you are wasting my time.” Generally related to flawed processes that confuse the customer requiring them to stop their work or personal life to call you, this also applies when you ask them for the courtesy of completing a survey. You are asking for a favor so make it as simple as you can, just like your transparent processes and procedures, right?

    I have also directly experienced misdirection from the interpretation of survey coding data. Use the data as a thought guide, but it is not all inclusive. It is very important to calibrate the data by listening to the calls that led to the surveys. My sample size was over 200 and I found that the survey data misdirected focus about 40% of the time.

    Thank you for pointing out how important it is to narrow your scope in the world of customer surveys. Having your surveys tested is another great piece of advice. I know I will follow it in the future.

    Reply

Trackbacks & Pingbacks

  1. […] blogged before about a major retailer’s satisfaction survey with its 45 questions, including number 21.1.2.1. […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *