Posts

Surveys – a Force for Good or Evil?

The Internet is a wonderful thing.  With little effort, we can connect to hundreds (or millions!) of people.  That access makes it really easy to conduct surveys.  So easy, in fact, that we no longer have to spend much time thinking about it.  And it’s obvious that many companies don’t.

While a proper survey can teach you about your customers, poor surveys lead you down the wrong path, sacrificing development dollars on delivering something your customers just don’t want.

Survey problems show up in three ways:

  • Asking for opinions instead of using readily available data
  • Outsourcing your thinking to your customers, asking them what you should develop
  • Piling on “just one more question”

Surveys vs. Data

It has now become easier to ask a survey than to do actual research.  Just because you can ask a survey, though, doesn’t mean that you should.

About 18 months ago the financial management website Mint conducted a survey that has been used by a host of speakers purporting to show the huge impact the economy has had on spending habits. One often-used slide:

You can see one presentation using this data at http://slideshare.net/MirrenBizDev/2010-new-business-conference-mintel. This slide is used to show how consumers are abandoning credit cards – 12% discontinued their use in 2009!

Not likely. Can you imagine the ripple effect if one out of every eight consumers completely discontinued the use of credit cards? The fallout would be massive!

The biggest problem is the question Mint asked.  Just because respondents said they discontinued credit cards does not mean they actually did it. Worse, the real data is only a short search away. What was the real change in credit card usage in 2009? According to the Fed, credit cards did decline – but by 0.2%! Yes, this is a dramatic change from the growth of previous years – but nothing like the impact that the Mint survey suggests.

Predicting the Future with Surveys

Survey data are frequently used as input to business decisions. Asking customers what we should develop feels right – but doesn’t work.  Consumers are notoriously bad at predicting what they want. Take this survey by the Consumer Electronics Association. While it’s dated, I saw the waste it generated at a consumer electronics retailer firsthand.

In this survey the CEA asked consumers what content they wanted to watch on their HDTVs. 47% said they wanted to watch home videos, while 44% wanted to view digital photos. This survey was cited in numerous business cases, and the retailer developed dozens of endcaps showing customers how they could do this through adding a computer to their home theater or connecting their Xbox 360 to the computers on their home network. We invested hundreds of thousands of dollars in these displays – likely millions when inventory is considered – yet sold very few.  What went wrong?

You can’t ask customers to predict the future – even their own behavior.  When asked whether they wanted to see their home videos on their computer, almost half the respondents clicked yes. Clicking a Yes box is a far cry from actually purchasing a thousand dollars of equipment and installing it into your home theater.  When it came to actually installing a computer to the home theater, very few were willing to take that step in order to watch their videos of photos.  Predicting the future is always risky business – this survey is just asking for trouble.

Yet, there is some truth to this data. Consumers clearly did want a better way of viewing their home photos. But when compared with the daunting task of getting computer content onto their TV, most took the sensible path of a digital photo frame – much easier, with almost the same result. Surveys are a great way to learn about your customer – but not a great way to learn what they will do.

Question 21.1.2.1

There is also the issue of the rapidly growing survey. Since it’s easy to ask 5 questions, why not 10? 20? Or, in my favorite “Bad Survey” example, why not 45?

This survey is by one of my favorite retailers. But it is a poster child for bad survey design, featuring a total of 45 questions, 40 of which are required. There’s even a question “21.1.2.1!”

When you’re writing a survey, it’s tempting to include everybody’s input. And that’s a good idea. But every question you add results in a few more customers dropping out.  Surveys require discipline:  prune the non-critical items to be sure customers will give you good data on what is left.

————

Does this mean that you don’t need surveys? Of course not – well-designed surveys provide critical input. But you need to spend the time to do surveys right. Some tips to success:

  1. Start with the end in mind. What is the #1 thing you need to learn? Is the rest critical? If you need to accomplish two very different things, consider a second survey.
  2. Decide whether a survey is really the right tool. If you want to understand behavior, observational data or behavioral analytics will typically give you much better results.  Surveys are best if you want to compare data over time, or compare results from two different groups.  Just keep in mind that the specific numbers (47% want to watch videos on their TV) are almost certainly wrong.  It’s not about predicting the future – it’s about understanding customer needs.
  3. If a survey is the right tool, determine how much patience your target market has. If it’s a free survey, keep it to no more than 5-7 minutes in length.  This is especially true for satisfaction or NPS surveys – keep these focused on this specific outcome, and use other surveys for market research.
  4. Consider using an expert to help you design the questions. Poorly-phrased questions will give you data – but sometimes customers answer different questions than you think you’re asking. If you cannot afford an expert, at least use an outsider to review what you develop.
  5. Test out the survey first. Have people outside the development team take the survey, and talk to them as they go through it – make sure their understanding of the question matches yours.

While the Internet makes it cheap and easy to do a survey, it also makes it cheap and easy to do crappy work. But if you take the time to do them right, surveys can be an excellent view into the Heart of Your Customer!

– Jim Tincher, Heart of the Customer

Review of The Ultimate Question 2.0

The Ultimate Question 2.0 (Revised and Expanded Edition): How Net Promoter Companies Thrive in a Customer-Driven WorldThe Ultimate Question 2.0 (Revised and Expanded Edition): How Net Promoter Companies Thrive in a Customer-Driven World by Fred Reichheld
My rating: 5 of 5 stars

NPS – opinions vary as to whether it’s the “best” way of measuring your customer engagement. The problem is that the industry is looking for a measurement that works for any industry or company. And such a tool does not exist.

Nevertheless, NPS is a good measurement, and Reichheld lays out how to be successful with the program.

The important thing that the author notes is that NPS does not stand for Net Promoter Score, but Net Promoter System. And it’s this System that is critical. In fact, if you replaced the measurement with Satisfaction, Engagement, Ownership, or your favorite home-brewed system, your business will still see significant growth if you apply the disciplines he outlines in his book.

A good book to get you started with NPS or any system. Highly recommended.

View all my reviews

Creating a great customer experience at Hawaiian Airlines

I ran across this great article on creating an outstanding customer experience at Hawaiian Airlines:  http://blogs.hbr.org/cs/2011/11/to_win_customers_get_out_of_th.html

I particularly like his three requirements to maintain an unbeatable customer experience:
1) Get very close to their customer;
2) Benchmark against itself on a consistent basis, and
3) Empower employees to address the unexpected.

#1 and #2 are quite common.  But it’s #3 that I particularly like, because it helps you avoid over-managing the experience.  When you hire the right people, you can train and empower them to delight the customer without having to over-manage the experience.

What are your three rules?

How Clear Are Your Instructions?

Have you ever eaten a frozen lunch by Michelina’s? These are inexpensive meals for a quick lunch. To heat it, you open the lid and put it in the microwave. After it has run for 2-3 minutes, stir it and put it back in. But for how long? The cooking directions are on the bottom of the box! If you didn’t memorize the timing before you started, you’re now in the position of either guessing the length of time, or holding it above your head so you can read the directions.

How does this happen? Do Michelina’s employees not eat the food? Do they spend so much time with the meals that they have the directions memorized? Or have they made a deliberate decision to sacrifice the functionality in favor of the branding on the top of the box?

This reminds me of the Tropicana rebranding failure from a few years ago.  You can learn more about it here and here.

To summarize, Pepsi (the owner of Tropicana) released new packaging for Tropicana. It looked okay on a carton-by-carton basis. But what about when you saw it in the store? First, it looked like a store brand. Second, all the versions looked nearly identical. Notice how the original clearly says “No Pulp Original.” Color variations differentiated the varieties, making it easy to shop. Now, look at the new version. Pulp free is there – but you have to look for it. It takes more work than the original. In addition, whereas the original had the iconic orange with a straw in it, the new one looks like a store brand! Imagine 7 varieties of the new carton all lined up together. It makes the shopper work harder.

Just to showcase the issue, I’d love to measure how may Tropicana shoppers actually look for the brand name when they pick up their juice.  I’m certain a sizeable minority just look for the orange. But don’t take my word for it – take the consumers’. Sales dropped by 19% before Pepsi reversed the decision!

This is an interesting product branding discussion.

But how can we learn from this in developing our own customer experience?

Let’s examine how the Pepsi decision-makers shop for orange juice. Not the consumers – the people involved with developing the product. Pepsi has a pretty sweet deal for employee purchases. Pepsi provides heavily-discounted beverages for bulk purchase at the headquarters. Why would you ever shop at a grocery store when you could pick it up at work and save yourself a bundle in the process? You wouldn’t.

At first this program seems like a good idea, as it will get employees to drink more of their own products, and they can reinvest that knowledge back into the product development process. From an R&D perspective, that makes a lot of sense.

But this leads to the employees missing the shopping experience. I have no particular insight into the team that decided on the branding. They probably spent a lot of time in stores, and they probably mocked up a store display at headquarters. But did they deliberately put themselves in the customers’ shoes? All signs point to a definite “No.”

Let’s contrast this with Best Buy, where I do have some experience. Best Buy has both an online and a physical store experience. They offer employees a discount to they shop in the stores – but not when they shop online. Why not? Store employees (particularly those near the corporate campus!) would like nothing better than to get those corporate employees out of their stores. So why not send them all online for their purchases?

The answer revolves around customer experience. Best Buy knows that if they offer the discount for online purchases, a substantial number of employees will never go to their stores. As a result, they will never gain that insight that comes from searching through three stores to find that special power cable for their phone. Only by physically walking through the customer’s steps can you gain the insight needed to create a great customer experience. Best Buy knows this. Pepsi didn’t.

What are you doing to make sure that you have a deep understanding of your customer’s experience?

Get out of here!

How often do you visit with customers?  Do you do quarterly visits?  Monthly trips?

Clearly, the answer will vary across different types of businesses.  A restauranteur can visit sites daily, and really should do it at least weekly.  A designer of nuclear plants probably can’t visit as often.  But the primary question is:  are you visiting often enough to stay fresh?

I typically find people on either extreme.  At Best Buy we had the Steve Jensens of the world (used without his permission!), a VP who visited stores for hours multiple times each week, talking to customers and associates to understand their thinking.  You could always count on Steve to know what was on the mind of our customers.  Unfortunately, we also had no shortage of people who had to be forcibly dragged to visit a store.  When a store is only a mile away from your desk, there’s no excuse for not visiting regularly.

Contrast that to a B2B company I once worked with.  After spending a few hours brainstorming on a new reporting package, I recommended we put together some mock-ups and run them by a customer.  The room went silent.  Finally, one person asked:  “Why would we do that?  It will take too long”  It didn’t take long before I realized that not a single participant in the meeting (besides myself) had ever met with a customer.  How does a product manager create great products from their desk?

When was the last time you visited a customer?  And, more importantly, when is the next time?