Contact Us

Behavioral Economics Explains Why Your Surveys are Flawed 

Jim Tincher Jim Tincher 10/10/2017

 

Daniel Kahneman isn’t known as a customer experience (CX) guru. A Nobel Prize winner, sure. Brilliant psychologist and leader in behavioral economics, yes. Author of a fascinating (but really dense) book? You bet. But he’s not really known for his CX chops. 

Yet, one of his findings shows why many surveys – as well as quite a bit of other CX research – is flawed from the beginning. 

Diving Deeper

In this book, Thinking, Fast and Slow, Kahneman introduces how we have two separate minds – the experiencing mind and the remembering mind. And that we live under the “tyranny of the remembering mind.” 

As its name suggests, the experiencing mind is what you feel as you go through the experience – whether purchasing a product, using it, or going through a customer service call. The experiencing mind feels the impact of the highs and lows – discovering a new product, bringing it home, and setting it up. It has the nuances of the positives and negatives and all their details. 

But once the experience is over, the remembering mind takes over and discards most of the data accumulated by the experiencing mind. What remains is heavily influenced by two points: The peak points of pain or pleasure, and how the experience ends. 

In this TED talk, Kahneman tells a story about how a visitor to one of his talks recounted listening to 20 minutes of music, which ended with a screeching sound. The visitor reported that the entire experience was ruined by this screeching sound. 

But the experience wasn’t ruined. Just the memory was. But that’s all that really mattered. 

In his book, he uses a different example, of colonoscopy patients (this was from 10-15 years ago, before the more effective pain killers they use today). He graphed the experience of two patients, reporting their level of pain as the colonoscopy progressed. He then asks the question, which patient felt more pain? 

 

When I ask this question in keynotes and workshops, most participants choose B. Patient B obviously did experience more pain, since this patient’s first few minutes were like Patient A’s, and then experienced more pain as the procedure continued. Yet, Patient B reported feeling less pain when the procedure was over. In fact, when physicians deliberately left the equipment in patients after the procedure was complete, not only did those patients report less pain – but they were significantly more likely to get a follow-up colonoscopy! 

He goes on to say that you can predict the reported level of pain by taking the average of the peak point of pain and the ending. So, patient A reported a 7.5 (a high of 8, an ending of 7), while patient B reported a 4.5 (8 and 1). How much pain that literally occurs during the experience is irrelevant to what patients reported afterward.  

Connecting to CX

It’s the same with your experience. If your survey goes out a few days after a support call or event, you’re relying on the remembering mind, which has filtered out most of the information about what happened. Yes, you can get a good idea of how they felt about it, which is useful for analyzing ongoing loyalty. But it’s a terrible diagnostic tool, because all the detail has been lost. 

This is what we regularly see in journey mapping work. If your methodology relies on after-the-fact recall – whether focus groups, interviews, or surveys – you’re not getting the real scoop. Yes, people will give you answers. And they may seem enlightening. But those answers are biased based on how the experience ended. 

If you ask a B2B customer who’s been using your product to discuss the sales process, their memories of the ups and downs will be tainted by the more recent usage experience. Disney discovered that all the joy of their experience was removed when visitors forgot where they parked. The rides were just as good, the experience was just as magical – but the memory of the experience was far worse. 

The takeaway? Look at your methodology. If it takes a few days to get your transactional survey out to your customers, rethink it, because you’re getting junk. When you develop a relationship survey, be careful about the questions you ask. Brand and relationship questions are fine. But specific questions about call center, website, etc. will be biased, because your customers aren’t using those touch points at the time of the survey. 

And if you’re trying to use focus groups to understand your customer experience? Then just give up. Because there’s no way you’re going to get accurate data. 

Keep reading.

Stay updated with our insights.

Stay updated with our insights.