Contact Us

Survivor Bias is a Big CX Measurement Risk

Jim Tincher Jim Tincher 12/18/2018

AsSurvivorship Bias defined by RationalWiki, “survivorship bias is a cognitive bias that occurs when someone tries to make a decision based on past successes, while ignoring past failures. It is a specific type of selection bias.” Applied to CX, it’s when you focus only on existing customers and ignore those who have left.

What Can go Wrong?

Let’s look at an example. There’s a national sports bar which saw its CX scores continually improving. Unfortunately, at the same time, revenue was declining. While CX was celebrating, the rest of the organization was panicked. 

One reason: The restaurant focused on the guests who gave the highest scores, which were the hard-core sports fans. To please them, the noise got louder and the food got worse. That sports fan loved it even more and continued to give high scores, but the family visitors were annoyed. As more families stopped coming (removing the guests who gave the restaurant lower scores), CX scores kept increasing, leading the restaurant to its death spiral.

Focusing on the “best” customers is a pretty common approach, but following this advice too far is a disservice to your company. You need to understand the needs of all your customers – and then make deliberate decisions on the trade-offs. It may be that focusing on your best customers is the right answer – but it can also lead you to trouble. By doubling down on the top customers, you may chase the rest away. 

What can you do to avoid this problem? Four ideas: 

  1. Follow customers over time. If your scores increase, are the same people giving you better scores or are the low scorers leaving? If it’s the former, pat yourself on the back – you’re making a difference! But if they’re leaving you, that’s a big red flag. 
  2. Rigorously segment your audience and look for the causes for the different scores. Evaluate: 
  3. Demographics. In the 2000s, Best Buy found that female shoppers wanted a shopping experience different from their core male shopper. Recognizing and acting on this difference led to a significantly better shopping experience for everybody. 
  4. Firmographics. This is the B2B alternative to demographics. I met with a digital leader last week who noticed that smaller customers provided a disproportionate level of profitability and were also cheaper to serve. He is focusing on their different needs, which are disguised in their existing survey process. 
  5. Product selection. We work with a manufacturer with three product lines and discovered that customers of one product line had needs that were divergent from the other two. 
  6. Usage. In the restaurant example, time of day was one way to suss out the different types of customers with different needs.  
  7. Attitudinal. When we worked with a financial services firm, we found their independent sales channel had two very different attitudes: a sales focus, and a consultative focus. Each had very different needs in their customer experience. 
  8. Map the defectors. Find out what the defectors want and how their journey varies from those who give you higher scores. Of course, you don’t have to do a journey map, but it’s an effective way to see how the audiences vary. 
  9. Customize your offerings. Can you provide segmented offerings? Perhaps a different menu for families, or a separate part of the restaurant with enough sports to fit your brand without overwhelming casual fans. For B2B companies, perhaps a different sales or service model with similar costs, but more-tailored outcomes. 

Doing this requires a very different approach than you may be used to, getting away from your PowerPoints, and instead meeting with your product and service teams to create a differentiated offering. It’s also the best way to make sure you don’t meet the same fate as the now-unemployed CX team at that restaurant.

Keep reading.

Stay updated with our insights.

Stay updated with our insights.