Why metrics like Facebook’s “7 friend in 10 days” can be tricky

Say, you’ve discovered that your customers, on average, spend on the website 2 minutes longer than noncustomers. What’s your first thought? “Let’s make everybody else spend 2 minutes more and we’ll increase the number of customers”, right? Well, not necessarily.

Correlation vs causation

Here is another example from Intercom’s handbook (which I highly recommend):

“Let’s imagine a ride sharing app, where all customers who complete two trips in the first 30 days after signup are 40% more likely to retain in month two, compared with the customers who take just one or no trips. [...] In this case you’d get laser focused on encouraging customers to take 2 rides in their first 30 days.”
The Growth Handbook, Intercom, page 24

Or a Facebook’s early growth metric:

“After all the testing, all the iterating, you know what the single biggest thing we realized? Get any individual to 7 friends in 10 days. That was it.”
Chamath Palihapitiya, Facebook’s first VP of Growth

But here is a question: why is that 2 extra minutes on the websites caused users to buy, and not the opposite — users, that were already interested in purchase just spent 2 extra minutes shopping?

Or that 2 completed rides caused users to become active — and not that active users completed 2 rides just because they like the app?

It’s common to confuse correlation with causation — think that A causes B whereas A and B just happen simultaneously, regardless of each other. Or maybe because some C causes both A and B, who knows.

There is even a website showcasing funny correlation, like divorce rate in Maine vs per capita consumption of margarine:

Back to our example: generally speaking, we don’t know what caused what, so we can’t say that making users spent 2 more minutes on the website will turn them into customers.

Recall vs precision

Following up on the ride sharing app example:

“...all customers who complete two trips in the first 30 days after signup are 40% more likely to retain in month two, compared with the customers who take just one or no trips...”

Let’s say I’m trying to find out what differentiate customers from noncustomers during week one on my website.

I assumed that two metrics could be the differentiators: number of 5-minute-long sessions (x-axis) and number of product page views (y-axis). Here is how customers and noncustomers are distributed

There are clearly much more customers (blue dots) than noncustomers with at least one 5-minute-long sessions during week one.

After doing the math, I concluded the following:

“Users that had at least one 5-minute-long session during week one are 2.5 times more likely to become customers, that those who didn’t”

Sounds like a useful insight except for the fact that 50% of the customers have had exactly zero 5-minute-long sessions — and yet they’ve eventually become customers.

How many users who haven’t completed 2 rides during the first 30 days returned next month? How many users who haven’t added 7 friends in 10 days eventually became active? Nobody ever heard about it.

Statements like that only tell how good the metric identifies customers within users who’ve completed it (it’s called precision), but they never mention how many customers were cut off (it’s called recall).

It’s easy to make a metric sound better than it actually is if you don’t look at both precision and recall.

What metric to look at

The framework above is helpful to think about and compare different metrics. In the real world, however, it’s rarely possible to find a metric that perfectly causes the result while having high recall and precision. The good news is: the real world rarely needs it.

Think about Facebook’s “7 friends in 10 days”, for example. Having an individual added 7 friends is a proxy for experiencing the core value of Facebook — the network of people.

Or Pinterest’s “Save” button. Casey Winters — ex. Lead of growth team at Pinterest — explains why they’ve chosen it as a key metric:

“People derive value from Pinterest in different ways, from browsing lots of images to saving images to clicking through to the source of content. Eventually, we settled on saving (pinning an image to your board), because, while people can get value from browsing or clicking through on something, we weren’t sure if it was satisfying. You only save things if you like them.”
Casey Winters, Why Onboarding is the Most Crucial Part of Your Growth Strategy

The best metrics always incorporate fundamental value of the product.

 920   2019