Bullshit metrics

Every day feels the same. A fledgling startup tries to appear like the up-and-coming market leader while the market incumbent aims to protect its dominance. It has become exhausting to keep up with how fast everyone seems to grow: 100,000 new users per week here, 20 billion monthly pageviews there, and let’s not pass up a watershed moment like 3 million members total. These are the industry’s most praised metrics.

Sadly, we haven’t moved forward over the past decade despite our whole industry becoming smarter about how it measures and analyzes data. Companies still pitch investors with a cumulative user sign up graph, sell advertisers on how many pageviews they get, and bamboozle reporters with the biggest numbers they can find regardless of whether they correlate to success. We can do better as an industry. We should do better because collectively we’re not benefiting–we’re all just fooling each other.

So, enough with the bullshit metrics.

The internet has evolved in the past decade. Websites are no longer static; they are dynamic. Businesses strive to build products instead of websites because they are more rich, more engaging, and more sophisticated. The web 2.0 era was characterized by its use of AJAX which made websites feel like applications instead of web pages. The concept of pageviews is dying. This is far more pervasive within applications we use on our mobile smart phones. Companies are still pressured to tout pageviews as a metric for success to entice advertisers to spend their dollars with them. And yet, users are not generating more pageviews while flipping through photos on Facebook. Extrapolating this idea further makes you consider what signups really mean. Facebook has done a great job with this. They undoubtedly passed their first billion signups long ago, but they’ve made a point to only talk about getting to their goal of 1 billion users who have been active in the past 30 days.

With any change in an industry, we need a mental model of mapping the old way of measurement to the new way.

Companies need to start using a new set of metrics that don’t simply make them feel good. They should use actionable metrics that provide insight, provide guidance, and help businesses make better decisions. I fundamentally believe that actionable metrics give companies a competitive edge.

Today, engagement is the better page view. Engagement is a specific action a user takes and it’s something specific to each business. Twitter can measure engagement by number of tweets and YouTube by the number of views a video receives. For Yelp, an actionable metric would be the number of reviews posted because more reviews likely lead to more users that find more value out of Yelp. More users means more reviews–a network effect. For Instagram, a mobile photo sharing application, an actionable metric would be the number of photos uploaded daily rather than the number of users that signed up and never did anything. Engagement helps us understand how actively people are using our products, whereas a pageview could simply be someone showing up to the product and leaving confused or unimpressed.

The most important metric today is retention. It tells you how often a customer comes back and uses your product again. Retention is an objective way to measure how valuable customers find your product so you can improve it if it’s not valuable enough. The best venture capital firms in Silicon Valley are beginning to expect this metric. Companies are starting to include this metric in their pitch decks to provide transparency into whether they can actually get a high percentage of their 100,000 newly signed up users per day to come back weeks later. The truth is, if less than 10% of people come back, then people don’t find the product valuable. To make matters worse, the percentage of people that come back as time goes on usually declines. Gaming best characterizes this decline because consumers eventually get tired as the replay value of a game diminishes. Now, only a small percentage of the original people who signed up are staying engaged. This particular situation is a reason the number of sign ups a company has attained is not an actionable metric.

But the most important thing is that companies find a specific, actionable metric they can focus vigilantly on.

My experience (I am the co-founder of Mixpanel, an analytics company that helps companies like Airbnb, Khan Academy, Dictionary.com, Match.com, and Path) has shown that companies should start by tracking a single actionable metric that they can literally bet the company on. I call this their One Key Metric (OKM). Companies choosing their OKM realize they must pick an actionable metric because pageviews or sign ups aren’t harsh enough and don’t correlate highly enough with the success of their business.

The best part of OKM is that companies can measure other things related to it, to understand how to improve it. In the case of Instagram, their OKM is likely to be photos uploaded. They can measure the percentage of people that download the app and uploaded a photo, the percentage of people who come back and upload a photo a week later, the growth rate of photos uploaded month after month.

Understanding your OKM often leads to deeper, more valuable questions. The answers often surprise businesses and guide them better than before.

Despite the internet’s evolution, bullshit metrics perpetuate a constant cycle of poor understanding. Let’s strive to understand how our businesses are doing and to pick better metrics–the harsher, the better. Let’s stop fooling ourselves with numbers that don’t represent reality. And let’s push the industry forward as a whole because collectively we’ll all benefit.

Thanks to Laura Savidge, Tim Trefren, Andrew Wei, and Nicole Leverich for your valuable feedback.


Now read this

The rise of engagement

The Internet is ever evolving. It is no longer just a series of pages that are linked together. It is no longer a place where we “surf the web.” It’s rich, dynamic, and engaging. Today, we’re using applications. The same applications... Continue →