Justin Bauer:
Hello everyone. I hope you're all doing well. I'm very excited to be here today to talk about two of my favorite topics, analytics, and product strategy, and how you can use them together to help you build a better product. Now, before doing that, however, a quick background on myself, like many of you have had a winding path to product. My career started in data science and analytics, where I was an econometrician before then applying those skillsets to the world of business. And that ultimately led me to product where I ran a gaming company, and now I've been the head of product at Amplitude for over four years. And my goal today is that after this conversation, all of you have a better understanding of how you can use data to help you set your product strategy.
Justin Bauer:
Now, you'll notice I said, help set your product strategy because we're not quite at the world where the data actually tells us exactly what to do, and that might work in the world of chess, but the world of product is actually much more complex, and we're about to see the games of product, way too much complexity, but instead it really is about the intersection of both art and science. And I found that many presentations and thought leadership is all around the art of product development, but there's a lot of science that is involved as well. And so I'd love to talk about today is how you combine both of them to build really compelling product experiences. But before we do that, a quick history lesson to demonstrate why it's so important that we think about both the art and science of product development.
Justin Bauer:
I'll start with this simple graph, which is a growth curve, any guesses as to what this graph represents? So this actually represents the growth of the internet and how long it took for it to get to a hundred million users. It's about eight years to get to a hundred million years. This was actually one of the fastest adoptions of a technology ever at the time, but we can actually compare that to this growth curve, any guesses as to what this is? This is actually how long it took Fortnite Battle Royale to get to a hundred million users, three months, on iOS alone. And in that period of time grossed almost a billion dollars, which is pretty amazing and speaks to how quickly amazing product experiences can actually transform industries. And it's not just in gaming. We're seeing this happen in B2B, in media, in e-commerce. All of these companies are really changing the games within their industries, and they're doing it by building transformative and innovative product experiences that their customers love.
Justin Bauer:
And that's really the question that we have in front of us. How are they doing that? And what can we do to bring back to our companies so that we can also build these types of experiences? Well, I can tell you what they're not doing. They're definitely not going through Michael Porter's five forces analysis. They're not getting a bunch of old, bald white guys into a room to do some annual strategy review, and they're not hiring McKinsey to tell them what to do. Instead, all of these companies are winning off the backs of their product. What we call being product-led, because they understand that their competitive advantage in the industry is the experience that they're building and delivering through their product. And all of them share three key things in common that we've seen with all great product-led companies.
Justin Bauer:
The first is that they set a vision and create clarity through a clear and measurable North Star. The second is that they set strategy, integrating behavioral science into their decision-making frameworks. And finally they're results oriented. They hold themselves accountable to outcomes and they measure whether the bets that they're taking are actually improving those outcomes and then doubling down on those, or if they're not calling those and figuring out the next thing. And what we're about to talk about is how companies are doing all three of these things. So let's get right into that. First, we'll start with vision. So as I mentioned, all great product-led companies set a vision to create clarity around what matters for that company. Now there's a lot of definitions of what a vision is, and the one that I like to use is the following. Vision is the measurable world that we want to create.
Justin Bauer:
Now I know that's quite a mouthful, so let's start on the second half, which is the world we want to create. So what does that look like? And to talk about the world we want to create, I love to bring up one of my old friends Fire Mario. Now you might be thinking to yourself what do some Nintendo's character from the eighties have to do with product and vision. Well, hold on, I'll explain that in a bit. This is actually one of my favorite analogies in the entire product world. It's actually from the user onboarding group. Many of you may have seen this image, but if you haven't before, let me walk through this because it's is a great articulation of what a vision is.
Justin Bauer:
So on the left, we have Mario and Mario represents our customer and who he is today. In the middle, we have the flower and that represents the product, or what we build and sell. And on the right, we have Fire Mario and how awesome Mario can become if he or she has access to our product. Now, when most people think about product vision, they actually think about the middle part, the flower, but actually product vision is about Mario, our customer and all the pain they're in today, and Fire Mario, how amazing they're going to be once we actually deliver this product to them. And it's really important as product leaders that we focus on both customer pain and the outcomes we want to drive. Mario and Fire Mario.
Justin Bauer:
Okay. But what does that look like in practice? Well, to do that, let's talk about a product that everyone I'm sure is familiar with, which is Spotify. Let me actually pull out Spotify's product vision, and see how they take that into action. And so here's Spotify's product vision, which is to give people access to all the music they want all the time, in a completely legal and accessible way. And if we break this down, we start to see a lot of Mario and Fire Mario in here, right? All of the music, not just the music that you have purchased. All the time, not just when the radio happens to have it on. In a completely legal way, not having to worry about illegal downloads or torrents and accessible in every single device.
Justin Bauer:
And you'll see here that none of this vision statement do they mention features at all, but it's all about pain and the ultimate outcomes that we want our customers to have, which is a great example of a product vision. Now, by this point in time you might be wondering, well, wait a second. I thought this is a talk about analytics. Why is this guy talking about Fire Mario, it's okay, don't worry. I run product for an analytics company. The data is coming, but it's not all a joke because the reality is that if we don't set a compelling vision, then strategy and everything else really doesn't matter because you need to know where you're going. But at this stage, we can come back to our original statement around vision and talk about the measurable piece. What does it mean to have a measurable vision? And that introduces the first concept, which is the concept of a North Star.
Justin Bauer:
Now I'm sure many of you are familiar with the North Star, but what I want to talk about is really how you put North Star into practice to help you set your strategy for your company. So what is the North Star? Well, the North Star is really what takes you from your vision to the impact you want to deliver. It really is that guiding light that helps you set a strategy to get there. And we have found, having worked with hundreds of different product-led companies, is that there are three things you need to do to really set a compelling North Star to drive your product strategy. The first is you want it to find the game that you're playing. Next, you then want to identify a North Star that drives the business outcomes that matter for you. And then finally choose the input metrics that are ultimately leading indicators of that North Star. So let's start with the game that we're playing.
Justin Bauer:
This is really important because if you think that you're playing checkers, but all of your competitors are playing chess, you're going to have a really tough time actually winning in your game. And what we've found through our research is that typically speaking, there are three different types of games that products operate within, and those are the following. The games of attention, the games of transaction, and the games of productivity. So let's dive into each of those. We'll start with the games of attention. So the game of attention is all around trying to deliver more value to your user through the amount of time they spend in your product. And so the types of North Star metrics that you're going to see here are things like total time spent, streaming hours, you might imagine some of the companies that are in the game of attention, right? Companies like Netflix and Facebook and others.
Justin Bauer:
The second game is the game of transaction. This is when the purpose of your product is to get them to engage in some sort of transaction within that experience. And so your typical North Star metrics here might be things like total number of purchases completed, or maybe number of seat upgrades purchased. And once again, some of the great examples of companies here might be companies like Walmart or United or Amazon. And finally, the third game is the game of productivity. And this is all around basically getting your users to some sort of task success. Typical product North Stars here might be things like total number of records created, or number of queries completed. And examples here will be the classic examples like B2B companies, Salesforce, Amplitude would also fall under the game of productivity. Cool.
Justin Bauer:
Now, once we've defined the games, it then is about how do we actually come up with an effective North Star for our specific company, because as you saw, even within the same game, not all North Stars are the same, and that's because you want to design it around your company and your goals within that game. The first thing you want to think about is how do you measure customer value? What we don't want to see here are things like opening the app, or ad clicked, because ultimately while those might be important for you as a company, those don't actually reflect value on behalf of your customer. Secondly, we want it to be aligned with your product vision, where are you trying to go, right? What is your Fire Mario, and make sure that this is a great measure of if you're actually being successful on that path.
Justin Bauer:
And then finally it should be a leading indicator of revenue. We don't want the North Star metric to actually be revenue because revenue ultimately is too much of a lagging indicator. But instead we want to see what is a product outcome that we know will lead towards revenue. And then you can get your company and product team organized around that. Now one thing that we definitely don't want your North Star metric to be is what we call a vanity metric. And I'm sure a lot of people here have heard of the concept of the vanity metric, and typically vanity metrics are top of the funnel metrics. It's not that you don't necessarily want to measure them, I think it's important to measure things like are people actually opening your application or how many active users do you have? But what we have found is that a really good North Star metric is much deeper into the funnel. It looks much more like a critical event that reflects the value that you're delivering to your customers.
Justin Bauer:
I think a great example of this is a customer of ours, Postmates, where their North Star metric is something they call happy deliveries. You'll notice it's not even a delivery, but actually it's a happy delivery, which means that the delivery is made on time and with no errors. And they measure that and have found in Amplitude that if they can deliver on that to their customers, then they're much more likely to come back, which drives retention and ultimately lifetime value for that company. Okay. So based on that, let's come back to that music streaming service, Spotify, and see if we can build up what their North Star metric might be.
Justin Bauer:
We'll start by asking ourselves a really simple question, which is what does the CEO of Spotify care about? And the answer to that is simple, revenue, because he's the CEO, he cares about driving revenue, but as we said, our North Star metric shouldn't actually be revenue, but instead should be a driver of revenue. So let's look at Spotify and what are its drivers of revenue. Okay. So first we going to ask ourselves, how does Spotify make money? And there's actually two ways that Spotify makes money. The first is that they have premium subscribers, and then they have ad supported users. Now I'd like each of you who are listening to think about which one do you think is larger for Spotify? Do they have more premium subscribers, do they drive more revenue from that, or from more free users? I'll give you a second to think about that.
Justin Bauer:
Okay. So if you're on the premium subscribers side, you're correct. 90% of their revenue actually comes from premium subscribers, which is pretty impressive. And actually, if we look at how many hours someone listens to music, once again, we see a way higher number of hours per month spent on the premium subscribers side. So given that information, if we were to articulate a North Star metric for Spotify, what might that look like? Well, likely would take into account premium subscribers because that's a key part of what they're trying to drive, and obviously people listening to music because they are in the game of attention. And that's actually what we find. So when we go to Spotify and we talk to them, we hear that their North Star metric is time spent by subscribers listening to music.
Justin Bauer:
Now, is this a good reflection for a North Star? We can bring back our checklist and ask us. So we'll start. Is this a measure of customer value? Yes. If people are listening to music that definitely reflects customer value. Cool. Is this aligned with their vision? Very much so, if you think back to their Fire Mario, their vision statement, this a hundred percent is aligned with that. And then finally, is it a leading indicator of revenue? Yes. If someone is subscribing, then that's a very strong indication that they will continue to pay month after month. So that's a great North Star for Spotify. Cool. So then we're just done, right? Spotify has defined its North Star. They can just go and tell their hundreds of different PMs to do whatever they want to figure out how to drive time spent by subscribers listening to music. Of course not because that would be chaos. Instead, what we want to do is actually set a strategy to make sure that we're all rowing in the same direction to drive that North Star.
Justin Bauer:
And so that's what the strategy is. It is the path to deliver on that product vision. And a key part of a strategy is understanding how you're going to get there. And what we have found is that with every North Star, you can actually break it down into actionable input metrics that you can create a strategy around. So let me explain what that looks like. So something that we have found, every single North Star can be broken down into four different metrics. First is breadth, which is how many users are actually doing that action. The second is depth, which is what is the depth of engagement with that action? The third is frequency. How often are these users engaging? And then finally efficiency, how fast they succeed.
Justin Bauer:
And what we have found found is every North Star is some combination of three or four of these different drivers. And so we can go back to our Spotify example and ask ourselves, what are the big drivers of time spent by subscribers listening to music? And what we find is that breadth, depth, and frequency are huge drivers of this metrics. So on the breath side we can look at listeners, number of trial users, or number of premium subscribers. On the depth side, we might look at something like number of hours per session. And then on frequency, we might look at number of sessions per week. And the great thing is that if we create a strategy around driving these input metrics, then ultimately we should start to see our North Star metric improve over time. Okay. But this was Spotify. Let's talk about another example.
Justin Bauer:
Maybe we talk about Netflix, but I actually don't want to talk about this version of Netflix. I want to talk about this version of Netflix, the Netflix of over a decade ago, the Netflix that was selling DVDs, the Netflix, that it wasn't clear if they were going to be successful, and talk about how they use this North Star framework to actually create such a compelling experience for their customers. So if we go back to Netflix, the first question might be, what game is Netflix playing? And so if you think about Netflix, definitely playing the game of attention. And then we might ask ourselves, well, what is Netflix's North Star? Luckily I've had an opportunity to talk to the former head of product at the time. And he told me that the North Star was number of subscribers.
Justin Bauer:
Okay. So that's their North Star, but they didn't just stop there. They thought about what are the biggest drivers of that North Star? And one of the biggest drivers there was monthly retention. And at the time they had a 92% monthly retention rate, which meant that 8% of their customers churned every single quarter. And they knew that if they're going to grow number of subscribers, they needed to improve the monthly retention rate, but they didn't just then go and tell PMs to go figure out how to drive that, they then broke that down again, and found that actually the number of DVDs in your queue had a huge impact on if you are likely to retain or churn at the end of the month. And in fact, they found that only 60% of all of their customers had three DVDs in the queue. And so they made their product strategy around how do we increase the number of customers that have at least three DVDs in the queue.
Justin Bauer:
And over an 18 month period, they were able to take that number from 60% to 90%. And by doing that, their monthly retention improved from 92 to 98%, which is a 4x reduction in overall churn. And according to their head of product, one of the main reasons why Netflix has been so successful to date is because of their ability to create a compelling experience that drives the growth of this metric. Okay. This is what we call behavioral science. It is about identifying what is the key outcome that you're trying to drive, but that outcome is a lagging indicator. It takes a long time to see significant improvement against something like a North Star metric. And so you want to identify what are the leading indicators of that? And then what are the input metrics that actually drive that leading indicator?
Justin Bauer:
And the great thing about doing this is then it helps you set a strategy as well as actually align your team around those key input metrics and have them drive those metrics through their strategy, and this is something that we do at Amplitude. So at Amplitude, our North Star is weekly learning users, which is the number of users who get to an insight and then share that with two or more people within their company. But then we break that down into what are its key drivers, and then we have pillars organized around those. So to get to weekly learning users, first, you have to actually get an organization to use Amplitude, and we need to activate that organization. We then need to make a really easy for people to get to those learnings and share them within the company. And then finally it needs to be very simple to consume that, to actually consume the learnings that other people are having. And then those pillars identify what are the key leading indicators and input metrics that they want to drive.
Justin Bauer:
So within that activated org pillar, they're trying to figure out how long does it take to get an account to create their first tax on me, get data into Amplitude, because we know if they don't have data in, then they're never going to activate, and if they're not activating, then we're never going to grow the number of users. And so they're focused on driving that metric and over time we will ultimately see WLUs grow with it. Cool. Now we are onto the final part of the conversation, we're in the home stretch, which is measuring results and being results driven. And this is really about making sure that we're on the right path and using an objective measure to do that.
Justin Bauer:
Now there are many ways to do this, but one of the most common things that I'm seeing product companies do within the last six to nine months is adoption of what are called OKRs, objectives and key results. Now I'm sure many people here are familiar with OKRs, and I'm not going to claim to be an industry expert here. In fact, if you want to learn more, I'd highly recommend reading John Doerr's book, Measure What Matters, but what I do want to talk about is how they're used in the product world, because I've worked with dozens of companies that have implemented OKRs. And there are some things that I'm observing that I don't believe are actually helping them build a better product. And so I want to talk about those next. And those are the following. The first is that, you got to remember they're called OKRs and not KROs for a reason, we'll talk through that. The second is that you should really design your OKRs around your learning cadence. And then finally, when you set your goals, it's really important to benchmark them for the game you're playing.
Justin Bauer:
So we'll start with the first one. OKRs and not KROs. Remember it is objectives and then key results, not key results and then objectives. There is a reason why we spent the first 20 minutes of this presentation talking about vision and strategy, and that's because those are the most important things to set first. Metrics are to help you measure if you're on the right path, but they're not going to set your vision for you. And so you should always start with the objectives first. A great example of this is a company in the gaming space, and this is actually their stock ticker in the 2012, 2013 periods. Some of you might've already guessed what company this is. This company is Zynga. Now I'm a huge fan of the impact that Zynga has had on the industry. In fact, you can argue that Zynga really invented product analytics, but unfortunately for them, I believe they took it too far and they actually lost sight on the vision. Ultimately, what is the pain that they're trying to solve for? What is their Fire Mario? And instead just optimized on the metrics.
Justin Bauer:
And when you do that, you might get short term outcomes, but in the long run, you're not delivering a compelling experience for your customers. And I think that is ultimately what happened with Zynga, unfortunately. Cool. So OKRs not KROs. Second is design around your learning cadence. Okay. What does that mean? Well, there's something I've observed with almost every single company that uses OKRs and is they measure them all on the same time period, the magical quarter. Now, I don't know what is so special about a quarter, but I guarantee you that it does not make sense that every company should measure their timeframe and how they evaluate their product bets at the same cadence. It literally just doesn't make sense.
Justin Bauer:
What is it about this quarter, right? 90 days. That seems pretty simple, but it's really not 90 days, because you got to take out weekends and holidays, so it's 60 days, but then if you think about it, it takes you about a couple weeks to figure out what your OKR should be, so now you're down to 45 days, and then of course, there's that project that probably spilled over by a couple of weeks, you're already at 35 days. And then you got to make sure you get something out before the end of the quarter, so you can actually have measured results, so now you're at 30, and basically now you've gone from what felt like a long period of time, but actually isn't that much. 30 days.
Justin Bauer:
And so what do you do when you only have 30 days? Well, you converge super fast on potential solutions because you want to get something out there. And then you wonder why at the end of the OKR period, you weren't able to hit your aggressive goals and you're super frustrated, and you just repeat the process over again. That really doesn't make sense to me. Instead, what I'd say is let's ditch the quarter, instead think about what's the long-term outcome that we want to drive. What's the thing that six, nine, 12 months or longer, that we actually want to create our vision, and that metric that we want to see improve, but I'm not talking about some 10, 20, 30% improvement, but how do we 2x that, or 5x that metric, right? Think about the North Star, and what would it mean to actually do massive improvements against that. And then give your team time to think big about what it takes to do that.
Justin Bauer:
And then think about what is your learning cadence in terms of how often you're going to ship to learn if you're on that path. And so if you're a startup that might be every single week, and so every week you're going to set what your input metric might be, and then actually see if you're actually hitting that. And if you're not, then go back to the drawing board and make improvements against that. For every one of our pods here at Amplitude, we encourage them to think and have a learning cadence of at least two weeks, but we actually let them choose what that learning cadence is. So some projects can move much faster and others, yeah, they might be big infrastructure investments, and so we might have a longer learning cadence. We really let them customize it to the type of work that they're doing. Cool.
Justin Bauer:
And finally, you should benchmark based on the game that you're playing. As you think about your goals, right? You really need to think about what is your game and what is the North Star metric that matters for you? A great example of this is with Facebook. So Facebook, when they were growing rapidly, talked a lot about a key metric that mattered to them, it was actually a frequency metric called DAU/MAU. I'm sure many people are familiar with it, it's basically what percentage of your monthly active users are using your product every day. And for Facebook, they hovered around 60%, which was pretty impressive. And basically after this, almost every single VC, that was all they talked about, was what was your DAU/MAU? What is your DAU/MAU? And if you couldn't compete with Facebook, then they wouldn't even fund you.
Justin Bauer:
But that doesn't make sense for every single business, for starters, actually DAU/MAU isn't the best reflection of frequency. There's a lot of other better ways to measure it. But secondly, it really only for something like an attention based product. If you think about something in the productivity space, take like an Amplitude. I actually don't want people using the product 30 days of the month because I want them to be able to be effective with their time so that they actually have more free time to think about other things. And so we might want to see them using it five days out of seven might be the most that we'd expect our users use it. And certainly within other games, it doesn't actually make a lot of sense. I'll give a great example of that.
Justin Bauer:
So there is a company that has a DAU/MAU ratio of 30%. And so on the back of just knowing that metric, you might say, well, that's not that great. Facebook had twice that, but if I told you that that company was actually Amazon, and on their mobile product 30% of people who go there monthly actually shop daily, you'd have a much different thought on that, because that's incredible for that game, for the game of conversion. And so once again, as you start setting your goals, think about what is the game that you're playing, and then benchmark your goals appropriate to that. Okay. Now, finally, we put it all together in what we call the product charter. This becomes the document, the single place that you can point people to who want to understand what is your vision? What is your strategy? And what are the results that we're seeing?
Justin Bauer:
It starts with a vision where you can articulate things like what is your vision statement? Who is your customer? Who is your Mario? What is their problem? And what is the North Star metric that you're trying to drive? You'll then go through your strategy, articulate how are you going to achieve that goal? What are the things you're not going to do? What is your timeframe, and most importantly, what is the learning cadence that you expect to see, to actually measure the strategy? And then finally the results, every single bet that you take, you articulate, what is the objective for that bet? What is the scope? And really importantly, how will you measure success? What are the input metrics that you're going to look at to see that you are effectively driving those, which are leading indicators of ultimately improving your North Star? Okay.
Justin Bauer:
So in conclusion, what we have found is that all product-led companies are great at doing the following things. First, they set a vision and create clarity through a clear and measurable North Star. They then use that to set their strategy, integrating behavioral science into their decision-making frameworks. And then finally they're results oriented. They understand the metrics they need to drive and they double down on the winners and call the losers to make sure that they're ultimately building a great experience for all of their customers. And with that, I will thank you all for this session. Hopefully you found it valuable. If you have questions or want to learn more about this or anything else, you can definitely reach out to me on Twitter at Justin J. Bauer, or just shoot me a note, Justin@amplitude.com, would love to continue on the dialogue. Thank you.
