Understand your product engagement score and how successful companies have approached increasing this metric to drive value for customers and results for businesses Tangible actions and creative approaches you can take to increase your product engagement
Rob Brazer:
Welcome everyone. Thank you for joining me here today to talk about project engagement, and how you can use a metric called the product engagement score to create better experiences for your customers and build better products. So a little bit about myself. My name is Rob Brazer. I'm a Product Manager at Pendo. I've been at Pendo since October of last year, and I cover our analytics and guides products. Prior to Pendo, I was at Amazon, AWS, and Bloomberg. So let's just go ahead and jump right in. So, product engagement score. We'll talk about what it is. We'll talk about how to measure it, and then we'll talk about why it matters to you.
Rob Brazer:
So, what does product engagement score? It's basically a holistic view of how your product is performing as a whole. It's really an uplevel do that combine some simple metrics that are used throughout the industry currently, such as stickiness, feature adoption, and retention, and combines those in the way to give you a single metric that you can really strive and look towards with a North star metric. So how do you measure this? Well, there's different ways that you can measure it, but for the purposes of this conversation, we'll just keep it simple and use one example, where we evenly weigh and take an average of stickiness, feature adoption, and retention. So here's just an example here that we made up and calculated. It just shows if you have these three different data points for stickiness, feature adoption, and retention, what your product engagement score would look like. As each of the three components are bound between 0 and 100%, that also means that the product engagement score as a whole will be bound between zero and 100%.
Rob Brazer:
So, why does it matter? It's a good metric to have, just to have an up-leveled view of how everything is performing. It's something that you can report out on and get everyone aligned and just have a good understanding of, are the things and efforts that we're doing, whether that's from a product marketing sales enablement, anything that you're doing to help improve your product and your customer experience, how are those impacting kind of a bottom line or single metric that we can look at holistically? So now that we have an understanding of what it is, how can we actually use it to improve our products? Now, what I'd like to do first is actually give a little brief, stickiness, feature adoption, and retention one-on-one, just to make sure that we're all on the same page here.
Rob Brazer:
So stickiness, just like we did with product engagement score, talk about what it is, give an example, and then talk about why it matters. So stickiness, it's a measure of how many users are returning to your products on a regular basis. It's pretty simple, and we can get into the specifics of it as we look at this measurement here. There's two main ways that companies measure this, either the percent of daily active users to monthly active users or the weekly active users to monthly active users. And which one you use really depends on the type of business that you have. We'll go into a little bit of the nuances later, but again, this is just to get some regular familiarity with it. So to do that even further, real quick, let's take a look at how this could look, just visually to help conceptualize it.
Rob Brazer:
This is showing over the span of 180 days, 180 data points where a stickiness metric was calculated using daily active users divided by monthly active users. So the takeaway if you're looking at this chart route, what should I see when I look at this is over time, stickiness is improving for this product that you're measuring. So in essence, that is a good thing. Any directional movement upwards with stickiness is going to be a positive impact for your product. So why does it matter? I mean, I would just sum this up. I say, what excites people enough to keep them coming back and to stick around with your product? That's really what stickiness answers as far as, what is driving people to go back to your product? What gets them excited? It's pretty simple.
Rob Brazer:
So, second component, feature adoption. Again, same thing that we did before, what it is, how to measure it, how to measure it, example, and then why it matters. So what is feature adoption? Feature adoption is the way to understand how many people of your total active user base are actually using a specific feature or set of features in your product. So, it can be measured in two different ways. For the purpose of this conversation, we'll focus more on number one. What we're saying is the feature-based analysis, but also just mention the second one, because some people find value in that as well. So again, as I mentioned at the start, it's what percentage of through a single time period. In this example, we have a month use the specific feature divided by the total number of users in that same time period, again, a month here, or just active at all in your application.
Rob Brazer:
So, if you had a feature that said, "Buy," you would take a look at over the course of a month, how many people interacted or used the feature, buy over the total number of unique visitors that visited your application in that month? The second one is a more of a portfolio view. It's almost taking the Pereto analysis approach of the 80/20 rule. And that's more of a holistic view as to what percent of your features that you've built. If you've built 10 features, is it two of those features or 20% of them that make up 80% of the click volume? Is just a way to understand what's really important and what's driving a lot of the usage from your product.
Rob Brazer:
So again, for the purpose of this conversation, we're going to focus on the feature-based version. And just like we did the stickiness, here's a time series chart showing how this can be measured. And this is saying, we had for a navigation dashboard feature, as you can see on the bottom of this chart, the percentage of users who interacted with that over the course of a month divided by the total number of users for that month. So, that would mean if you had 80 people, click the navigation dashboard feature out of 100 total users for the month, you would have the feature adoption of 80%. And again, this is just something we want to touch on. It's not as useful as far as the product engagement score calculation, but this is just a look at, "Oh, what percentage of my features make up a majority of the click volume or interaction volume?" And [inaudible 00:00:06:16] to the value that my customers get out of the product.
Rob Brazer:
In this case, you can see on the chart here, this is saying that only 12% of the features that I've built make up 80% of the volume. So you have a very core set of features here that make up most of the interactions that your customers use with your product. So, why does it matter? I mean, feature adoption really, it's two things. It's aligning the expectations that you had when you went out and started to build the product. When you had your research and design sessions. Did those expectations for how many people were going to use it. What type of personas people were going to use it. Is that actually realized based off this metric? Or the percentages that you're expecting to hit, are they being hit? Are they being exceeded? Are they not being hit? And if so, you can do some things with that information to improve it.
Rob Brazer:
And it's also just a measure of just from the customers choose. It's, "Hey, am I getting that quick realization of value?" Those aha moments. And not only just am I getting it once, but am I continuing to get it over time? So, you'd expect that maybe when you announced something and you have a new educational or enablement program to get customers ramped up on it, that you'll see a big spike. Seeing how that progresses over time is another thing, because it could drop off or it could continue to go up.
Rob Brazer:
So last of our three components here, we have retention. And again, what it is, how to measure it, a quick example, and then why it matters. So retention, it measures how many customers a company holds onto over a given period of time. And it can be looked at from a product-based view, a feature-based view, or just entire application view of, "Hey, did this group of customers after they, let's say they first came in and joined my product, are they keep coming back month, over month, over month?" So, how to measure retention. You can virtually use any time period you want. For this example here, we're just going to use three months. But let's just dive right in, just because I think this is one that seems a little bit more nebulous. But once you get an example here, it's really easy to follow along.
Rob Brazer:
So in this chart here, what I'm actually going to do is just start really simple, start really small. And we'll jump straight to the third row of this chart here, where it says, "June, 2019 with 21 visitors." And we're just going to read this from left to right. And so, what this is saying is that in June, 2019, we had 21 visitors that were new to our application. So these are new users who haven't been there before. And in month zero, you see for all four of the rows that 100% of the people are there, which makes sense, because we're just saying, "Hey, if you came in June, then yes, you used the product in June." At least for our example here with this specific row.
Rob Brazer:
As you move up to the right, we go from 100% to 81% for month one. So that's saying, of the 21 users that used the product in June, 81% of those 21 still used the product again in July, meaning you lost 19% of customers. Same thing as you continue on to the right to month two, when you now have August. And this is saying that 57% of your original 21 visitors are still using the product two months into it. Now, generally what you expect to see with retention is a steeper drop and then kind of a flattening, which is kind of the goal of retention. You want to keep that drop short as possible, and have that level of support where that line starts to flatten out as high as you can. So, retention is really most valuable when evaluated over time. It really ensures that the customers you've acquired are getting the value that they expected to get out of the product when they first signed up for it.
Rob Brazer:
A lot of the sales and pre-sales process, people will get excited about it, and then will be able to conceptually understand the value. But once they really start using it, it's a really good way to measure, are they continuing to get that value over time? So now that we understand the three components, did the quick one-on-ones and refreshers on what they are, let's just take a look at some quick recap again of the measurement, and then talk about some best practices.
Rob Brazer:
So again, this is the same slide that we saw earlier, but with a little bit more detail than before. And this is just getting a little bit into the weeds, more with the calculation. Again, same slide as we saw before, that same example. Just the straight line average is what we're using here for stickiness, feature adoption, and retention. So best practices. Just idea number one, keep it simple. Don't over-complicate it. It's a lot of information when you're looking at something entirely new. So just think about what are the core events, and key value drivers for your customers? And tailor these measurements around those activities. Next, because of what I just said, number one, I think it underlines even more focus on your own number, not necessarily what your competitors report, and track your number over time as a way to gauge if you're being successful or not.
Rob Brazer:
And the reason why I say this is because not everyone is going to measure stickiness the same way, nor should they. Not everyone's going to measure feature adoption the same way or retention, nor should they. And get a little more on the next slides about how you should tailor it to your specific needs. But the basic takeaway here is, measure yourself and see how you're performing against yourself all the time. Next, once you start to get, the keep it simple and you have the right measurements in place, you can make this even more powerful by applying it to certain populations of individuals. And you can say, "Hey, how are my promoters versus my detractors versus my passives doing as far as PES, if I only applied to this population?" Or how are my enterprise accounts comparing to my SMB or startup accounts? What does that look like as far as the differences in the number?" It's a really a good way to get insight into where you're doing well or could be doing better.
Rob Brazer:
Lastly, and again, this goes back into number two about tailoring it to your needs. Adjust the weights between the three of these to really fit your needs. Again, in the example, we have one third retention, one third stickiness, one third featured option. You might be at a stage where it doesn't make sense to have one third. You might be more focused on stickiness. You might be more focused on retention. You could be focused on feature adoption. And again, we'll get into that in these next couple of slides. All right. Stickiness best practices. So, one of the key things that I get asked a lot when people are talking about, "Hey, Rob, I want to start measuring stickiness. What should I measure? Should I use daily? Should I use weekly?" And the answer to that is, it really depends on the nuance of your business. But it's a really easy question to answer once you get it conceptually.
Rob Brazer:
As an example, if you are a company that relies on a bunch of users to come in every day as a reason that drives your top line revenue or the profit of your company, you're going to want to use daily. As an example of that, take a look at some of your bigger social media giants. Your Facebooks, your Instagrams, your Twitters. Those companies make revenue by selling ads. And those ads become more lucrative as the number of daily active users increase over time. So if you're an advertiser and you see these huge numbers that these platforms bring, you are therefore willing to pay more money to it. So if you were one of those companies, daily is probably the best metric for you to use over monthly active users.
Rob Brazer:
On the other hand, I'm going to go with an extreme example here. If you are a tax preparation software company, having daily active users is really not going to be that useful to you. It's not going to be helpful for you to know if someone's logging in August or September. That's really not going to matter as much. So, even going past the weekly, you could look at monthly or you could look at a specific month. It's really just important to understand that this is something you should tailor to your specific situation. Next, be critical on what active means. So again, stickiness is a measure of active users over another measure of active users. I would keep them consistent as far as the numerator and the denominator.
Rob Brazer:
However, I wouldn't just necessarily automatically default to logging in, or did they just touch my website or did they open my app? Because that's not necessarily going to mean that they're actually getting value of the product. For companies, the ones that I mentioned earlier that rely on this metric to help drive revenue, I would understand just keeping it as broad as possible to catch many people. But if you really want to get a little bit more tactical and think about, "Hey, it's useful now, they logged in, but did they run a report? Did they publish something? Did they go through a certain workflow?" That I think is really important to customers. You can define what active means is that versus just any activity. And I say all this to say, if you want to measure multiple dimensions of stickiness, you can, but just start with the most important one first, and then you can measure it in other ways.
Rob Brazer:
So features adoption best practices. You got to really contextualize the rates that you're targeting. And this is more, not as much about the metrics, but just think about this going to be really different feature adoption based on the type of products that you're pushing out. So, if it's something that you expect to be used by the entire company on every single customer that you have, you might expect higher numbers than if this is a specific workflow that only one person in the entire company might need to use in order for that whole company or customer to get value out of it, which brings me to number two. You can apply feature adoption to different populations. You don't necessarily have to just view it from a visitor situation. So as an example of that. Roles and permissions are something that is huge for enterprise software. If you have hundreds of people that are playing in a software space, you're going to want to have some permissions and roles associated with what the customers can do, because if not, you could end up with people making significant changes throughout the application.
Rob Brazer:
So if you're looking at feature adoption for something like a roles and permissions-based feature, you might only want to make sure that someone in the account is taking ownership and creating roles with new permissions or creating new permissions or assigning these out to new users. You don't necessarily want to... It's not going to be a concern if of the 500 people, a company who use your product, if only 10 people are properly assigning those, or it may be even one person. Point being, just contextualize the feature too, in the population that you're applying it to, to what the feature is actually intending to meet. What pain is it solving?
Rob Brazer:
Lastly, identify the right features to be included. You can do feature adoption from a single feature view. You can do it from a collection of feature views. But my recommendation is just really start small, as what is the most important action that a user can take? And start there. You can always go bigger, but people get into trouble when they're like, "I want all my customers to look... They must have completed this 20 different features for them to be part of what I think is true feature adoption of my product." Start small, keep it simple, and you can always expand out as needed. Next, retention. Align your metric date based on your needs. Earlier, we mentioned three month retention is a good recommendation, but that might not be appropriate for your business. You might want something that's a little bit more narrow. It's a little bit more aggressive and maybe bumps it up to a count of weeks or one month.
Rob Brazer:
On the other end of the spectrum, you could also have a longer time period. This might especially be useful for enterprise software that is based off an annual contract. And you might want to be wary of, from when they sign, how are they progressing? A certain number of months before their renewal. So if you have annual contracts, maybe you set something more on the six to nine months range. So as you're preparing to talk about a renewal conversation, you are really understanding and looking at how customers are doing overall. Now, another thing, just like with stickiness, you really got to choose your measure for activity here. Because you could just say, "Yes, we have great retention. We have 80% retention for over three months, because everyone is just logging into the app."
Rob Brazer:
But if you really don't know what those users are doing once they get in the app and you just know they're logging in, that might not really align with the value they're getting. That doesn't mean necessarily that they're getting the value that they think they are, or that you think they are out of the product. They could just be checking in saying, "Hey, it wasn't anything new launch. No, not really. Okay, I'm just going to go back to my normal day job." So it's really just important to align the measurement for active with what you think is important for users to do. So, now let's get into some tactical recommendations. So we've talked a lot about how to measure things, which I think is like, I spend a lot of time on that just because I think it's so important to really understand what you're measuring and measure the right things before you just start going wild with this new metric.
Rob Brazer:
So, let's actually take that next step though, and talk about some recommendations as to how you can use the score to then improve the experience that your customers have in your products in general. So, we have a list of seven items here. I'm just going to go ahead and dive right into the first one. So product-market fit. I think product-market fit is one thing that was honestly not talked about enough when we were looking at general metrics for feature adoption, stickiness, retention, or in general, a lot of the metrics that you'll hear talked about throughout the series. It's something that will really impact which of these three from the product engagement score are most useful to you at a certain time. When you're looking at your initial concept and first customers, and you're just starting out, it might be really important to assess feature adoption and stickiness. How many people that come to my app, how many of them are coming back continuously? And how many of them are using what I think is the main feature that they need to be using?
Rob Brazer:
As you move towards the predictable revenue and then the actual predictable revenue stages where what you thought was actually predictable revenue, you realize it's not. And then later you actually realize that retention is one that you might think becomes more important throughout time, because then customers are continuing to come back. They're continuing to renew contracts if that's the way your business work, or they're continuing to take actions with you throughout time. And the last one, demand outweighing capacity, just thanks to a lot of the infrastructure and platform. Infrastructure as a service providers we have out here. For a lot of the software-based companies, this is really not a situation that happens. But for more of your traditional manufactured products, this is something that does happen on the product-market fit line.
Rob Brazer:
Overall, I would just say the important thing is, understand where you are on the spectrum to help you better think about which ones that you should be driving. First-time user experience. This one is huge, if not the most important thing. As soon as you have customers that become aware of your product, you really only have one chance to really knock that out, that first impression. And so with that, you should be able to get them straight to value as soon as possible. Facilitate early wins, and do it in a way that kind of like highlights and hypes those up. Let them know that if by doing this action, you get this sort of value and play it up a little bit, make it fun, make it something that they realize that right away.
Rob Brazer:
And with that, I have planned ahead for idle time here. I think sometime if you install something, it needs some time to collect data, or if you just... There's a big sales suck, pre-sale cycle that they have to go through before being able to try out your product. Just think about ways to minimize that. Because a lot of the times, once you get customers using your product and understanding, it's so much easier to have those conversations anyway. So setting up as few barriers as possible to let them actually get their hands dirty with a product is a really, really great way to generate a good first time user experience. And then lastly, think about what I'm calling here champion checklist. So you have a bunch of customers now let's say, and there's some that you're like, "Yes, customer XYZ. They're the best. They talk about how much they love us. They will do referrals for us or talk at conferences for us."
Rob Brazer:
Think about the actions that they take inside your product, and think about... Let's actually break that down and compare that to customers who aren't getting as much value out of it. What can we use on this sort of checklist that these customers are doing to help other people realize the same value that your key champions are uncovering with your product? So really leverage that and think about that. It's another really useful way to think about what you're measuring when we talk about active for stickiness and retention specifically. And then featured option as well, because you're looking at, "Hey, what are the features that they're adopting and using?" So, engaging and re-engaging users. Now, have the usual channels and rewards. Reward people for coming back to your application. Give them a reason to do so. And the channels that you can do that with push notifications, emails, and whatnot.
Rob Brazer:
But I think it's really important to think about using that in a way that creates usage habits, and that offers precise and meaningful messaging. And also, bring in users to get their input specifically. And so, a couple of the recommendations here, just always think about if you are going to be trying to get your customer's attention, do so in a way that is focused on creating usage habits that align with value. Second, if you are going to ask for their time, make them feel special about it. And maybe show them some new things that are in the pipeline if you have mocks or user experience sessions. Or just in general, wanting their feedback on something new that's being developed. A lot of times you might get no for that, but it's really important to make customers feel loved and know that you're valuing their opinion. And by doing this, it's a really good way to do that.
Rob Brazer:
And lastly, offer precise and meaningful and contextual messaging. As you're using these channels, you don't want to get something that you read and immediately is like, "Wow, this just completely misses the mark. It doesn't apply to me." There are enough data and analytics platforms out there that you can really understand users and how they're using your product very, very precisely that then you can do the same with your messaging to make sure, yes, this is going to resonate with these users. And I have a 99% confidence level that this is going to just be something that they read, they're interested and they actually want to engage with versus everyone's had that moment where they got the email. And as soon as you read the subject and scroll through the first line, you're going down to the unsubscribe button because it's just misses the mark entirely for what you're looking for.
Rob Brazer:
Next. These next three slides are really going to focus on NPS and the general buckets of populations of users here with champions, passives, and detractors. And really, I think as we talked about earlier with the customer checklist for your champion checklist. What are the things that these users do when they find value? Really drive to understand that and talk with these people. These people already like your product. They're definitely going to be more willing to talk to you specifically about how and why and what they love about your product so much. And maybe what they like about your product over your competitors, or what else is out there. You can also think about these people and using them in different ways too. Creating a product advisory board. I think that's one thing that's awesome. You can really get in front of customers, show them strategic, maybe year plus out vision. What's coming up on the roadmap over the next couple of months. Not a simple product read out, because there'll be excited to see it, but they really can't give you their input on that.
Rob Brazer:
So, really think about it from that perspective and how you can get their input, and get opinions from them in a way that will help you ensure that this vision and future strategy you have from your product is going to meet the needs of your customers. Next, show some love and empathize with your detractors. I think if I'm looking at our own MPS submissions and the texts that's provided with it. It's never exciting to see a detractor, but it's also inspiring usually because the content they provide and when customers are willing to do it and say, "Here's what speaks well to me, and here's just where this misses the mark entirely." That's usually so valuable for you in many ways. It can help you understand how to improve your products. You can understand how to improve messaging, and understand how to improve enablement.
Rob Brazer:
There're so many things that you can do that will really help make your product better. Because some people I think shy away from this, because sometimes it's a harder ask. You're asking someone who might already be frustrated with your product for their time, or if you want to follow up with them further than what they provide. But I think sometimes you'll find those people, even though you'll get nos that really spend the time to really talk to you and say, "Hey, this is where it's missing the mark for me, and this is how it could be so much better." Lastly, passive. So passive, I think are even less talked about than detractors or our champions just because people think really in the extremes. They're like, "Hey, this is really good. I feel good with my champions and my promoters." And on the other end, these are the people that really don't like us.
Rob Brazer:
But the passives you see sometimes and you shrug and you're like, "Oh, okay." But I really think that's a missed opportunity. And why I say that is because these are the people that are closest to becoming promoters and champions for your company, but then you can use and continue to grow that army of people who are giving you good feedback or believing in your product or giving you referrals for customers. When someone says, "Hey, I don't want to speak to your company directly. I want to speak to someone who uses your product, so I can really dive in and understand from the unbiased opinion what the product is like, and what value I would get out of it as a customer."
Rob Brazer:
And lastly, understand your competition. This one's really a given, but I think some people just miss this a little bit too much. You get out there and get your hands dirty, use the products that your competitors offer. Really, just the best way to understand how they are going to do better than you for certain cases, or really even identifying your strengths, is just to go use it yourself. Next, ask your former prospects. Just like the detractors that we talked about earlier, this can be an uncomfortable situation. But me as a product manager, we're used to saying no to a lot of people just because we have to prioritize all the time. So being on the other end of that, I think it's something that we can especially empathize with.
Rob Brazer:
So, what I mean by this is if you have a customer that ended up going with one of your competitors, or just didn't go with your solution at all, and just decided to try to maybe bolt something in house, talk to them. Say, "Hey, we appreciate your time. Thank you so much. We would be interested in if we could talk a little bit more about the reasoning behind your decision." And worst case scenario, they say no. But with even just taking that action itself, you're letting them know you care. And you're really opening yourself up to a good opportunity to learn more about ways that you can improve your product, maybe your selling experience or anything about it that really got those customers looking at something else.
Rob Brazer:
Lastly, leverage your network. And what I mean by this is, I don't see this enough. I feel like on LinkedIn, where I love when I see these sort of messages that just say, "Hey, I'm evaluating product XYZ. Has anyone used these before?" And I think that's just a really good way to understand, because again, it's like understanding and using the competition yourself, or asking people what was something else. You can just put those messages out there. And usually, I've had some great conversations myself, just regardless of where I've been working of someone has spent 15 minutes, 20 minutes talking to me about something. If you're going to meet up with them, maybe buy them a drink or some lunch or whatnot. Just show some extra gratitude for their time.
Rob Brazer:
But I think it's really awesome and a great opportunity to just ask, put it out there. Again, the worst case that is going to happen in really any of these situations is just someone's going to say no, and that's okay. So, that concludes our talk on the product engagement score and product engagement in general, ways to improve it. Again, I'm Rob Brazer. I really want to thank you all for your time. Please feel free to reach out to me. I love to talk with people about these things, about products in general. It's something that I get really excited about. And it's really interesting for me as I've talked to people about setting this up for the first time, because it can be scary to learn more about the different businesses, and how people are using this, and how people are even thinking about it in the first place. So again, I just want to say thank you. And thank you to the people at ProductLed for organizing this.
