Everyone agrees teams should evolve to become outcome-focused. But how do you know if the features you release are leading to real business outcomes? It certainly ain't easy. We use the Product Impact Framework to see how one leads to the other.
Brian Donohue:
Thanks for joining me in this talk. My name is Brian Donohue. I'm a director of product management at Intercom. I've been with Intercom going on six years now, started there when we were about 40 people or so, and stuck with it to the point where we're about 650 people now. So been quite a roller coaster ride and seeing a huge amount of changes in the way we work and operate. And today what I want to do is share some of our stories, some of what we've learned so far in moving from thinking about outputs, to outcomes, a common theme, a big theme a lot of us are trying to deal with today. And I'll share a context of where we've come from in doing this, a simple framework to think about it because everyone needs a framework, and then I'll talk about an outcome driven approach, a feature driven approach, and share some conclusions.
Brian Donohue:
So a little bit about Intercom quickly. Our mission is to make internet business personal, and we do this through modern business messaging at scale. And really this is about we want to do this through the whole customer life cycle. So prospective customers through lead generation, coming through customer engagement, getting them to actually see value, and then actually supporting them and turning them into loyal customers. So why is outcome such a hot topic? And I think John Cutler, who's written a lot about this, really captured it. It's like this convenient, no brainer, never lose tagline. I mean, who doesn't want outcomes? And we're certainly fit into that bucket. So I think Intercom is actually, in many ways, almost a poster child for a product first company, our founders lived and breathed product. We were really slow to build sales and marketing teams. And I actually gave a talk a couple years ago about all the traps that come from this product first mindset, like believing the better product always wins.
Brian Donohue:
But really I think this output focused approach was another one of the traps that really, at least for us, came alongside of this product first approach. So for us, context of where Intercom is coming from. Back in 2013, Dara, our head of engineering, wrote this blog post that shipping is your company's heartbeat. And he coined this mantra and it really stuck through and it still bleeds through to how we operate today. New engineers when they join Intercom, they shipped to production on the first day, they'll ship some small feature in their first week, and they're going to demo that to the whole company. So this really is in our company's DNA of how we operate. In that same year Paul, our head of product, soon followed was his second mantra, which turned into ship to learn.
Brian Donohue:
And when you're in this startup phase of a company, the speed of shipping and shipping uncomfortably early so you can learn what actually works is really essential for success. We certainly consider it a key ingredient to our early success. And of course, we're not alone in thinking this. So you have David Sacks who's ex Pay Pal, Yammer, Zenefits, and he talks about, "A team of grain engineers regularly shipping code is a form of compounding, the most powerful force in the universe. This is a major reason startups beat big companies." So we were really wary of letting go of this product first and this output driven approach, because it had served us really well.
Brian Donohue:
So fast forward a little bit to 2018 and Dez Trainer, one of our co-founders, and as we're really trying to mature our product and our company, he talked about a new mantra for us, which was market impact, which is really another way of talking about outcomes, it's talking about business outcomes, market impact. We wanted our product teams thinking much more commercially. So it's not enough to just think about efficiently solving customer problems, we now needed to be thinking about our business and he repeated this phrase numerous times, literally market impact, market impact, market impact, really trying to change the way we were thinking about building product. And it stuck. We were all fired up about this new way of thinking about it with this one little problem was we were like, "How exactly do we have market impact? Sure, we can work more closely with sales and the go to market teams. We're all able for that." And conceptually, we get it's about building product that's going to translate to commercial value, not just good product. But how do we know for a projects are really having impact?
Brian Donohue:
And this is really the core problem is that the gap from shipping and improvement to measuring its impact on your key business metrics is, for most of us, this huge wide open ocean that we don't know how to get across. So we were similarly al trying to get better at this. And actually going back to summer of last year, 2019, we had folks from our biz ops team actually come over to our teams in Dublin and London and SF and do trainings with us. And one of the first things was like, "Hey, first of all, you got to understand how our business works. Here's all the leavers that we need of how we make money, how we lose money, our current business health." And this seems like it should be obvious, but particularly for SAS businesses, the details of the business model are often more nuanced than you'd expect, so it's no longer just enough to do a hand-wavy explanation of, "Yeah, here's those terms we hear the business folks, the finance folks throw around."
Brian Donohue:
We actually wanted our product folks understanding these terms properly and being fluent in this business language. And I think this was one great thing is like, "Hey, everyone in R&D needs to understand how your business works. If you're going to be outcome focused, this is implicit in it. And they should be fluent in your business language." And that was a great outcome we had from those summer workshops with the biz ops teams. But it wasn't nearly enough. And what we were trying to do is to push us further towards moving towards this outcome focus, our biz ops team actually created this super-powered Excel spreadsheet and it was hooked... We actually had to download Microsoft Excel. It was crazy, right? And it was hooked into real business metrics and it could calculate our notional business value for individual R&D projects. And the goal was to help us prioritize the most impactful projects, make sure we're getting the outcomes we're looking for.
Brian Donohue:
And this is actually what this spreadsheet looked like here. And you'd go in and you'd plug in adoption rates, what percentage of folks actually will realize value, how much they're going to spend over time. You just had to plug in a couple of things here and you'd magically actually get an outcome. Hey, here's your 12 month ARR increase, notionally increase, that you're going to see here. And it was like, "Wow, that's kind of amazing." But of course the little problem here was this felt like voodoo disguised in Excel, no one had any faith in the numbers that they were putting into this spreadsheet, which was pulling real customer numbers to pump stuff back out. And so the result was this thing didn't really gain any traction because we didn't have faith in our little part that we had to put into it. And so we were kind of left with saying, "I'm not sure if we're really actually able to size the value of these projects and really help us get to outcomes."
Brian Donohue:
So it turns out this shifting to outcomes thing, it's not just a switch you can flick and say, "Hey, now we're going to be focused on outcomes. Now we're going to be able to do it right." So a simple little framework to think about how to do this. And this is basically simple things of you ship product features which solve customer problems and therefore drive customer behavior and ultimately that change in behavior is what impacts your business results. Seems pretty simple, right? So here, a simple example for our team inbox, and this is where people who use Intercom go and write into reply to other customers writing into them. And so we have this team inbox app. So how does this fit in? So the product feature is this customizable sidebar that lets you integrate with other tools like Stripe or Salesforce. What customer problems is it solving? The inefficiency and friction in switching between tools and the customer behavior it's trying to drive is a reduction in average reply time. And the result is improved retention for our support customers. Makes sense, it seems pretty simple.
Brian Donohue:
So one key point to understand that we were slow to recognize is there's really two different approaches to building product, but they both work along that same simple framework. So let's make this more specific. So an outcome driven project is really a bottoms up approach, and so it makes sense to start from the very bottom. If you're going to move to an outcome driven approach, start from the bottom, impact business results. And we actually tried to do this. We had a project that we gave to the team. We ended up trying to give it to a couple different teams, which affectively was please move our business metrics, but actually what it was, was we asked product team to improve inbox seat expansion, and no team had ever focused on this before. It seemed like a really big opportunity, a big lever that we could pull. But in reality, it turns out it's only a half step removed from saying, "Hey, expand our revenue." This is one way we expand our revenue is just slightly dialed in there.
Brian Donohue:
And we didn't realize it at the time, but when we gave this to the product teams, they were like, "How do we even start with this? How do we go about doing this?" And this project actually became a big failure, it never really got off the ground. And the insight of why was this so hard for teams to tackle? There was an interview I did with Josh Seiden who wrote a book last year about this. And the big insight that came for it was you have to define these outcomes really precisely. And the outcome is that change in human behavior that drive business results. And that was the key insight we got from that there. And what we were trying to do here is we were trying to skip a step, "Hey, improve seat expansion," which is the business results, we were trying to skip a step of what's the customer behavior that actually mapped to inbox seat expansion.
Brian Donohue:
We didn't know, we hadn't done the work to figure that out, and were asking the team to skip a step. And there were kind of like, "Hey, we don't know how to do this. This isn't how we work. We're used to working in the opposite direction." So Gibson Biddle is from Netflix and has been doing a lot of talks around this and written a lot of blog posts. And he talks about those things that you need are these proxy metrics. We hadn't figured out the proxy metrics for inbox seat expansion. And he talked to describe these as basically being, "They're the standing for your North star product metrics, they're easier and faster to move. And ideally moving them will move that high level metric." So you need these proxy level metrics here in figuring this out.
Brian Donohue:
So we actually did this recently a couple of months ago with our onboarding team where they actually had to figure out, "Okay, well we start at the bottom, improve 12 month revenue retention," and worked their way up to, "Okay, we can correlate that to three month company retention, which actually correlates up to one month active retention, and actually we can get that to two log-ins in the first week and we can correlate and map these down so we have a high degree of confidence. It actually maps all the way down to the bottom." And really, of course, what teams are looking for here is leading versus lagging indicators. Because lagging indicators take way too long to see if you've been able to change them and it's way harder to be confident that the work you did three months ago was the thing that had that impact.
Brian Donohue:
So the key point here is though think about it, if you start with a team of saying, "Hey, we want you guys to optimize for two log-ins in the first week," a lot of people are going to go, "How the hell would we know if that actually matters?" So you really need confidence that these proxy metrics actually map to business results. Oh great, now we've got this customer behavior we want to drive, which is two logins in the first week. Now we need, again, not to skip a step. The next step we need to do is figure out the customer problems. And this is actually a big temptation when you've got that metric you're going after, frequently what teams will do is just jump to hypothesize and jump to, "Hey, we can do this, we can do this and let's ship a bunch of stuff and experiment with it," but you can't skip tha.t resist that temptation.
Brian Donohue:
And so this is the example of what the team did here is of course you've just got to talk to customers and understand what is the friction, what are the problems, why aren't they coming back that second time in that first week? And so you've got to do your homework, you've got to do your research here. There's no avoiding that. And only now can you go into experimentation mode now that you understand the problem and you know what the outcome you're shooting for, and that's where the team is at today in that experimentation mode. But the dirty secret of moving to outcomes is finding the right proxy metric is really hard and time consuming. You can't just pluck it out of the air and take a couple of hours and brainstorm a few things and say, "Yeah, let's go for that." And this was from Gibson Biddle's stuff, he talks about, "Hey, it took six months to make sure we were at the right proxy metric. This is hard, hard work to do to actually get this. And unless you have this proxy metric, you can't work in this outcome based way."
Brian Donohue:
And even worse is when you have that one proxy metric. So in this case, two log-ins in the first week, we couldn't use that for our other product teams. So you have to repeat this process multiple times for multiple teams and groups to find their own proxy metrics that you're confident are worth going after. So the second point here, and this sounds really obvious, but it wasn't to us, was because we were thinking, "Hey, we need to move from this output focus to an outcome focus, and everyone should adopt the way we're working," but not all projects actually start from the same point. And they don't all need to.
Brian Donohue:
So an outcome driven project starts from the bottom up, but you also have feature driven projects starting from the top down. So we can whiz through this approach because everyone's far more familiar with this way of working. So this could come from your sales teams, meaning we need a feature build or comes from plugging a feature gap, or it comes from a new idea or customer feedback telling you about problems that they have. All the standard way that populates our product roadmaps. And like when we launched our messenger with apps on the home screen, this was this new feature. And this was a great example of like, hey, were we shooting for a particular outcome here? No, we had this idea and we thought, "Hey, this feels like the next step and where this is going. We think there's huge product opportunity here." We were really focused in at the top, not at all on the outcome we were trying to drive. That was the way we thought and operated.
Brian Donohue:
And of course, good product equals a product problem focus. This is multiple talks. We've written huge amounts about this. I won't go into much detail here because we've really obsessed about the importance of being problem focused when you're building product. So it goes without saying that you've got to dive into doing this, and this is one of our principles, "Hey, start with the problem. Start with the problem. Anchor to the problem." But I'll skip over that, because that's another talk, another set of things. And then you're moving to the challenge becomes not in focusing on that customer problem, that's PM 101. It's actually about trying to figure out what's the customer behavior you're going to drive and then moving even further down to the business results.
Brian Donohue:
So again, let's look at real examples that we've gone through. So this inbox apps, and our team inbox here in this project here. The feature was clear of course, you're always clear on the feature. The problem, we were pretty solid on what the problem was going for here, but the outcome and impact, those were much trickier. So we actually were able to get testimonials, we were able to see specific customers who were able to improve their first response time, which was great. But in reality, the actual team who built this, here was the success metrics that they looked at and defined that project and it mostly was about adoption and about were people using it enough, were they actually engaging with the feature here? And what this shows is that when you're in this feature driven approach and you have success metrics that you're measuring, this doesn't actually make you outcome driven.
Brian Donohue:
And I think this was something we struggled with because we always had success metrics for our projects, but that didn't make us outcome driven in what we were going for. And there's an acid test where are you really outcome driven, which is if the feature you shift doesn't move that customer behavior, are you going to can it and try something else instead? Because if not, you're probably not genuinely outcome focused there. And so really one way to think about this is there's this natural center of gravity and these two ways of working. When you're talking about features and problems, that center of gravity at the top there for feature driven ones versus customer behavior and business results down at the bottom, which is the real center of gravity for this other way of working. And it's really hard to change that center of gravity for a particular project.
Brian Donohue:
So you can try to pull it down and figure out, "Hey, let's articulate that customer behavior that we're trying to move," and then actually stretching that all the way down to the bottom of the business results usually is way more speculative because you started at the top. And we've done things in our process to try to help us do this. So this is intermissions, which is basically a project brief that we write and we went to versions that had quite a bit of depth to them and what we're doing, we really tried to simplify this to here's our newest version where it's like, what's the problem, why are we solving this now, and what's the outcome we're shooting for? And trying to help us move further down that stack to simplify our approach. But again, this is only helping you extend down. It's still fundamentally a feature driven, a problem focused project. It's really hard to change that. So you can't really shift a project center of gravity unless you have those proxy metrics, you're only going to be outcome informed, not truly outcome-driven.
Brian Donohue:
And this is the other insight for us. It's like, "Hey, that's actually okay, there are different types of projects there." So if you take inbox search, so this is another feature we shipped last year, simple to understand. That's why I'm using this as an example. Our search in our inbox just wasn't good enough. We knew there were lots of problems, lots of shortcomings. We had heaps of customer feedback. We understood this problem inside out. And when we shipped this, the actual outcome, the customer behavior we were looking to change, yeah, we had success metrics, looked how often it was going to be used, how often we could see successful searches, but could we map that down the stack? We couldn't. And we had to say, "Hey, this is okay." Sometimes you're so confident of the importance of a problem. You know you need to solve it and that's okay to not give it that burden of going all the way down the stack.
Brian Donohue:
So when we think about these two types of projects, really it ends up being a more growth oriented mindset on the right, which is a standard outcome driven projects to where core product teams traditionally tend to think. And if you look at it, what are the differences? There's some generalizations you can make that at least help us think of, hey, there's two different ways of working here. Are you confident in the importance of the problem or are you more confident in the value of the outcome you're shooting for? Are you looking about giving new value or trying to get customers to see value? And then just go the way through, are you experimental driven or is this really beta and feedback driven? Hopefully these resonate and you say, "Oh yeah, one of these is the way we're actually operating here." And you can go down through this list and see, there are two general ways of working.
Brian Donohue:
And so the takeaways here from what we've learned so far, and again, this is all a work in progress, in another six months or another year, I'm sure we'll have refined this a good deal more of how we're thinking about this. But the first thing is recognize there are two different types of approaches both operating on that same framework. And we need to try to stretch each approach to hit all the steps of that framework, but recognize where it gets harder for each one to do that, because you can't shift that center of gravity, you can't turn what is fundamentally a problem focused, a feature focused problem into an outcome driven approach. You have to have those proxy metrics to do that. And it's hard work to get those.
Brian Donohue:
And ultimately it's not moving from just one approach to another, it's having a healthy balance of both approaches. And that's what we're in the process of trying to figure out to do is how to have a good balance of both of those. So that's it, that's the scoop. Hope you found that useful of us sharing where we've been on this journey of trying to get to outcomes. If you're interested in learning more, we've got heaps of content on our blog there, and you can go and just dial into the product and design stuff or go broader on the tons of content we've got there. Thanks for listening.
