Our digital products are fast becoming a hub of customer experience as sales activities, marketing messages, support, education and training begin to converge inside our applications. While these product-led motions are arguably a good trend—the ability to self-service help, education and purchasing empowers users—they also present new challenges for product teams, requiring an entirely new strategy for delivering product. In this session, learn how product teams are evolving from shipping code to driving adoption of products, factoring customer feedback at every stage of product development, and creating “product ops” to organize and rally entire companies around the product.
Todd Olson:
Greetings everyone. My name is Todd Olson, CEO and Co-Founder of Pendo. It's my pleasure to join you today to talk a little bit more about how we rethink product delivery in the age of being product-led, and it's a fun topic for me and I like to go back in time, of course, when I start this talk. When I started my career, I had the, I don't know, let's call it unpleasant experience of having to deal with physical medium. So yes, that means we shipped CDs. I even have fun recollection of designing boxes and inserts and documentation, essentially it says insert CD and hit installer. So not super sophisticated, but this is the way things were not incredibly long ago. So, certainly within my lifetime professionally, and what's interesting is a lot of our processes, a lot of the way we operated as business were by-products of this methodology.
Todd Olson:
How we built, how we created, how he communicated releases all revolved around this style and this methodology of delivery and the reality is the world's changed, and a lot of folks are now embracing more product-led techniques. Product-led techniques, well they've been now talked about for years but they're gaining more and more momentum. I'm highlighting here the product-led organization. This is a recent book that I published, which kind of is a good compendium so to speak of a lot of the ideas, concepts practices employed by product-led organizations and to try to document those here in a single place. Now people often ask me what is a product-led organization and a lot of people conflate, I think product-led growth with product-led organization and I would contend that growth is a piece of course, of an overall organization.
Todd Olson:
But for me, product-led organization is pretty simple. It is an organization which puts product at the center of how they operate and in a product-led organization each and every department can and should think about and explore ways in which the product can better help them do their jobs. An example I love to use is around recruiting, I was using a software product not very long ago.
Todd Olson:
I was curious how something was working inside this SAS product and I looked in the browser console which admittedly not your average user probably does that, it's more technical thing, but I was honestly just curious how something worked and within the comments, there in the code, there was this message, it was like, "Hey, if you're looking at this, you're most likely a good candidate for working here, please apply here." Which I thought was amazing. It's recruiting, asking the product to make a change to the product, which helps them with sourcing and if you think about sourcing, sourcing is the highest volume, lowest value activity within a recruiting organization. If they can get the product to offload some of it and get great candidates, that's a win. And that's an example of how product-led organizations think. Now on becoming product-led, putting practice center of your customer experience.
Todd Olson:
Well, it fundamentally requires you to rethink the way you deliver products. If you're delivering in the old way that is inspired and or influenced by CDs, you're not going to fully realize the benefits of being product-led. And if you think about it, an extreme example Etsy, this is a bit of a dated example because people talked about them years past, Betsy is of course a marketplace solution that, it was very popular for expressing and sharing that they ship many, many, many times per day, and that, what that allows them to do is move very, very, very quickly. So the more you release, and of course releasing that frequently requires a lot of infrastructure to deal with situations when things break and you need to roll back or, that act, the act itself like shipping frequently. Yes. Cool act.
Todd Olson:
The infrastructure in process, an organizational mindset you need to employ to actually do that, that's the stuff that matters. And those things are the things that actually help unlock your organizational ability to move really, really, really fast, and that's the part that I think is very exciting. So I want to talk a little bit about how to decompose these different practices. And one of the first practices I want to talk about is launch, how we launch things, how we shift things and of course I want to think beyond just the actual shipment of bits on a server or in customer hands, right. I already shared CDs, in the CD world, pretty challenging to have different release cycles. Sure, I could ship a beta CD to customers months before, but every single bug they find I need to drop ship a CD ROM to their house with all the updates.
Todd Olson:
Let's be honest folks that's not really a good way to transact, so of course we did not do that back in those days. We may have had a beta program, but very different than what we can do now. Now with the advent of cloud technologies and the fact that we can reach our customers and users with cloud technologies, we can do fun, interesting, different things, right. So you can have a process where you just ship code as fast as it's ready but you can have that code hidden behind, I don't know, a feature flag, which I think is a really core foundational technology idea which allows you to decouple how fast engineering moves with how fast customers adopt things. That's one of the, essentially, the impedance mismatch between engineering and customers. Your customers and what they can adopt probably is it a different speed than what your engineers can do.
Todd Olson:
And you actually want them to be separated, think about organizational change, can you keep up with things that are changing constantly when you're using them? No, you can't. Well, we want our engineers to move really fast. So feature flags enable us actually that they sit in the middle and allow this impedance mismatch to exist and how to manage it more effectively. So you can see here we have the opportunity to rethink the release, and this is just an example, folks, you can have any release process that of course that fits your organizational needs, honestly, whatever feature you're shipping may have different needs but I like to think about, these are potential steps, which an internal step would be all of our employees can see it, which is really useful.
Todd Olson:
It not only helps you catch things before customers see it, it also helps your internal organization start to learn. It's enablement, it's education. If they get a month or so of extra time of the feature, just think how much better they will be at supporting customers when it's out in the field. That is of course the notion of a limited beta, which is a hand selected, typically set of customers to try this out and look these can be your best customers, your nicest customers, your, honestly, your lowest risk customers if you break something or maybe it's a high-risk customer, maybe it's one where, look you're really worried about breaking them. You want to find out before you're in GA that you're going to break something, right? So maybe they're one that use your product in a unique way that you feel it's critical to help inform whether this thing's ready for the wild.
Todd Olson:
So I think limited beta is an opportunity to really learn. I focus on... In some features, do you need a lot of learnings? Maybe you want to skip this step. Maybe you're like, this is renaming something, do we need to really learn. You could... If you're naming the most core thing in your product, maybe you do want to test it out. But my point is that this is an optional step of potentially very important steps. I've even talked to some customers or in companies that they charge money to be in this program. Why? Well, there's value to the customer? The customers now have a chance to shape the future direction of the product. That's awesome. You will want to co-create. This is a great opportunity. The next open beta is your opportunity to deliver something but still declare it being not quite ready. So it's intentional. I think many of us recall that Gemia was beta for years.
Todd Olson:
And honestly I don't know exactly what they were looking for in terms of that graduation step but you certainly have a lot of flexibility here at GA. And look, the other thing I think is really important when you define these stages, define the criteria. What do you want to see before something graduates from limited release to open beta? What do you want to see when things graduate from open beta GA, declare it. This helps you not have things just running endlessly with no end date. Another great technique when thinking about release process is we think about beta, I described a limited beta, which we were asking people do they want to be in it. I also talked about an open beta where we just shipped something and say, "Hey, it's not quite ready yet." This is, I'm calling it an opt-out beta, which is your ability... so it's a little bit different in that you ship something, then we're giving this switch to legacy page view and what it's saying is, "Hey, this is new. We know it was instructed. If you want to go back to the old way, click here."
Todd Olson:
So what's nice about this technique is that a lot of people are afraid of change. And if you ask someone, "Hey, do you want something new?" Folks who've read the book, Who moved my cheese, if we asked you like, "Do you want a bunch of change in your life?" You are like, "Whoa, no, no. I was pretty comfortable the way things were." So you're not really getting good feedback in that particular instance, right? In this case, you're forcing change upon people, which at first may be uncomfortable to them, however, what if they just give it a quick whirl, which most people will. They are like, "Huh? This is actually pretty delightful."
Todd Olson:
It's actually a much better test in some instances because what you don't want do is you don't, we all know inertia exists in, like saying, "Oh, these people didn't opt in to change." You're not testing a lot there. You're just testing the percentage of your users who have interest in change, which isn't really a great feedback on the product itself. Now what's cool about this is that over time you can look at cohort analysis as fewer and fewer people switch back to the legacy view, then you know it's ready to get rid of the button and then get rid of the code for the legacy view, to clean it up. So great opportunity to really rapidly test something. This is a different way of thinking about betas. When you do ship something, focus on driving adoption, look at this UI element, this upper corner, I've got four or five icons in the American flag.
Todd Olson:
If I had that fifth and I didn't tell someone I had the fifth, would you have ever noticed it? Like, come on. It's a light blue icon in the upper right of the product. Your teams may have spent months in that little light blue icon, there might be something amazingly cool behind it but if you don't enable people on it and you don't teach them, do you honestly think they're going to just figure it out on their own, especially, this is one of the challenges with design, if something's really well-designed, it seems like it fits in some instances we have discoverability issues, right? So here's an example, just a very basic tool tip, which directs the user to this item like, "Hey, try it out. You can now add new things to the dashboard." And sure many people have been asking us forever but this is literally brand new.
Todd Olson:
You have to enable, you have to educate, you have to push people to change their behavior. Again, inertia is powerful. Find ways to overcome inertia. Another topic, which I think is very interesting is letting go, and specifically rewrites. Rewrites are the bane of most of product managers existences. We don't like rewrites. It's a situation where you're trying to build something new, but yet people have massive amount of expectations around the old and you're constantly chasing parody. Oh, you used to have this one thing I used once in your old product. I know you're completely redesigning the whole thing and it's world's better but with that one thing I just have a hard time letting go of it, right? So painful, because guess what, if you've got an eight year old product, you've eight year old little details of parody that you need to figure out what to do with, right?
Todd Olson:
So there's lots of different examples. And this slide is inspired by this fantastic blog post, Security, check it out. It goes into even more examples. There's a lot of ways to do this. The Basecamp folks, which are, there's just great, great examples of product in general because they love to employ such extremist views in so many different ways. They have strong opinions, of course, they don't actually get rid of old products. They build new products and people can just simply opt into them. They even fix bugs in those old products and their idea over time is eventually the replacement will out pace the legacy products. And maybe over time they finally will end of life a few but generally speaking, they're like, "Hey, people pay for these old products, great product. Why get rid of it?"
Todd Olson:
But I got a new product and if people want to opt into that, great as well. So they take very much of this purest philosophy. FreshBooks is an interesting one and kind of close to us because we were partnering with FreshBooks around the time they did this. They actually set up a whole new company, and that company was meant to compete with FreshBooks. It was called BillSpring. They launched BillSpring, different brand. You had no idea it was part of FreshBooks. They just rebuilt FreshBooks but in a completely new way and they knew they started winning when FreshBooks started to get losses to this new upstart called BillSpring, which of course they owned, but no one knew. Now over time they've let people on to, in situation eventually merged it back in, but that's a really cool way, very disciplined, experimental way of helping replace a legacy product.
Todd Olson:
And the last one I'll highlight is Gmail. I talked a little about Gmail before, but this is their inbox product, which for those who are familiar, they launched a thing called Inbox. It was also like super secretive, you had to get an invite to it, it was kind of special and so it made it super special and people got on their phones and it was dubbed as a Gmail replacement but didn't do everything Gmail did. And over time what we saw is, as they experimented with Inbox and you use it or you didn't use it, they just simply pulled the best features of Inbox back into Gmail and then killed Inbox off. So it was one of those things that they used as an experiment. They knew that if they just give all those new features to the Gmail crowd they probably wouldn't be able to test but they created this kind of early adopter crowd who ended up opted in to this cool new thing.
Todd Olson:
You wanted to revolutionize their inbox on a phone and all it did was informative features in Gmail, which is awesome, very awesome, great way to run it. These are all fantastic ways by the way to run good experiments but three fundamentally different philosophies and strategies. So that's the first area. Second area is around, of course, engaging companies create more engaged customers. You want your customer integration. Engagement is not, we shove a lot of content to our customers and they listen, engagement is two way communication and the first thing is the second you ship something, people are going to have opinions. They're going to have feature requests. They're going to find bugs. They're going to be talking to various people at your company in various mediums, email, phone calls, you name it, there's many feedback. What you want to do is you want to find a way to create a centralized repository for that.
Todd Olson:
You don't want them to be sprawled. You don't want a hundred spreadsheets. The other value in having it in a single repository is you have the opportunity now to create that two way conversation, right? Because if someone just says, "Hey, I want blah, blah, blah. I want the export button." Would it be nice to say why? Don't you want to go back and forth. So by having some really clear documentation, you can have this further clarification which will help inform better product development and the last piece is when you do build that thing, you do ship that thing, you want to tell them, you want to get points. You want to get the benefit of having listened to someone and then take an action. That's the best piece. So you can say, "Hey, thank you for this. We actually delivered it. Please give us feedback."
Todd Olson:
So there's a lot of power and a lot of goodness in having some strategy here outside of chaos. And when you do that, it's also important to kind of think about how are you going to prioritize this feedback? Does everyone just get a vote? And does each person get intimate voting? Everyone getting a vote would capture every single customer/user that was interested in that. Okay. Maybe that's what you really care about. You want to know who everyone is interested. Now, the budget voting is nice because look, if you ask the customer, if they want three things, they're going to probably say yes to all three. Now, if you say, you tell the customer, "Okay, you get three votes and you can put it on anything." They may not put one, they will put three votes on one item.
Todd Olson:
They may put two in one and one on the other, right. Asking someone when they want something and then creating a constraint, a budget where they can only vote for one thing, what would they vote for to guarantee they get it? Very different question, very different prioritization, very different data. The other thing is pairwise voting is pretty cool. This is some kind, it can be overwhelming if you ask the customers like, "Oh, here's a list of a hundred things, budget vote." They may give up after an hour of trying to figure out how to properly do their budgeting. Right? But what if you just presented users with a consecutive set of questions of what's more important, A versus B, B versus C, A versus C, et cetera, right?
Todd Olson:
You'll start to, through that process, getting more refined picture of what people prioritize and that's a great way you can also think about building a prioritized backlog. The other thing to think about is publishing your feedback strategy. Be transparent. How do you want feedback? Do you want people just to tell you what they want or do you want them to give you the why when they're fulfilling it? Customize the form where you're asking things.
Todd Olson:
Ask them the business impact of their feature request, ask them if they actually try to work around, these are all interesting data points that you'd want to know that helps inform what's going on. Then tell them what you're doing with the feedback. The last thing someone wants to do is give you a feedback and then they hear nothing, black hole. Tell them, "Hey, thank you for your feedback. This is what we do with it. This is what you expect to hear from us. So you can check in on status of it." That makes the user feel good. Like, "Oh, that's off my plate. That feels good." Right? So create this communication strategy to help customers know what's next.
Todd Olson:
And then of course, give them more feedback on how they continue to be engaged in the process. And one of the great things, examples I saw in my past was whenever you submitted a feature request to a last scene, they would tell you not just thank you, this is our process. They give you a little snapshot of what they are working on, their current roadmap. And sometimes they user be like, "Oh wow, what they're working on is clearly more important than what I just asked for." So it actually builds a little bit of trust. Like, "Oh, they're thinking about things that I wasn't even thinking about." And that feels good. Again, instills confidence. Good stuff. And then let customers help, once you get this prioritized list, do something with it, put a team on it. My past, I once had a development team that all they did was pull items off this list, called them customer delight team or something to this effect.
Todd Olson:
Look, you got to be of certain size where you can afford to fund it. And for us, we did define the product or some percentage of our capacity of our engineering resources was going to devoted to those customer love. And that was something we said as a business was important to us. And that's okay. And we communicated back to these customers like, "This is where these items are on our roadmap and this is when you're going to expect to get them, et cetera." And that's a great thing. Doesn't mean you have to dedicate a hundred percent of your capacity to it but just carving out some, built more trust and confidence with our customers. And then again, use roadmaps as a communication tool. I've heard, a lot of you are anti-roadmap, don't show it. People feel it can be an engineering commitment.
Todd Olson:
So there's probably a lot of best practice when it comes to roadmaps. But I'm a big believer in roadmaps is a two-way communication tool. All it is the declaration of intent. We think we're going to be building this going forward. No, by the way, the things that are near the roadmap. Yeah, we're really confident on those things, later much less confident. I will caveat this for product managers out there. Don't show things in your roadmap you've never talked to an engineer about, that's just a great way to piss off your engineering team. Again, first off, show it internally, get buy in. Have you asked questions? And so again, that's actually a great thing for engineering teams, what are we building next? What is coming? How can I think about architecture?
Todd Olson:
This is helpful for them. And then once you get this, publishing it externally, there's a lot of value to that, and customers are, they're buying a company, not buying a product very often. What that means is that the roadmap where you're taking things, your vision, they're buying that. Right. So they want to see it and they want to know what they're buying and that's okay. Show it to them and let them get feedback. So this is a great way to say, what do you think? Do you see anything missing? Is this the right things? Anyway. Great thing. So last thing I want to talk about is product operations, something to orchestrate all the stuff. I share a bunch of things, and look, the product management role is already really diverse one.
Todd Olson:
And the last thing we want to do is just add a bunch of other tasks to an already overloaded role within your org. And what's powerful, product operations has emerged as a new role, it's not as shocking emergence. We've seen operations roles, obviously it started probably in, well it started ops in finance but then of course we have revenue ops, which is the evolution of sales ops, we have now customer success ops, we have marketing ops, lots of ops, functions, not a shocker that we now have a product operations function, but it's important because these functions, these responsibilities just fell in the back of product managers. And it's hard if something's 5% of your job. And what we want is product managers focus on working with customers, working with engineers, building and innovating amazing product.
Todd Olson:
We don't want them managing and feedback request program. We don't. Trust me. Product operations will change your life if you find a way to invest in it and power it with both the tools and the resources to be successful. Here's just a quick, meetings where people ask me what they do. Well, they do a lot. Sales requests, working with customer success, managing customer commits, if you're doing those sorts of things, reviewing feedback and roadmaps, working with product marketing and competitive intelligence and making sure we're thinking about that with the product road mapping process, managing your beta process and release process, where are things in flight, someone's got to think across all teams, across all products, what's the customer experience? Who's managing that? Feedback in the form of NPS, who's handling that? All of these things are important things.
Todd Olson:
And again, it would be 5% of a product manager job. Make it a hundred percent of your product operations person's job and responsibilities. And the other one is this tool stack. Yeah, when I started my career in CDs, state-of-the-art tools for product managers was PowerPoint and Microsoft Excel. And now we have, I don't know how many categories are on this matrix, 12 categories of tools, all of which have a minimum of 10 tools in them. So 120 tools. How do you pick them? How do you manage them? Who manages them? How do you provision them?
Todd Olson:
Someone has to, right? A human, maybe you can say our IT team, No, your IT team doesn't know what your product management team needs. They're not experts. They can certainly evaluate the solutions. So this is an opportunity, an opportunity for having a role that owns this as an expert at it. And I think there's a lot of learnings there. Product is not used to procuring software, yet. But software powers the world and it's going to power this function, so it's just a matter of time, and product operations, to me, is the owner of the department that will do that. So look, we've covered a lot of ground. Thank you for sticking with me this long. So, as you see, putting product at the center experience, it means rethinking the way you deliver products. It means breaking away from the shackles of the CDs and those processes to find ways to do things different, to question those norms, why do we do things this way?
Todd Olson:
Find yourself asking those questions. So last question, how has your org evolve? Has it changed its practices? Has it released things? Has it managed customer feedback? Has it kept up with what state of the art is today? Who handles the tool stack for product? Do you even have a tool stack for products? Maybe you should. There's 120 tools out there, they weren't created for nothing. So who owns office? Right. So this is an opportunity, of thinking about the future and how to go. And look, all this is just about figuring way to steer through, when you're presented with different circumstances and different, in this case traffic, of course, how do you get through it? How you change the way you work? How do you question the norm? I mean, I think these are the things that I think, I want you to focus on leaving this talk. Here's a set of examples.
Todd Olson:
These are just three fantastic customers that we've had the benefit of working with. Jamf is invested heavily in product data to help inform their release cycles and success of certain new feature launches. Rapid7's UX teams is partnered, more part of the development process, focused more on customer feedback. I think there's a lot of great work they've done. Firefly has been one of the early adopters of product operations role and has leveraged that to redesign the release process and frankly really matured the organization, in this area. So just three examples, but I think, and there's plenty more out there, but this is an opportunity to be not afraid of change and to think about how you want to evolve your organization. And with that, I appreciate you having me. I appreciate you listening. Here's my Twitter profile, here's my LinkedIn profile. Feel free to direct message me, contact me. I love to engage folks in the audience. Well, thank you very much. Have an awesome day. Take care.
