Top 7 Expert-Backed Product Experimentation Tips

Product experiments are a vital part of scaling a product-led growth company. 

We drove a double-digit uplift by changing the value proposition of an anti-virus software company – it was unbelievable!

Experimentation can be as far-reaching as changing the structure of your organization to optimizing a tiny part of the funnel.

Through my years of experimentation, I’ve observed that optimizing the funnel can yield significant results, especially when playing with a large amount of traffic. While there are ample opportunities to do this in a B2C environment, you might have to get a little more strategic in B2B SaaS marketing environments.

Without further ado, here are seven vital tips and lessons I learned in my five years of leading experimentation with various product-led teams. 

Tip 1: Be prepared to fail… a lot. 

In the experimentation process, most of your product experiments will fail.

This is because you don’t yet have a sense of the type of business, customers, or users, especially when you are starting on a new team or business. 

Gradually, you’ll build a knowledge base through your first experiments, and the successes that follow will help you gain confidence. 

Eventually, you’ll discover that the most exciting part is to build learnings, and with the appropriate user research, you’ll gain a method that will underpin all your experiments.

Tip 2: Experiments that work within a B2C environment may not work in a B2B SaaS marketing environment 

What worked in one experiment in the past may fail a current experiment, so detach from preconceived ideas about a process or solution.

For example, experimentation in the B2C environment looks different from experimentation in the B2B environment, and applying the same approaches to both will not yield the results you want.  

In both cases, detach yourself from replicating product experiments that worked in the past and shift your mindset towards creating value for your users.

Here are a few ways B2B saas experimentation differs from B2C experimentation:  

• Traffic: B2C companies tend to draw more traffic, so you can afford to experiment more within a short period. As a result, you’ll get statistical validity quicker. 
  
  Conversely, there is limited traffic in the B2B environment, so fast experimentation is limited, especially in the startup world.  Small changes or optimizations are more appropriate in the B2B  environment. 

• Buyer persona: In the B2C environment, your buyer persona is your user persona. So when you combine these two aspects, it’s easier to design experiments. For example, when you optimize purchase flows, you can combine core aspects such as willingness to pay experimentation with value realization experimentation. 
  
  In the B2B saas marketing environment, it's much more complex. 
  
  You have to try different methods in order to show users the value by carefully nurturing their journey from awareness, consideration, and conversion. Sometimes it’s not even the user persona making the purchasing decision. 

• Experiments: The role of experimentation is to get validation that your idea is driving the desired outcome. Everything that comes before experimentation is right.
  
  In the B2C environment, you can jump right into experimentation and should focus more on bigger bets by leveraging traffic.
  
  While in B2B, the emphasis is on optimization. In the B2B environment, you must focus on small incremental learnings to make a big change, and you do this by optimizing for bigger bets. User research plays a crucial role in establishing these learnings. 

Tip 3: In B2B, don’t waste your time brainstorming.

Brainstorming is ineffective and expensive, especially when you prioritize ineffective ideas and experiments. 

Throwing ideas at the wall to see what sticks is intuitive, but it's a waste of time if those ideas often aren’t supported by anything. 

A common misstep is when executives or the highest-paid person in the office decides what the experiment should be. Oftentimes they come up with ideas that are not worth testing, nor are these ideas backed by user data.

There are a lot of risks involved in implementing product experiments without doing proper user research, and you’ll discover that part of the art of establishing an autonomous growth team involves nurturing their ability to fight back against misguided ideas.

Instead, root your ideas in combining business outcomes and consumer outcomes.

Since you don’t have the luxury of experimenting in the B2B SaaS marketing environment, focus on where the biggest opportunities lie in terms of product-led growth. This opportunity is often represented in an overlap between business and customer outcomes. 

Here are some strategies to find those opportunities. 

Start with user research

Ask vital questions about your user that will enable you to add value.

1. What are my user’s needs? 
2. What are they trying to achieve with your product? 

These are questions that drive value.  

Look for low-hanging fruit

A major clue is to look at areas in your business or customer journey that can drive the biggest outcome and quickest success? What quick changes can make you a consumer happier as well as help your team build confidence?  

Play with your funnel

Another technique is to look for problems in the business model. Categorize your growth area as either acquisition, activation, monetization, engagement, retention, and referrals. When you can place your customers in those areas, the opportunities become clear.

For example, if you have a freemium offer embedded in the engagement aspect of my funnel, I can question whether or not I’m giving users too much value and thus undermining my ability to monetize. So when you combine both business and users, the biggest opportunities will start to reveal themselves. 

Tip 4: Develop a hypothesis for your experimentation.

Leverage the scientific approach to product experimentation.

This involves building a hypothesis, which can reveal whether or not your experiment is worth testing. 

Structure your hypothesis by thinking of an Objective and Key Result (OKR). 

First, identify the area with the biggest opportunities and build an objective out of it so that when you experiment, your hypothesis will revolve around this focus area. 

Don’t try to experiment everywhere in the funnel.

Focus is important. 

One mistake companies often make is to try to experiment everywhere in the funnel all at once, which means that you won’t be able to focus on learning or establishing a good foundation for your area of interest.

You can narrow your focus in three steps:

1. Choose your area of interest: this is a particular area in the funnel, such as activation or consideration.
   
2. Define your OKR: your OKR is a measurable goal that guides your experiment. For example, “to increase activation by 30%.”

3. Work out a hypothesis that moves towards this goal: a good strategy is to choose an action led by the customer that achieves your desired result. This hypothesis explains why this experiment is worth doing. 

So you need to have action, the outcome, and the theory behind it in your hypothesis, which helps you frame your idea. If there is no strong theory behind it, you should ditch the idea.

Tip 5: Size matters (...at least in the beginning).

Experimentation isn’t one-size-fits-all. 

The degree of experimentation can vary depending on the company’s growth stage, resources, and niche. 

In the B2B saas marketing, you’ll need to get good enough traction – around 30K visitors per month – to drive meaningful experimentation.

Here’s what to do if you don’t meet the 30k threshold:

1. Focus on user research instead: User research will help you scale the growth of your product to be able to be driving experimentation. 
2. Gain Product-market fit: Product-market fit is proof that customers are buying into your idea. Once customers start buying, you’ll have a basis for your product experiments.
3. Focus on retention: Acquiring customers and retaining them is proof that you can offer a lifetime value.

But before jumping into product experiments, focus on low-hanging fruit. Think about areas that have a high volume of traffic where you can experiment more quickly. These areas must present a big problem, and this is a bigger opportunity for some quick wins.

For bigger companies, coming up with quick wins by going after low-hanging fruit can drive impact and help kickstart a change in the company’s culture regarding experimentation. 

My recommendation for big companies is to start with optimization. 

Optimization requires a very low engineering effort and exposes you to some quick wins. 

Tip 6: Friction in the customer journey can be a good thing.

In the conversion optimization world, we often hear that we should remove friction to make the customer’s journey as smooth as possible. 

My experience working with a company called Product Bar taught me that there is value in adding friction to the customer journey.  

For example, one of the core value propositions at Product Bar was connecting customer user insights to a feature to build a customer-driven roadmap. 

To gain feedback from users into Product Bar, users had to set up integrations to Zendesk or other tools to push feedback. Sometimes they needed authorization before they could integrate tools, which demanded more time. 

Although there was friction, users were able to gain access to the value quicker. Furthermore, we ran an experiment and got a triple-digit improvement in Activation just by introducing this integration friction.

In the B2B saas environment, friction points are necessary to distinguish the user persona.  

For example, you can offer an ebook to collect emails from potential prospects and retarget them with nurturing flows. A point of friction would involve asking for company mail instead of the personal email to further segment your user.

While this introduces more friction, you're acquiring the kind of people who are willing to pay for your product.

It's always good to consider adding friction If it can drive value for your users. 

Tip 7: Involve the whole organization.  

Some organizations think that product experiments are solely the job of the Growth team or Marketing team. 

This is wrong. 

From my experience, you need the whole organization to rally around running experiments. 

Introduce the hypothesis-driven mindset to the company culture to increase the chances of your organization’s success.

At Productboard, we leveraged cross-functional teams to spread this mindset. Our cross-functional team was composed of the Product Growth, Demand Generation, Customer Success, and Sales teams. 

While we were focused on impacting a specific part of the funnel, there were moments when everyone would think about experiments and try to come up with their own. Getting to this level of engagement in the organization indicates that your program is successful. Otherwise, experimenting in your siloed Growth team won’t work. 

Key Takeaways

1. Don’t be afraid to fail in the beginning: Your failures are an opportunity to develop learnings and improve your experiments.
   
2. Skip brainstorming: This can be a waste of time without the appropriate user research.
   
3. B2B and B2C methods differ: Know your category and experiment accordingly.
   
4. Be hypothesis-driven: Back up your experiments with data, and don’t forget to establish your OKRs.
   
5. Traffic size matters: If your traffic is small, take a step back and focus on user research. If your traffic is big, start with optimization.
   
6. Friction can be a good thing: Don’t be afraid to add friction to the customer journey, especially if It can drive results for your company.
   
7. Evangelize your organization: The Product Growth and Marketing teams shouldn’t be solely responsible for growth. Leverage the power of cross-functional teams to show the effectiveness of experimentation.

Most importantly, get started. Arm yourself with user research and aim for low-hanging fruit to build confidence in experimentation. Once you have built a confident team, you’ll be ready to spread that mindset to other teams in your company in no time.