Experimentation Is Essential for Every Startup. Are You Doing It Right?

The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill – Albert Einstein

Experimentation is a powerful tool for businesses to innovate and test new ideas, but few seem to be using this tool right. Erroneous setups lead to failed experiments and lost time, which causes businesses to give up on experimentation and innovative ideas.

Understanding how to run experiments right will help you and your company put your innovative ideas to use and drive higher value for your business. This article is among a three-part series where I discuss experimentation in detail. This one is specifically focusing on the dos and don’ts. Here are the five key elements of a successful experiment:

Related: How Experimenting With My Idea Back In College Helped Me As An Entrepreneur

1. Risk assessment

Sometimes an experiment can negatively affect the “participants” and lead to churn. It’s best to plan for the potential consequences of an experiment on the users and their mitigation plans as part of your experiment design. Analyzing these risks helps avoid counteractive effects on the users and business.

2. Success metric

Often, there are several metrics that are essential for companies to track, and it’s easy to get lost in between these metrics as the experiment is underway. Identifying the most critical metric best suited for the business before launching the experiment is crucial, so that experiments don’t leave you hanging with inconclusive results.

If you have multiple success metrics, it is best to identify the most important metrics which are independent of each other. You should also clearly define the outcome possibilities and their possible interpretations, since metrics can go in different directions as part of the experiment.

3. Sample size calculation

Every study will have an ideal sample size based on the requirements, and one must carefully calculate the sample size range for every experiment. While it’s suitable to keep the sample size to a minimum to reduce the risk of an experiment’s failure, the sample size should be large enough to minimize the error.

Related: How I Built a Company the Lean Way — by Using the Scientific Method

4. Mutually exclusive cohorts

Teams in startups move fast and try to learn through experiments by launching numerous experiments simultaneously. While parallel experiments are excellent for improving product features on all fronts, it is vital to have separate sample populations (i.e., distinct control and test populations for different experiments). With the same cohort of users exposed to various experiments, there may be varying and contradicting impacts on the success metrics, leading to unpredictable outcomes. Hence, ensuring that the cohorts chosen for all the experiments are mutually exclusive, is crucial.

5. Quantification of the impact

Although overlooked, quantifying the impact of the success metric on the business (typically revenue or profit) helps understand the importance of an experiment. An estimate of how much revenue or profit the company stands to make with every increment or decrement in the experiment success metric will help you compare the experiments. Experimentation can be expensive, so by quantifying the impact of each experiment, you can prioritize the experiments and understand whether it is reasonable to conduct one.

4 common reasons experiments fail

Several considerations are required to derive conclusive results from experiments. The team often has several innovative hypotheses but struggles to understand the challenges of randomized experiments in this hyper-competitive business landscape. While challenges that sabotage experimentation results could be innumerable, let’s focus on the four most common reasons:

1. Weak hypothesis: Sometimes, the team tends to go with an incompletely defined hypothesis, planning to make changes and make decisions on-the-go as they get directional results with an initial experiment. This strategy could end up in multiple ineffective experiments with unfavorable business decisions. It could potentially waste significant development and design hours, making it difficult to follow up with actionable change recommendations.

2. Unclear success metrics: Don’t experiment without a clearly defined success metric. You must identify the most important metric you want to influence with the feature change. Ideally, an experiment should have a maximum of two success metrics, and one must design the experiment considering both. One could often get lost in the various indices without a clear success metric. People often think — “This is a complex test. Some things will go up, and others will go down. How do I test it?” One way to deal with this is to create a combined metric you know you want to influence. Another common issue is that the success metric is a longer-term one. For instance, it could be a 90-day retention or engagement rate. How should you measure this by running the experiment just for six days? Historical analysis helps in this context by looking at earlier patterns that can help you forecast the 90-day outcome from a 6-day experiment. This approach will help you define a success metric for a 6-day experiment that you can easily measure. There is always a way to build a robust success metric, but the key takeaway is that if you launch an experiment without a success metric, you are unlikely to get a successful outcome.

3. Missing risk analysis: Sometimes, a poorly implemented change experiment can disappoint an active user. The issue here is not assessing the potential risks while checking the feasibility of an experiment. You must remember that some experiments can have counterintuitive outcomes that can damage your business. Hence, a risk assessment before experimentation will save you from surprises and help you make data-backed decisions.

4. Online calculator: There are several online experiment sample size calculators. The key here is to understand that determining the sample size is not a purely mathematical calculation, and several qualitative and industry subtleties influence them. Other elements, such as success metrics and criteria, also affect the experiment sample size. If you do not account for these, you might not get the ideal size that will give you the answers you seek.

Related: Transform Your Business by Encouraging Experimentation and Change

Experimentation can be your friend and a business enabler, too. An experiment design with a well-articulated hypothesis, success metrics, cohort identification and a risk mitigation plan will help you get the best results from your experimentation ventures and help boost your business success. The following post in this series will discuss the common consumer behavioral biases influencing experiment outcomes.