5 Steps for Developing a Good Hypothesis for Product Experimentation
In this article, we will discuss how experiments can be best utilized, all by asking better questions.
What is a Hypothesis?
In the scientific method, a hypothesis is an educated proposal based on limited evidence. Through direct testing and experimentation, proper empirical evidence can be gathered to support or counter the initial proposition, thereby proving or disproving the hypothesis.
A hypothesis must also be testable. It is not just a guess, but an informed prediction based on existing theories and evidence. It must be falsifiable in that it can be refuted from the perspective of science, following the laws of cause and effect.
A hypothesis requires two types of variables: 1) one where a researcher changes or controls (independent variable), and 2) one where the researcher observes and measures (dependent variable). Declaring clear and defined variables will increase the integrity of the experiment, as it’s ultimately what’s being measured and quantified, providing evidence to support (or refute) a relationship between the two or more variables.
Steps to Developing a Hypothesis
So what are the steps to writing a good hypothesis anyways? Generally speaking, it can be broken down into repeatable 5 steps:
- Ask a Question
- Do the Preliminary Research
- Formulate the Hypothesis – Give an Answer to your Question
- Refine the Hypothesis
- Create the Null Hypothesis
While these steps work for any empirical research topic, our interest is for product experimentation – so remember to always tie your questions and focus onto business impact. Furthermore, it’s time to go deeper into the steps.
1.) Ask a Question
First step’s first: What are you curious about?
Within the bounds of the scientific method, the question should be focused, specific, and researchable within the constraints of your product experiment. As with all questions, it will be posed by the classic six: who, what, where, when, why, or how.
Some examples include:
- How does [change x] affect [product feature a]?
- Why does [behavior n] occur, when [change y] is introduced?
- What customer group is affected most when [change z] is introduced?
While these types of questions might be an okay place to start, it does not make for a good hypothesis in itself, especially when measuring metrics leading up to and after a product experiment. But before we can transmute our initial question into a testable hypothesis, we must first do the preliminary research.
2.) Do the Preliminary Research
Fire up your favorite Internet search engine, because it’s time to start your preliminary research!
To begin answering your initial question, you will need to consider the following:
- What do you already know about the topic?
- What studies have already been conducted?
- What variables should best be selected, based on this preliminary research?
From here, we now have more information to anchor our thoughts on as we continue to formulate the hypothesis statement itself.
3.) Formulate the Hypothesis Statement (AKA, Answer the Question)
Now that you have done your preliminary research, you now have a general idea of how to answer your initial question, or at least the arguments available that could answer it.
For more granularity, it is recommended that the hypothesis statement be framed as an ‘if… then’ statement, for example:
- If [change x] is implemented to [product feature a], then [behavior n] is expected.
A more specific example, using the template above is as follows:
- If [Character Versus Poll ] is [embedded / placed ] directly into the content or article, then customer participation is expected to [meet a threshold of 1.5% ].
This statement shows you:
- What is being studied: Character Versus Poll
- The Variables: Placement of Poll (e.g., Top Menu, Embedded, Right-Rail)
- Prediction: Participation Rate meets a threshold of 1.5%
4.) Refine the Hypothesis
You may want to refine your research hypothesis to address these two additional instances: 1) correlation study, and 2) studying the difference between two groups
A correlation hypothesis might be:
- Changing the “Buy” icon from Green to Orange, will have a negative impact on the click-through-rate to the checkout page, and decrease sales by 50%.
A hypothesis showing difference might be:
- Registered Users with less than 10 followed franchises are less likely than Users with more than 10 followed franchises to participate in the website’s Poll.
5.) Create the Null Hypothesis
In Product Testing & Experimentation, oftentimes your study will require statistical analysis on the data you collect – especially if you are looking to find if a correlation between a predictor variable and an outcome variable is statistically significant.
The null hypothesis is the default position that there is no association between the variables. The null hypothesis is written as H0, while the alternative hypothesis is H1 or Ha.
- H0: The number of lectures attended by first-year students has no effect on their final exam scores.
H1: The number of lectures attended by first-year students has a positive effect on their final exam scores.
Experimentation is a centerpiece of an iterative approach to innovation, yet to experiment means to test the assumptions we have about our customers, our products, and our business. Clearly stating those assumptions in a format that allows them to be disproven is the idea behind developing a hypothesis that is subject to experimental rigor. Once a hypothesis and its null have been articulated, we can proceed to design an experiment which can bring us one step closer to clarity in our decision making.