Validating Hypotheses
After identifying core hypotheses in the Design Workshop (or optionally in the Validate Workshop), you are ready to validate them by conducting experiments. We describe key activities to design and conduct these experiments and explain examples. We pay special attention to the level of reality of experiments and customer investments you have to consider.
Sabine Schoen
Business Design Field Researcher
Content
1. Purpose
In the Design Workshop (or optionally the Validate Workshop) you defined and prioritised your testable hypotheses and figured out experiments to validate them. Again, be careful with the word "validate". In this context, validation means that we have a clear idea in mind what a new product may look like, a certain way of selling to customers and specific marketing slogan we want to test etc. If our level of knowledge about this idea is quite high, you can definitely think in "hypothesis" and experiments to validate or (more accurately) falsify them. If your idea is still kind of vague or you are still in the mode of understanding your innovation context, don't use the word "validate". This is not what you need. Keep exploring the world and formulate open questions rather than hypotheses - similar to the Discover Phase. This happens a lot specifically in the first iteration of a Business Design project.
Let's assume, you have hypotheses and want to validate them: Now it's time to design, prepare and run experiments that help you gather the facts you need. There are many ways to find out if your business model is feasible, profitable, desired by customers and fits to your organisation. Experiments never look the same! They have to be planned wisely.
If your hypothesis does not include numbers (= "threshold"), the likelihood is high that you better switch to "exploration mode" (see figure).You don't know enough to really validate things!
A statement such as "We believe our new product will meet our customers' expectations!" is not a testable hypothesis and eventually a sign for insufficient knowledge to phrase a hypothesis. In contrast, a statement such as "We believe that 30% of our selected customer of segment X will keep using our new product on a daily basis for at least 3 months." is what you need to move on here.
2. Duration
5 weeks (see Validate Phase)
3. Key Activities
The following activities represent the core steps to validate hypotheses:
Design experiment: The research design of experiments is highly dependent on your topic, the stage you are in and your team. Therefore, we cannot provide you a blueprint of the RIGHT experiment for your question or hypothesis. We always strive to design experiments that are repeatable, which means they reproduce always the same results if we execute them over and over again. In many cases, experiments consist of a combination of several Research Tools (e.g. a landing page incl. a tracking mechanism plus a survey). Create a structured plan of your experiment including
Method(s) & tool(s)
Procedure & period
Participants
Prototype(s) or Lean Offering(s)
Timeframe
Threshold (and how you measure it)
...and do a dry run if possible. Discuss the level of reality / evidence of your experiment in your team (see below).
Prepare experiment: After having a clear picture of how the experiment will look like, usually some preparation work follows. This may include the development of guidelines / storylines, the development of a prototype / lean offering, the identification and invitation of participants or marketing activities, the setup of tracking mechanisms, the briefing of team members, the collection of data you need for calculations and so on. Again, there is no blueprint. This is highly dependent on your setup.
Run experiment and analyse results: When running the experiment, tracking results and measuring the output based on your threshold is crucial. A dashboard helps to stay tuned. Stay flexible to adapt your research design immediately if you learn something new on the way (e.g. how to improve the storyline of your pitch or how to phrase your headline on your landing page).
4. Participants
(Potential) Customers
5. Tools & Materials
See Research Tools for an overview
6. Instructions for Coaches
Make sure your team is testing the right things. Teams tend to focus on ease of use issues and forget about the general problem / solution fit or if their customers see value in their offering. When you have the feeling that your team is having a hard time to focus on the most important hypotheses, you could ask them whether they would invest their own money based on the results of their experiments. If not, they should redesign experiments on which they can rely on.
Furthermore, teams tend to gather and interpret information in a biased way to support their existing beliefs (see box below). Take care that they act like "real" researcher and design, conduct and analyse their experiments in a neutral way.
Make sure that the whole team is involved when they are analysing the results of their experiments (combined with your neutral view as Business Design Coach) to mitigate biased interpretations.
Whenever your team wants to test the desirability of their offering, they should think about customer investments as a threshold (see below). Take care that your teams are defining appropriate investments before conducting experiments. The investments should fit to the phase you are in. At the beginning of the Validate Phase low levels of investment are typical. As soon as you can allow more reality, the level of investments should increase as well.
Whenever teams gather information to make decisions we must be aware of the so called confirmation bias (find more about it here). During the Validate Phase it may influence how we conduct our experiments and how we interpret results. The term validation already tends towards confirmation bias to be honest. As a coach, you should be aware of it and create awareness in the team. Follow our instructions for coaches (see above) to mitigate biased interpretations.
7. Level of Reality & Customer Investments
While planning your experiments, consider the level of reality / evidence. Some experiments e.g. interviews allow just a low level of reality - even if you include a prototype. It's always an experimental setting (see a funny example in this video). However, the highest level of reality can be achieved when you charge your customer and money comes in. Have a look at the examples in the picture for inspiration.
Another example for the downside of market research (in experimental settings) is the launch of New Coke in 1985 (see here). It is highly risky to focus only on one dimension (e.g. taste) and ignoring all others.
If you aim to test the desirability of your offering with customers & users, you should think about "investment" you are expecting from them (as threshold of your experiment). Compliments are not enough - they cost nothing! Investments can be financial, time and social and show how committed customers and users are regarding your offerings. Examples are:
Financial investments
Pre-ordering products
Deposit
Time investments
Invitation to event / next meeting
Co-creation (Contribution to development)
Test user for trial version
Active research on the web
Dwell time on webpage
Registration for newsletter / event / training
Social investments
Introduction to colleagues or friends
Recommendation
Testimonial
Exceptional emotions
The investment you ask for depends on you, your hypothesis and experiment, your prototype / lean offerings and your customers & users. The higher the level of investment your are looking for, the more reality is needed.
8. Examples
Hypothesis | Experiment | Prototype | Customer Investment |
"We believe that we reach the majority of our customers by online ads on the webpages of our three partners." | Online ads We advertise online for two weeks on the chosen partner pages and count clicks. We are convinced, if we reach a conversion rate of 3%. | We prepare a landing page presenting the core value of our offering(s) and include a newsletter registration. | Registration for newsletter (T) If users are interested in our offering(s) described by the information given on our landing page, they sign up for a newsletter to stay tuned. |
"We believe that our offering(s) lead(s) to significant time savings of X for our customers." | 2-weeks trial We offer a 2-weeks trial and ask 20 potential customers to participate. We are convinced, if we get ten participants and five positive testimonials. | Functional prototype We prepare a functional prototype including our core value that should save the time of our customers. | Participation & testimonial (T/S) If potential customers are interested in our offering(s) they participate in the 2 weeks trial and if they are convinced afterwards they agree to write a testimonial we are allowed to publish. |
"We believe that we can get enough (= X) deposits to finance material costs of our first product series." | Sales pitch We pitch our offerings 20 times within the next two weeks and show our sales brochure. We are convinced, if we get 10 deposits of 100 €. | Sales brochure We prepare a sales brochure presenting our offering(s) as well as the core value we provide for our customers. | Deposit (F) If customers are highly interested in our offering(s) after our pitch, they will pay a deposit. |
"We believe that we can get enough (= X) pre-orders to buy the machine we need for production." | Kickstarter We start a kickstarter campaign and count pre-oders within a given timeframe. We are convinced, if we get 500 pre-orders. | Kickstarter page / landing page We prepare a kickstarter page (and landing page) presenting our offering(s) in detail. | Pre-order (F) If customers are interested in our offering(s) described by the information given on our kickstarter page (and landing page), they place a pre-order on kickstarter. |
F = Financial investment | T= Time investment | S = Social investment