Rapid Experimentation: The Road To Innovation (Complete Guide)

You read it left and right, companies that owe much of their success to experimentation….

Of course, experimentation can be understood in a hugely broad way, so in this article I’m going to get you started with:

1. Understanding why experimentation is important
2. Starting experiments
3. Supervising and organizing experiments

This blog article can also be seen as a summary of the book Testing Business Ideas by David J. Bland and Alex Osterwalder.

Let’s get started quickly.

Why start experimenting?

Innovation today is happening at an unprecedented rate, it’s more important than ever to innovate.

To innovate with your business you will need to start implementing ideas, but how do you avoid spending attention, time and money only on ideas that look best on paper?

Experimentation allows ideas to be tested without spending a lot of time, energy and resources on them:

Hypotheses + Experiments + Insights = Less Uncertainty & Risk

1. Design

Phase 1 is the startup, or the structure of the team and the course of ideation….

1.1 Form the team

Teams almost always consist of the following disciplines:

It is important to have diversity on your team because everyone has different skills, past experiences and perspectives.

Behavior

There are 6 factors that a successful team meets:

1. Data influenced: insights from data fill the backlog and inform the strategy. Here, it is not necessarily necessary to be data-driven, but you must be influenced by the data.
2. Customer centric: it is important to know the ‘why’ behind the work.
3. Iterative approach: teams should always work towards a desired outcome in the form of a repeating cycle of processes.
4. Experiment driven: teams must dare to be wrong and not just be focused on making features.
5. Entrepreneurial: move quickly and validate assumptions. Think creative problem-solving at high speed.
6. Question assumptions: don’t be afraid to test out a disruptive business model, don’t always play it safe.

As experiments become more complex, your team will also grow:

Environment

Teams need an environment in which they can develop themselves, for this purpose teams must be the following:

1. Dedicated: teams must be dedicated to the work. Multitasking between different projects is bad for progression.
2. Funded: experimenting costs money, based on the learnings during stakeholder reviews budget can be allocated (venture capital style, to follow).
3. Autonomous: teams must be given room to work. Don’t micromanage the team where progress is delayed.

For a company it is important to offer the following:

1. Support
   1. Leadership: a facilitative leadership style is desired because no one knows the exact solution. Lead with questions, not answers.
   2. Coaching: coaches with extensive experience of experimentation can inspire the team.
2. Access
   1. Customers: teams need access to customers and should definitely not be isolated from them.
   2. Resources: teams need enough resources to make progress and generate evidence.
3. Direction
   1. Strategy: teams need to be clear about where they are going and what strategy they are going to use. By not having a clear strategy you will confuse being busy with making progress. You could use the OGSM canvas for this.
   2. KPIs: teams need to be able to show how much progress they have made. Think for example of the North Star Metric.

Strategyzer recommends the Team Alignment Map.

1.2 Describe the ideas

The design loop has 3 steps:

• Ideate: come up with as many alternative ideas as possible based on your intuition and insights. In doing so, don’t fall in love with your first ideas.
• Business prototype: start with a low-effort prototype such as an outline, filling in a Value Proposition Canvas and a Business Model Canvas / Lean canvas. As time passes, your prototypes will become more professional.
• Assess: eventually you will start asking yourself questions such as:
  • Is this the best way to address our customers’ jobs, pains and gains?
  • Is this the best way to monetize our idea?
  • Does this take everything we learned in testing?

2. Test

In phase two, we actually get started….

2.1 Make up hypotheses

In making hypotheses you start with ‘we believe that …’, but avoid trying to prove only things you believe. In fact, you can also make a hypothesis based on ‘we believe that … NOT/REALLY …’.

There are three properties that a good hypothesis must meet:

1. Testable: hypotheses must be testable and have an outcome of true/false.
2. Precise: it must be exactly clear what success looks like. Ideally, it also describes the what, who and when.
3. Discrete: an experiment should only test one thing otherwise it should be split into multiple experiments.

There are three types of hypotheses:

1. Desirability: customers are not interested in it. Plays into the following components of the Business Model Canvas:
   • Customer Segments: we target the right segment and that segment is large enough.
   • Channels: we make (good) use of the right channels.
   • Value Propositions: we have the right (unique) value proposition(s).
   • Customer Relationships: we have a good relationship with customers and we know how to keep them.
2. Feasibility: it is impossible to realize. Takes advantage of the following components of the Business Model Canvas:
   • Key Activities: activities can be scaled up without loss of quality.
   • Key Resources: important resources can be scaled up.
   • Key Partners: we have the right partners.
3. Viability: there is no money to be made. Speaks to the following components of the Business Model Canvas:
   • Revenue Streams: customers actually want to pay for our product or service.
   • Cost Structure: costs can be managed and are under control.
   • Profit: we can make a profit.

Make sticky notes with the different types of hypotheses and paste them on the appropriate section in the Business Model Canvas.

Then you can start prioritizing the hypotheses:

2.2 Experiment & Learn

To experiment, it is helpful to get started with the following tools:

• Experiment Card: describe the reason for and the setup of the experiment.
• Learnings Card: describe the insights you gained from the experiment.
• Experiment Canvas / Airtable: document your experiments.

There is a difference between weak and strong evidence:

To share your experiments with stakeholders you can use the Experiment Design & Analysis.

2.3 Decide

Insights alone don’t do you much good, it’s about what you do with those insights.

Deciding is about taking action, in the sense of:

• The next steps you will take in removing uncertainty and risk.
• Making decisions based on the insights.
• Deciding to complete, change, or adopt an experiment.

2.4 Manage

To share knowledge and provide structure, it is necessary to schedule the following:

This seems like a lot, but it’s not too bad in the end:

Weekly Planning

Duration: 30 – 60 minutes.

When: once a week, after the weekly learning.

Participants: core team.

Agenda:

1. Devise hypotheses.
2. Prioritize experiments
3. Schedule experiments on your project management board:
   

Daily Standup

Duration: 15 minutes.

When: every weekday

Participants: core team.

Agenda:

1. What is the goal today?
2. How are we going to achieve that goal?
3. What is still in the way?

Weekly Learning

Duration: 30 – 60 minutes.

When: once a week, for weekly planning.

Participants: core team and extended team.

Agenda:

1. Collect evidence from experiments.
2. Generate insights, look for patterns in outcomes of experiments. Be open-minded in this, make sure you don’t overlook any (unexpected) insights.
3. Go back to your Business Model Canvas, Value Proposition Canvas and Assumptions Map with the new insights and make updates. Then you can incorporate your insights into your strategy.

Biweekly Retrospective

Duration: 30 – 60 minutes.

When: once every two weeks, after the weekly learning and before the weekly planning.

Participants: core team.

Agenda:

1. What is going well?
2. What needs to be improved?
3. What are we going to try next?

In addition, you can add other options, such as:

1. What should we start with?
2. What should we stop doing?
3. What should we keep?
4. What should we do more of?

Monthly Stakeholder Reviews

Duration: 60 – 90 minutes.

When: once a month.

Participants: stakeholders, extended team & core team.

Agenda:

1. What have we learned?
2. What is holding back progress? (document blockers during the month)
3. Pivot / Persevere / Kill decision

To communicate effectively with different departments, there are guidelines you can follow:

• Our customer segment is _____
• The total number of customers involved in our experiment is about _____
• Our experiment will run from _____ to _____
• Information we acquire in the process is _____
• Branding we use is _____
• Financial resources we need are _____
• We can launch the experiment by _____

3. Experiment

In phase 3 we dive deeper into the different experiments….

3.1 Select an experiment

To select an experiment you can use the ICE Framework:

1. Impact: how big could the impact of this experiment be?
2. Confidence: how confident are we that this will work?
3. Effort: how difficult is it to conduct this experiment?

Some thumb rules:

1. Be cheap and fast at the beginning: you hardly know anything about the result yet, don’t make yourself too dependent on this experiment.
2. Strengthen your evidence with multiple experiments on the same hypothesis: don’t make decisions based on weak evidence or one experiment.
3. Choose experiments with strong evidence: design experiments so that you always have a strong evidence as an outcome.
4. Reduce uncertainty as much as possible: to test a hypothesis you don’t necessarily have to build something from A-Z.

3.2 Discover

3.3 Validate

3.4 Sequences

4. Mindset

When experimenting, it’s important to provide the right leadership from the top down

4.1 Avoid experiment pitfalls

There are several pitfalls you should not fall into:

1. Time trap: not dedicating enough time. You get what you invest in, spend enough time each week testing, learning and deciding.
2. Analysis Paralysis: thinking too long about things you just need to test and apply. Get out of the building instead of just keeping thinking about ideas, timebox your analysis.
3. Incomparable data / evidence: unclear data that cannot be compared.
4. Weak data / evidence: only measure what people say, not what they actually do.
5. Confirmation bias: only believe evidence that matches expectations.
6. Too few experiments: doing only one experiment on key hypotheses.
7. Failure to learn and adapt: spending too little time analyzing evidence and generating insights and actions.
8. Outsource testing: outsource what you should be doing and learning from yourself.

4.2 Lead by experimentation

There are some things you need to think about as a leader:

1. Language: be careful with your choice of words, even if you have a lot of experience and knowledge don’t steer your team too much in the direction you want. Eventually your team will start waiting for you to assign them experiments.
2. Accountability: focus on business results, not just features and dates.
3. Facilitation: don’t talk too much in I, me or mine and what date something has to be finished, but more in we, us or our and how certain results will be achieved.

There are several steps leaders can take to facilitate this:

1. Enabling environment: make enough resources and time available to iteratively test ideas.
2. Evidence trumps opinion: experience and track record don’t mean that much, evidence from testing is more important than the opinion that comes from experience.
3. Remove obstacles and open doors: lack of access or specialized resources must be fixed, often there is even insufficient access to customers.
4. Ask questions rather than provide answers: important to push teams to test better value propositions and business models.
5. Meet your teams one-half step ahead: bring your team into the process, don’t leave them behind. Check with yourself where you want to see your team standing and then figure out how you are going to get them there. One-on-ones, retrospectives and walk-throughs can help with this.
6. Understand context before giving advice: before you start giving advice, you need to understand the context. Let people talk and then ask questions to make it clear.
7. Say “I don’t know”: if you don’t know something, just say so. You can’t have all the answers, ask questions to back this up:
   • How are we going to do this?
   • What do you guys think?

4.3 Organize for experiments

Often it is not exactly clear at the beginning what a solution will look like. Things change along the way, which is why it is useful to have cross-functional teams:

This will, in fact, allow for faster response to change.

In addition, it is important to adopt more of a venture-capital funding style rather than large budgets on an annual basis, because that incentivizes bad behavior:

4.4 Testing principles

There are a number of principles to keep in mind while experimenting:

1. Evidence is better than opinion.
2. Learn quickly and reduce risk by embracing failure.
3. Test early; perfect later.
4. Experiments do not equal reality.
5. Find the balance between learnings and the vision.
6. Start with the most important tests, which can undermine your entire hypothesis/idea.
7. Make sure you understand your customer first.
8. Make it measurable.
9. Accept that not all facts are equal, an interviewee may say one thing, but do another.
10. Double test important irreversible decisions.

Are you ready?

So, now you are armed to start Rapid Experimentation….

Now I want to know from you, what has been your most successful experiment so far?

Let me know in a comment.

P.S. if you want any additional help, let me know at [email protected].