Experimentation as a business function: what to focus on?

We have all heard of the Obama administration using A/B testing (which led to the founding of Optimizely), the scale of experimentation and culture that Booking.com has reached, and the commercial success of Google.

So, how and where do you start?

There are numerous articles that highlight experimentation from Netflix, Disney+ and similar. Each one is amazing, as they are written from a view of engineering, data science, and analytics to progress and further knowledge of experimentation within the community.

However, this article is written from a different perspective — experimentation as a business function.

What is experimentation as a business function?

The Lumery sees experimentation as the facilitation of quantifiable customer choice to achieve consistent growth at scale. Consisting of three key areas of growth: Creation (experimenting with something new), Iteration (optimising something to improve), and Lift (focusing on a conversion rate change).

As a business function, experimentation supports an organisation to:

• Quantify and support decisions
• Reduce the risk of new concepts
• Discover customer preference

What to focus on when building Experimentation as a business function

First and foremost, those who are responsible for experimentation within a business are the enablers for other people and teams. Experimentation as a craft is an enablement function for others to work within, to generate outcomes.

It is not about providing opinions, or changing someone else’s design and/or strategy. Instead, it is about enabling others with the knowledge, tools, and processes to achieve a stronger, more informed outcome.

To achieve this, we look to the following five pillars to support the creation of this function within any organisation:

Mindset & culture

Almost every executive state that they want an experimentation mindset, or a test and learn culture. However, creating and fostering that environment is where things either thrive or fall over. When supporting a large Australian airline build their experimentation culture, I remember sitting around a table where a product owner sheepishly had to admit to an experiment that cost the business money. The head of digital at the time, praised the person for taking a risk and asked for the learnings. This interaction immediately set the tone for what was possible for that business from there on out.

Connected technology

Experimentation has no boundaries if your data and technology is connected, allowing teams to experiment with more than websites. Connecting technology can allow you to test algorithms, offers, search results, and even business models. We once supported a client experiment with the creation, delivery, and fulfillment of a new physical offering. This would not have been possible if their core systems were not integrated with the right data, not only for the facilitation of the product, but for measurement of the outcome.

Scale via process

If you think about experimentation being an enablement function, then process is key. The answer is not to simply put more humans into the mix just to do more things. Instead, it is tweaking the methodology of delivery, enabling teams, and educating them for scale.

Insight led experiences

It is important to address the HiPPO (“highest-paid person’s opinion”) in the room. Assuming that your organisation has already solved for an experimentation mindset and culture. Everything should be insight-led, bringing together the ‘What’ (Quantitative) and the ‘Why’ (Qualitative) of each customer experience. Remember, experimentation is about facilitating quantifying customers' choices — not muddling that process with biased opinions. Your teams must be grounded in data-backed and insight-led decision making.

Framework based strategy

Finally, repeatability is key. Experimentation should be scaled by creating repeatable strategic patterns that others can adopt (Frameworks). We once supported a company with 12 development squads (around 80 people) build out a scalable framework that enabled their teams with repeatable patterns to experiment. If we didn’t do this, it would have been the wild west of product and development.

The takeaway?

Do not ask for a culture of experimentation to happen instantly. Put in the time to support those within your organisation to be bold, try new things, and learn from their failures along the way. Focus on the capabilities and mentality needed within an organisation to embed a function that enables teams to scale growth.

‍