A/B Testing is not art, it is science.
A/B Testing is not art, it is science.
By Ramkumar Ravichandran, Director, Analytics, Visa Inc.
Call it A/B Testing, call it Optimization, call it Growth Hacking, call it Test & Learn, call it whatever you will, the fact remains that it is the most potent weapon in the arsenal of a Decision Maker. It is something that will help determine as close to certainty as possible, what will work and what will not.
What has changed?
Experimentation is complex & intricate task from the analytical technique that goes into it, to setting up the experiment, tracking and reading it. A decade ago, a Champion-Challenger Test would take a year to set up, send out mailers, wait for a quarter to check for impact and few more months to read it. All the what-if scenarios were either analytical exercise in themselves or a new test. The advent of technology, change of mindset, dynamic market conditions and ever-evolving Consumer needs today made A/B Testing simpler, faster, more cost effective and easy to understand. This has created an interesting challenge of with democracy comes chaos ☺. Everyone can use it, but not everyone should do it. So we end up in situations as in the video (it is just a satiric video and has no relation to any firm or team)
Why would you say so?
A/B Testing seems to be simple exercise of testing two variations, measuring some numbers and knowing what is what.
However the real picture is infinitely more complex. Experiment Design is a science on which whole concept of A/B Testing is based at. Yes, we shouldn’t be nerdy and overcomplicate a simple task but if we deviate too much from the principles, the results are muddy. At the same time, it is costly to set up & run and so RoI calculations are must. There are many cheaper & quicker alternatives (Research & Analytical methods) to get directional insights for decision making at various stages of the product.
Wait, so what should we do?
Create a cross functional group (Product, Marketing, Analytics, UED, Engineering & QA); lay down guidelines to determine what/when/where/how long should be tested; work with engineering closely to integrate the optimization tool with the analytics tools & even be able to iterate the implementation itself; standardize framework for readouts and recommendations; level set and manage expectations and learn from A/B Testing.
Hmmm, you saying there is still hope in the world…
All the major firms, institutions, governmental bodies, etc. have realized the power of A/B Testing and are making significant investments in the space. This combined with the breakthroughs in other areas – software development & internet technologies; data collection, storage, processing and analysis; Instrumentation in IoT world making it possible to connect better; tools and technologies to bring all the various components together easily and focus on effective Decision making practices, is making it the “data driven” era. Yes we can and yes we will ☺. We just need to keep our eyes open for latest trends, ears stuck in the direction of thought leaders, heart for accepting feedback and gut for taking on bigger challenges and making it happen.
This and more would be discussed at “Business Analytics Innovation Summit” at Chicago May 21st and 22nd, 2015.
The summit is lined with amazing speakers from across a wide spectrum of industries, Government, Institutions, etc.
It is also full of great audience with ideas, passion and feedback. I was part of this event last time and it was an enriching experience. I came out a better professional than I went in. For further questions about the event, please reach out to Elliot Pannaman, the International Events Director of The Innovation Enterprise.