The Online CMO by Philip Hallenborg

Despite many weeks of resistance, I had to yield to team pressure and start learning about the latest buzz word in site and offer development: multivariate testing (MVT). Apparently good old split testing (A/B)  is no longer good enough.

Jupiter Research state in a report from 2006 “Multivariate Testing and Site Optimization, STO-06-C06” that “Thirty-two percent of site operators in companies with $50 million or more in annual revenues have deployed site testing and optimization applications. Site operators should incorporate site testing and optimization into a usability framework that leverages traditional usability principles, customer satisfaction measurement, and Web analytics to comprehensively measure and improve Web site usability.” 

We are now in 2008 and I can clearly see that MVT has gained a strong foothold in my organization. I guess the $50m kind of defines the scope of MVT as a growing set of applications for big corporates.

So what is Multivariate testing? 

MVT can be described as a quantitative way to understand and influence your customers’ user experience. According to Wikipedia multivariate testing is “a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous split tests or A/B tests performed on one page at the same time. Split tests and A/B tests are usually performed to determine the better of two content variations, multivariate testing can theoretically test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computational power.” 

The primary purposes of testing in order of importance (according to Jupiter research) are: 

1. Site content
2. Promotions
3. Step processes e.g. checkout
4. Landing pages
5. Site navigation
6.  E-mail marketing
7. Performance metrics e.g. revenue, average order value 

In setting up an MVT, Jupiter recommends that you: 

1. Create hypothesis for testing on user behavior e.g. lifestyle images connect better with target segment than product images.
2. Establish multiple objectives such that not only conversion is tested but the general interaction with the site and fulfillment of KPI e.g. revenue/ user.
3. Segment test participants based on the hypothesis – go narrow rather than wide.  

As I start to use this new tool I will update my blog with more in depth views on this emerging field of optimization tools.