Tuesday, November 14, 2006

Case Study: A/B Testing

During my consulting stint for an online electronics retailer, we came accross a "situation" where we needed to run an A/B test. Below is a description of the test and the results.


The online retailer had 3 web sites that related to this case.

1. Site A - A main portal site (the branded site with the most visitor traffic) with direct links to their other sites

2. Site B - The retail site where visitors could shop for and purchase the electronics

3. Site C - The educational site where visitors could learn about the products before they made a decision to purchase

In this case, the marketing team decided that they wanted to push the awareness a particular new camera product. To do this, the team created an internal banner and placed it on Site A (the high trafficked portal site). Upon clicking on this banner, visitors were sent to Site C so that they could educate themselves about the camera product in more detail.

Once on Site C, if the visitor was interested in purchasing the product, they could click on a "Buy Now" link that took them to the camera's product detail page on Site B.

The "Situation"

After the banner was running for a short time, a major question arose.

1. Were we losing revenue by sending the visitor directly to Site C (the educational page) first instead of sending them directly to Site B (the purchase page)?

The asumption was, that by sending the visitor first to Site C, the visitor had to take an extra step in their browsing path in order to reach the purchase site. Why not just send the visitor directly Site B from Site A and save the extra step?

The reason was because Site B and Site C were run by two different groups within the company. The business goal for Site C was to get as much visitor traffic to their educational pages while the business goal for Site B was to generate as much revenue as possible from their pages.

This of course led to some internal corporate headbutting. The group that ran Site C wanted a direct link from Site A because it helped them meet their traffic goals. But the group that ran Site B felt that revenue and conversions were being shorted if the visitors went to Site C first.


To resolve this problem we decided to run a test and let the data speak for itself. The two groups agreed on running a two week test. The first week the camera banner on Site A linked directly to Site C (the educational site) and second week the banner linked directly to Site B (the purchase site).

Assumption: With this test, we wanted to see whether or not total revenue was significantly affected if visitors went through Site C first (A --> C --> B) instead of going directly to Site B (A --> B).

I first measured the revenue generated from the Site C "Buy Now" link to Site B (we tagged the "Buy Now" link with tracking code). This was a little tricky to do because I had to measure the percentage of traffic that only came from the camera banner (there were other ways in which visitors could reach that camera learning page). I factored in that percentage when gathering the revenue total from the "Buy Now" link.

Site C --> Site B

The second week, I measured the revenue generated from the banner on Site A to Site B (we tagged the camera banner with a different tracking code). Again, I had to measure the percentage of traffic that just came from the camera banner (there were other ways in which visitors could reach that camera purchase page).

Site A --> Site B


With my results in hand, I found that by sending visitors through Site C first (A --> C --> B), there was a drop in overall revenue compared with sending visitors directly to Site B (A --> B). However, this drop was not the significant drop off that the managers of Site B were expecting. The assumption that, sending visitors through site C was killing conversions and sales, was false.

Not only was there not a significant drop off in revenue, but I found that the AOV (Average Order Value) was actually higher (compared A-->B test) when visitors were sent through Site C (A--> C --> B) first. Visitors bought more or bought higher priced ticket items when sent through the educational site first.


In optimizing a web site, you're going to run into certain situations where assumptions may arise from fellow employees about what currently works and what doesn't. The great thing about web analytics is that allows you to test out those assumptions and find actionable solutions to help answer these assumptions. Make sure that you constantly test and make changes based on the data results....not assumptions.

In this case, upon reviewing my findings, the company decided that sending visitors through the educational site did not have a significant negative impact on sales and conversions. We also found that by sending visitors through the educational site that it actually helped to increase average order value.

Friday, November 03, 2006

Web Analytics & Privacy Concerns

I made this post on the Yahoo Web Analytics message board, but I wanted to post it here because I feel that it's important for anybody involved with web analytics to get an understanding of what's going on out there.

There are petitions being put out by consumer advocacy groups requesting that the FTC investigate the regulation of data collection. Depending on the decision that the FTC makes, there could be a large impact on the way web analytics is used today.

With that being said, I think the FTC is smart enough to see the value in web analytics and that it's not designed to infringe on a web site visitor's personal information. We all know that web analytics is designed for helping online businesses improve their site performance and marketing efforts.

Anyway, here is my post. Feel free to read the article and leave me any comments.

FTC Asked to Stop Microsoft's AdCenter

I'll bet $1,000 that these consumer advocacy groups, who petitioned the FTC, have members that go to their local grocery stores on a daily basis and use their Vons, Ralphs, Albertsons, Safeway, etc. club cards to save money on groceries. Yet, I don't see them petitioning against grocery chains.

I bet they've swiped those discount cards dozens of times without realizing that while those club cards are great for getting discounts on an overpriced box of cereal, they are really designed for tracking every item you buy.

Whoops....did I say that out loud (sorry grocery stores)?? I guess I'm all for grocery chains making my shopping experience better. I know....I know....I'm crazy! Do I really care if they know that I bought a gallon of milk and a six pack of Guinness beer??? No. In fact, I think it's great when I get two coupons in the mail a month later for discounts on what else....milk and Guinness beer.


I say...keep swiping those discount cards and let's keep getting our beer discounts!