Marketers who aren’t A/B testing will be left behind

I’m passionate about testing. So much so, that I wrote a guest post for SocialFresh.com with a case study from our testing at Phonebooth.com. An excerpt is below, but check out the full post, “You are not marketing unless you are testing.”

An excerpt from SocialFresh.com

testingAhhhh, metrics and analytics. For those of us that aren’t deeply involved in conversion (cough cough making money), we resort to labels for the “numbers people.” Some call them analysts, technical folks or SEO / analytics gurus and tend to think that this isn’t our job as marketers.

Friends, you are mistaken. Marketing without data is like diving into a pool with no water – it ain’t pretty. Gone are the days of creating messaging with a finger in the wind or creating ideas in your fancy ivory tower.

Good marketers cater to potential customers by learning. What works? What do they want to hear? What do they do with your product? Where do they go on your site? When do they buy? How do they buy? Where do they buy?

If you can’t answer a few of those questions, do a Google search for your local unemployment office and save that in your GPS.

An A/B testing case study

At Phonebooth, we launched a new website late last month after months of research, analysis, content creation, planning, stakeholder meetings and other things you do to try to make sure everyone is on board. One major element of our redesign was to become masters of testing. We hadn’t entirely neglected it in the past, but decided to make testing a focus.

Our first test was the main hero image (an elegant title for the stuff below the top nav / header). We had research telling us that customers buy our business phone solution because of value. Saving money makes people happy and we had data to confirm that.

We established five different messages with similar styles, colors, fonts, etc. The main variance was the actual copy.

  • Control: $20 per user / month
  • Variation B: Save up to 60%
  • Variation C: A usability version better explaining our product with a few key bullet points
  • Variation D: Save up to $2500 per year
  • Variation E: Free Your Voice – a creative concept

Visual Website Optimizer was our tool of choice and we built out the different variations. Each website visitor was randomly assigned one of the variations and that became their homepage thanks to a cookie. Every visit to the homepage would give each user their “unique” homepage and we had several goals to track.

Testing is nothing without goals

Metrics are great. Measuring the correct metrics is even better. It is critical to have a thorough understanding of what you are measuring and what it truly means. We created three distinct goals to track for this test.

  1. Engagement – this is defined as clicking anything on the website. What this means is clearly debatable and it may be completely irrelevant, but I wanted to see how many folks clicked something on the page in a digestable way.
  2. Enters the Store – we sell our business phone solution online. Tracking the funnel is extremely important and we wanted to see how many folks clicked into the store.
  3. Buys Phonebooth – does this need an explanation? This is the most important metric for us. While entering the store and engagement are both important… frankly my dear, I don’t give a damn. I want money. It helps to keep a roof over my head

Pretty simple, huh? Now, we’re ready to see what happens. From here, we start our test early on a Monday morning – a clean week.

Have some fun and test for yourself

Before you run a test, I highly encourage you to make an assumption. Guess what you think will perform the best and why. Heck, create a contest amongst your team and have everyone pick their favorite. Give a prize to the winners!

Leveling the playing field for testing will help everyone understand that a website isn’t a symphony of dorks and geeks coding and designing away to make something that they think looks good. It is science. Well, if done correctly that is.

While my opinion is arguably worth two cents, I thought that our control would perform the best. $20 is a really low number. It is inexpensive for a business phone solution. Plus, I had stared at this version for two months. It was stuck in my head, but I thought that folks would gravitate to a killer price and buy our product.

Click to view the Results!


Thanks for reading this post.

Follow me on the rest of the webs.

Still not sure who wrote this post? I'm Chris Moody.

Join the conversation!