3 Things Marketing Experts Don't Tell You About Split Testing
Jim Stone

I've been split testing for a while now, and even use some pretty
advanced methods these days. It has worked out well for me, but
I still remember many early disappointments.

Many very smart marketers will tell you there's a lot of power in
A/B split testing.

They'll even tell you that you CAN get 1000% improvements by
changing a single word -- or even a single letter.

These marketers are right. But if you think they're telling you
the whole story, you're in for a rude awakening! I found out
there were a few things the experts hadn't told me.


Here are three of the most important things:

1. It's More Difficult To Set Up And Run A Simple Split Test
Than You Might Think:

In order to set up a good split test you must:

* upload a new sales page with just one change to it. You
might also have to create two different "thankyou" pages,
depending on how you set up your tracking.
* get a script that splits traffic between the two pages.
Ideally this traffic splitter will split the traffic
randomly, and not just in an "every other" fashion.
* make sure your script sets a cookie on the visitor's machine
and serves the same page to them the next time they come to
the site.
* put a tracking system in place that will track your visitors
and your sales for both pages separately.
* set up a database (perhaps).


And getting set up is just part of the hassle. It's also
difficult to run the test and interpret your results.

One time I set up a test, and started sending traffic to the
pages being split tested. Then I noticed a typo on ad B. So I
corrected it. Now, "strictly speaking" I shouldn't have changed
anything during the test, but I was probably OK doing this, even
though it's not technically good form.

That wasn't the problem. Here's the bad part: when I made the
change to my page, my editing software (without my knowing it)
changed the names of the fields in my opt-in form on page B, and,
as a result, it didn't work any more.

I didn't realize the mistake until a week later. During the week
I had marveled at how ad A had done so well, while ad B pretty
much just stood still.

That was one week wasted. My results were meaningless.

(I don't know about you, but this kind of thing happens to me all
the time. These are the stupid pesky gremlins that infest every
new thing I try. Why can't things ever go as planned?)

Other things can happen, too. Ad campaigns can stop. Hurricanes
can strike. Terrorists can strike. Christmas can strike. Your
page can get listed on the front page of Yahoo, and suddenly
you're getting a different kind of traffic -- and lots of it.
All of these things can create "noise" that affects how your ads
perform.

You also have to know how long to run your test before your
results are "statistically significant".

Truly, simple split testing can be something of a hassle.

But the hassle isn't really the worst part about split testing.
Here's something else the split testing "gurus" haven't told you:


2. You Can't Consistently Get Great Results On Your Own.

If you look at your landing page, and ask yourself, "How can I
improve this?" You will probably come up with something to test.
And then you will test it.

Good for you, right? You were Johnny on the spot. You got going
right away. You didn't sit around debating the merits of your
choice. You took action. What could be wrong with that?

The problem is you probably came up with some safe little change.
Some timid little tweak. And it probably didn't help much, if
any at all (and it might have hurt). It probably wasn't a golden
nugget that significantly boosted your response. It might have
been, but it probably wasn't.

The truth is, to get best results, you need to get others to help
you brainstorm for testing ideas.

But the experts haven't told you that.

And there's one more thing they haven't told you:


3. With Simple Split Tests, You Should Be Prepared For Lengthy
Strings Of Disappointing Tests -- Even When You Generate Great
Ideas.

Did you know that most of the changes you make to your landing
page will not help?

According to Charles Holland of Qualpro (Breakthrough Business
Results With MVT), if you run 25 split tests, about 13 won't show
any significant difference. And 6 of the proposed changes will
have a negative effect. That leaves about 6 to have a
significant positive effect. That's 6 improvements out of 25
tests.

But it gets worse. Did you know it takes about 100 ideas to get
25 elements that you can easily split test? That's because 75%
of the ideas you will generate for improving your ad will be
things that are too expensive or time consuming to implement, or
can't be combined with other things you want to include on your
page.

That means you must generate 100 testing ideas to get 6 that will
actually make a positive difference on your page.

Please, re-read that sentence, and let it sink in this time. It's
that important.

If you have ever been discouraged by a string of 5 or 6 A/B split
tests that made no difference, all in a row, you know in your
heart that these numbers are very close to the mark. You might
have felt alone -- just an unlucky sap for whom nothing ever goes
right -- but you were actually just experiencing normal, run-of-
the-mill split testing.

The experts didn't tell you that, either.

They didn't tell you how much of a hassle it can be, they didn't
tell you what you need to do to generate good ideas, and they
didn't tell you how long it can take to get meaningful results.


So How Do You Deal With These Problems?

1. The hassle. You're just going to have to bite the bullet.
Split testing is one of the best ways to improve your profits.
Get some good software, and some good information about running
and interpreting your tests, and just dig in. This doesn't
exactly made it any easier, but at least you've been warned.

2. The help. Imagine what kind of ideas you would get if you
assembled 20 people in a room. Other marketers, potential
customers, existing customers, maybe your mother. And you ask
them the question "How can I improve this sales page?" And you
let them throw ideas out, and let other people's ideas inspire
new ideas, and you do this until you have 100 ideas -- ideas that
come from many perspectives, from people with nothing to lose,
and from people who are thinking more and more "out of the box"
as the session goes on.

If you could do THIS, then you would have a good chance of
finding some truly good testing ideas.

Such a big brainstorming session is impractical for most people.
That's true. But it can't hurt to be told what the ideal is. At
least now you'll be thinking of ways you can approach this ideal.

For instance, you might consider checking out what your
competitors are doing. You might pay better attention to the
marketing books you've read. You might solicit ideas from
discussion forums.

And maybe, just maybe, you will go to the trouble of actually
getting a few friends and relatives to sit around with you and
brainstorm a bit.

3. The discouragement. First, it helps just to be told what to
expect. Winning ideas are rare, but they usually deliver big
results when they come. So the hunt is worthwhile, but it can be
discouraging.

But there's actually something you can do to change the situation
in your favor.

There's a new method of split testing called "accelerated split
testing", or "Taguchi testing", or "Multi-variate testing". It
allows you to take all your ideas -- the good ones and the bad
ones -- and test them all at once. Your chances of hitting a
winner (or two or three) somewhere in that jumble is actually
very, very good. If you set your tests up right, you should get
significant improvements from almost every single test.

So there you go. Three problems the experts haven't told you
about, and three ways to deal with the problems.

Now you've been told.



---------------------------------------------------------------------
Jim Stone is an accelerated split testing expert. He has
developed an accelerated split testing software product, and
writes regularly on the subject. You may receive his free course:
"10 Steps To More Sales With Accelerated Split Testing" at:
http://www.splittestaccelerator.com