Reading Test Results. Or, How to Make Sure Your Test is Statistically Valid

Published in Target Marketing

 

At what point can you believe test results?

 

If your minimum allowable response to meet your profit objective is 15 orders per thousand (1.5%) and your test mailing of 5,000 pieces brings in 87 orders (1.75%), are you in clover?

 

The answer is emphatically no!

 

The reason: Retests and rollouts virtually never pull precisely the same as tests. Statistical probability dictates that future results will fall either on the plus side (or on the minus side) of past test results. What’s more, as a matter of experience, it’s rare that a reconfirming test or rollout will do better than the test. The possible reasons:

1. You ordered a straight nth name select, but the list owner sent you hotline buyers giving you artificially high results.
   
2. Seasonality. You tested in January, read the results in March, mailed again in May.
   
3. Your test offered a brand new product to a brand new market; now competition is creeping up around you. For that reason, a basic rule of thumb is: Your continuation should never exceed 10 times the test. Test 5,000; go back to that well for no more than 50,000.

 

The exception: If the results are simply phenomenal – double or better than what you expected … plus you know the list… you know the list owner to be honest… and underneath it all, you feel really confident about the entire proposition. In that case – where you have real confidence – turn around and jump on the opportunity immediately.

 

Why immediately? Chances are results of the continuation will decline in direct proportion to the amount of time spent diddling around. All of which is easier said than done, since most direct marketing budgets are set in stone and few companies have this kind of mobility.

 

Confidence Rates

Think of any test as buying information. The more you spend, the more confidence you should have in the information you are acquiring.

For example, you mailed 5,000 of test package “A,” and 5,000 of test package “B”. “A” pulled a 1.75% response, and “B” pulled a 2.50% response.

 

“B” performed better, right?

 

Of course it did. But if you mailed those two test packages again, would “B” perform better again? If you want to be confident that 95 out of 100 times “B” would perform better, the likelihood of that happening again is questionable.

 

Based on statistical models, 95 out of 100 times “B” will pull between 2.07% and 2.93%, and “A” will pull between 1.39% and 2.11%. The fact that “B” could perform as low as 2.07%, and “A” as high as 2.11% suggests there could be mailings where “A” would pull higher than “B.”

 

Visualize Response on a Bell Curve

If you seek a 95% level of confidence that “B” will pull higher than “A,” you’ll be disappointed because there is an overlap in projected future response ranges between the two.

 

It’s helpful to visualize the range of response on a bell curve to understand why you can’t be confident 95 out of 100 times that “B” will pull higher than “A.”

This graphic illustrates the distribution of future response 95% of the time at the “A” test cell response rate of 1.75%. In this example, the normal distribution of response in future mailings will be that 47.5% of the time response will fall between 1.39% and 1.75%. Another 47.5% of the time, response will fall between 1.75% and 2.11%. Add those two 47.5% ranges and now you have 95% of future response between 1.39 and 2.11.

 

Because it is possible 5% of the time future response will be both above and below that bell curve distribution, 2.5% of the time the response could be below 1.39% and 2.5% of the time it could be above 2.11%.

 

Why demand 95% confidence?

Does operating at a 90% confidence instead of 95% put your business in jeopardy or at other significant financial risk? If 90% is okay, is 85% acceptable? Or 75% (three out of four times–and you mail four times a year)? If you are mailing in high quantities, with considerable sums of marketing investment money on the line, you should want a high confidence factor. But the confidence level decision rests with you.

 

The Formula to Arrive at These Ranges

You don’t have to be a statistical genius to arrive at these response ranges. An understanding of statistics and the relationships between the numbers is advisable, but otherwise, you can calculate these numbers on your calculator (assuming your calculator calculates square root at the push of a button). The formula to arrive at these projected future response ranges is simply this:

1. Take 100 minus your response %.
2. Take that number times your response %.
3. Divide that number by your mail quantity.
4. Take the square root of that number. What does square root do? It squares the number down to a multiplier of itself. The square root of a number backs into the multiplier of itself. For example, the square root of 9 is 3 (3 times 3 equals 9), or the square root of 81 is 9 (9 times 9 equals 81).
5. If you desire a 95% level of confidence, multiply that number by the confidence interval of 1.96.
6. To arrive at your low response range, take your actual response and deduct the number you calculated from the previous step.
   

To arrive at your high response range, take your actual response and add the number you calculated from point 5.

 

Here are how the formulas look using Examples “A” and “B,” mentioned earlier in this article:

To arrive at the 90% level of confidence, instead of using 1.96 to determine the confidence interval, use 1.65. Here are other factors:

These fixed numbers are based on another formula that permits the calculation of confidence levels.

 

The Predictability of Future Results

The foundation and hallmark of direct marketing is using numbers to measure and predict results. If you’re an established direct marketer, you already have benchmarks you can use to base your test decisions and know if your test package has beaten your control.

 

You also know that to be competitive and keep up with our changing times you must keep testing fresh packages, whether it’s a new offer, new creative, or new lists. And when you’ve found the new offer, creative or list that performed well in the test, you want to have an indication of the likely range of response you will have in the future, especially if you are rolling out from 5,000 to 50,000 names, investing up to 10 times in the rollout what you spent for the test.

 

If you’re testing a new product, or are new to direct marketing, it’s vital that you identify the winning combination of offers, creative and lists in a test mode before rolling out. It’s often desirable to test a myriad of lists, different creative approaches, and offers to identify the magic combination of success, but it is critical that the test matrix be set up so you don’t compromise the integrity of statistically predicting your results. If your budget constraints only allow you to test a quantity of 25,000, and you want to test five lists, two creative approaches, and two different offers, chances are you won’t be able to construct a combination that will yield a valid test.

Use This Number	For This Confidence Interval
.67	50%
1.15	75%
1.28	80%
1.44	85%
1.65	90%
1.96	95%
2.58	99%

 

So what’s the most important element to test if your budget is limited to a quantity of 25,000? Lists, creative, or offers? Here are two options:

• Mail five lists (5,000 each), but use only one creative package and one offer. Finding new customers is crucial, and testing lists with significant quantities for continuation potential is key. When considering rental lists, consider the total size of the list for rollout or continuation potential. In this scenario, this means you must make the best offer you can, and use the most powerful selling creative package you can create.
  
• Mail 5,000 to three lists. Using a fourth list you either know from experience, or whose buyer profile most closely matches your likely buyer, mail 10,000 to it, splitting it into two groups of 5,000 each on an nth name basis. Use that 10,000 quantity split (5,000 each) to test either two offers or two creative packages. If within a package you test different creative and use a different offer, your test will be meaningless. Only one element should vary from one test package to the next.

 

Since many direct marketers believe the success of your effort is influenced 40% by the lists you use, 40% by the offers you make, and 20% by your creative, testing two different offers may be the most prudent choice since it affords you to test an element with more potential impact on your bottom line.

 

What You Must Remember

You must include in your test enough prospects to obtain reliable results. To find out what’s right for you, run assumptions at varying quantities and response levels. The more you mail, the tighter the spread will be for your future response ranges. A quantity of 5,000 is usually considered the smallest number you should mail, but sometimes the quantity needs to be higher, depending on the response you expect.

 

While these numbers give you the ranges you’ll likely generate in the future, the results next time will be different if you change the offer, products, or creative. The results may also change due to things out of your control such as the economy, time of year you mail, or changes in the competitive market.

 

Be aware that using these techniques will help you make a more informed decision on determining which test pulled better and at what level of confidence. The numbers will be just one more piece of the puzzle you can use to base your investment decisions in future marketing programs.