How to Test Big and Win Big in Membership Marketing

One thing that disappoints me the most in the practice of membership marketing is the lack of testing that is conducted.  In our 2013 Membership Marketing Benchmarking Report only 16% of responding associations said that they conducted A/B split testing (a control versus a test segment) as part of their membership marketing efforts.  And that is down from 20% who said that they did so last year.

Besides under spending on marketing, lack of testing remains the single biggest missed opportunity to grow membership.

The three basics to test in marketing remain the list you market to, the offer you make, and message/package you send.

1.     Testing Lists – It is not unusual to see a huge variance between list response rates.  Right now I am looking at a results report where lists were tested.  The best performing list produced a 1.88% response rate.  The worst list – one tested as a sample – produced a .14% response rate.  The good news is that I know what each list achieved and can make decisions on my next campaign based on these results.

2.     Testing Offers – It is not unusual to see a variance of 100% or more in an offer test.  Offers can include how you package or the products and services you include in your membership.  Is it like a luxury car loaded with options or is it a basic model.  But more often the offers that you will be testing are the special incentives you use to encourage a prospective member to join.  These might include a free trial, a discount, a voucher for future purchases, or installment payments.

3.     Testing Messages/Packages – These tests can influence response by 50% or more.  In direct mail, a test package might include all new format, graphics, and copy.  In email efforts, it may be just the look and the copy.  Message testing can go to the core of your membership offer and test various presentations of your value proposition or it can simply present a new and different appearance that will help a prospective member take another look at your organization. A very simple test that many forget to do is trying different “From” lines and “Subject” lines in your email marketing to a portion of your list.  Then send the remainder of your file with whatever “From” line or “Subject” line had the best open and click through rates. 

In addition to the elements that should be tested, there are also important practices to keep in mind.

First, be sure to test big things.  I have relearned this lesson the hard way with a client of late.  We tested several individual pieces in an engagement and renewal program.  The problem was over the course of a year the member received hundreds of communications from the organization, so when I looked to see if our one “special” new member touch made a difference in renewal rates, I was disappointed to see that the extra “pebble” we gave them of free content at the start of their membership did not seem to hurt or help their ultimate decision to renew.  We should have tested a significantly different engagement or renewal program not simply one small element of an existing system. 

The same idea applies to membership recruitment.  Don’t test a postage stamp against a pre-printed indicia.  Test a full membership price against a substantial new member discount or a new list that will open up a new market for your organization.

Secondly, test with statistical accuracy.  There are times that our tests receive such small returns that the results are not really valid.  As a very general rule of thumb, in direct marketing recruitment, if you achieve a 1% response rate, you will want a minimum of 40 responses to both your control and test offers to achieve a statistically valid test.   Remember it is not the number of people who you promote to that matters, but the number of responses you receive. The smaller the variance you want in statistical accuracy or the lower your response rate, the larger the responding sample size you need.  If you want more information on testing statistics, here is a site with some nice tools you can use.  

Finally, testing something is better than testing nothing.  Any test that you construct is going to create additional work for someone in your organization and create push back.  So find something that can be tested quickly and easily.  Get some fast testing wins.  When you do this, testing and reading marketing results can become an exciting and fun part of your job and your organization’s culture.