Control Group Marketing – A Campaign Best Practice

How much do your marketing campaigns actually affect customer behavior? It's an old question and an important one. It's often asked about branding campaigns but also comes up with ad campaigns, product campaigns, win-back campaigns and pretty much all types of promotional programs.

If your campaign isn't producing any effect, it's obviously a failure. If the campaign costs exceed attributed revenue from the campaign, it is less obviously a failure. I say less obviously because many companies don't compare revenue generated with the cost of the campaign.

With consistent measurement of the basic performance metrics you can eliminate unprofitable marketing campaigns and double down on the profitable ones to scale marketing sourced revenues.

Even without sophisticated marketing automation platforms or CRM analytics, there's a simple way to track the impact of your campaign. A control group will give you a measured validation of how well you're doing.

The basic idea of a control group is simple. Select a random (or nearly random) sample from your campaign's marketing list and exclude them from the distribution. Then measure and compare the control group's activity with the activity of the group targeted via the campaign. The difference between the control and campaign group gives you a pretty good notion of how effective – and profitable – the campaign is.

The theory is that a certain fraction of the customers in the campaign are going to purchase from you anyway during the campaign period. The control group lets you filter out that effect, as well as the effects of other channels which may be influencing behavior, such as display advertising, and shows you how much the campaign has affected customer behavior.

This is hardly a radical notion. Control groups are a standard best practice in all kinds of marketing analysis. In fact they are fundamental to statistical studies. However relatively few companies use them in their marketing. Marketing control groups become even more effective when combined with the customer analytics found in most marketing automation or customer relationship management systems.

With a CRM system and a control group you can also detect the halo effect of your campaign. These are purchases and other actions which are influenced by the campaign but don't come in through the normal campaign channels. An example is a customer who is so inspired by your campaign that he or she picks up the phone and orders your product directly instead of through the call to action channel.

Another example is the customer who doesn't use the promotional coupon you included in your campaign but who purchases the product anyway. You can assume that customers in the test group who respond in unconventional methods are still influenced by the campaign and so should be counted as part of the campaign effect.

Because CRM software lets you track all points of customer contact, and not just the direct response to the campaign, it can capture these halo customers.

Control group marketing is commonly combined with test marketing where only a sample of the marketing list or target audience is sent an offer to test the campaign. This allows you to determine the probable effects of the promotion without spending a lot of money.

A common number for the size of the control group is 10 percent of the size of the campaign or test group.

Ideally you would like your control group to be a truly random sample of your campaign list. In practice complete randomness is hard to achieve. Many companies select their control group by a simpler process, such as selecting every 10th name on the list to make up the control group.

If you don't draw a random sample it's important to avoid getting an inherently biased sample. For example if you use every 10th name on the list, be sure to go through the entire list and not stop partway through because you've reached your sample size. If your sample is less than 10 percent, switch to a larger interval between names, say every 20th name.

This is especially important if you are working with a list that's ordered other than alphabetically. If, for example, your list is by customer number, the lower numbers are likely to have been customers longer. If you stop partway through the list you end up with a sample weighted toward your long-time customers.

Of course control group marketing works best when your campaign medium is highly targeted, such as a direct mail, email distribution, nurture marketing or other digital marketing program. When you are using a mass medium promotion like display advertising or television it gets harder to get exact results but can still deliver measurement within confidence levels.

The usual way of handling the broader media is by either suspending the campaign or limiting its effect. For example, a restaurant might advertise specials that are only good on Tuesday, Thursday and Saturday and track the business on those days compared to Wednesday, Friday and Sunday (many restaurants are closed on Monday).

Or run the special for a month and compare it to the previous or following month – taking into account seasonal fluctuations. This is closer to conventional test marketing and it isn't as rigorous, but it will still give you a solid indication of the incremental effect of your marketing investment.

CRM tools ease the analysis of the results and the information it yields. Or this technology can take the results a step further and group control and non-control responses according to various user-defined categories, measures or dimensions, and then permit 'what if' modeling or otherwise seek out relationships or patterns between those segments to deliver customer insight and unique ideas for follow-on campaigns.

However, with or without CRM systems control or test groups can pay off big in designing future campaigns, reallocating the marketing budget and demonstrating proven financial impact to the executive team.