A Data Science Central Community
I would like to know your opinion about this subject, as lately a lot on emphasis has been put on this issue.
In telecom industry, CSPs are interested to determine if their marketing campaigns are successful, if they have actually achieved their goals. Analysts in this field support the idea that the use of a control group when analyzing and evaluating a campaign is needed in order to give reliability to your analysis results.
But how do you select a control group for a certain campaign?
What are the characteristics that this group should have?
Looking forward for your comments/opinions,
I'm going to be fairly specific about this.
1) Set up the universe of the campaign, the selection criteria used to target customers
2) Identify the success criteria for the campaign
3) Make an educated guess as to the effect of the campaign; use this in power calculations to estimate the size of the control group.
4) It may be that you do not have enough customers to hold out an effective sample, or the needed sample may violate established policies. If so, make a decision about using a control group methodology.
5) Use a random number generator to identify x % of the campaign for the control group.
6) Make sure that the sample is representative.
6.1) If your campaign has multiple cells, make sure that the sample is pulled equally from all cells. You may want to pull the control group from each cell seperately.
6.2) Select a few critical dimensions, and then test the campaign group against the control group to make sure that there are no meaningful differences between the two.
The idea is to make sure that the campaign is the only difference between test and control, so resulting differences can be attirbuted to the campaign.
I have a similar doubt and I am new to this topic on how to do a post campaign Analysis.
Say I have some 100 retail store outlets which has one loyalty program for all these stores. The customers are segmented in 3 levels ('X', ‘Y’ and ‘Z’) . and I am launching a campaign for one of the (say 'ABC') retail store .
Let me know whether I am right in selecting the test and control group.
1) So now my intial aim is to form a test group who are ‘ABC’ retail customers who are active since last 1 year or so. Out of 3,00,000 (0.3 million customers) I had overall for all my 100 stores, I found around 50,000 customers were active for ‘ABC’ retail store since last 1 year. So I went ahead taking 50,000 as my target population.
2) Next my aim is to select the control group. For this I am blindly taking around 4000 customers as my control group from the 50,000 target population.
These leaves me around 46000 customers as test group population and 4000 as control group population.
1) Am I right in selecting test and control group population?
2) Is it compulsory that my test and control group should be of same proportion. I mean here do I need to split as 25000 each for test and control groups?
3) On what basis or variables do I need to select the control group? ( I have entire POS data of all my clients)
4) Do I need to check for normality test for control group? How to do?
5) How do I remove any outliers from the control group? How to remove?
6) Do I need to do some statistical analysis using t-test or Chi-square test for significance testing for both the test and control group, If needed on what parameter I need to check? Which test??
7) How do I calculate the incremental benefit from these two groups?
8) Finally what all the parameters I need to show in my Post-Campaign report? Right now I am showing total revenue, No of customers participated , revenue per customer.
What else I can show???
Please answer to these questions as early as possible. I am a newbie to this post-campaign analysis J
This will be of a great help to me!!!
So you've got your campaign target population selected (although 50k out of 3mm seems pretty narrow to me). The next step is: what measurable effect do you want the campaign to have on your customers? Once you've got that, that will tell you how to measure the campaign and also how big a population to have (although if you are starting off with 50k, I would not be surprised if the population so small that any test v control group will be noise).
I am not getting with What do you mean by "Measurable effect"?
I am looking at the metric total turnover generated by the campaign?
Please tell me how to calculate the incremental benefit? Here the 50K i mentioned is just sample target population (An example i gave), it can be of range from 50K to 150K as well.
If possible can you please answer my above posted questions ....thanks a lot for your time!!
Let me try helping you out. First of all, you need to clearly define what is the objective of this campaign. Some potential candidates are -
1) Campaign should increase the #transactions per customer at ABC store
2) Campaign should increase the spend per transaction per customer at ABC store
3) Campaign should increase the '#High value' transactions (Spend > $1000) per customer at ABC store
This is important because you need to design the test accordingly. E.g. if very few customers do 'High Value' transactions, you'll need to select a much larger test population
Let's run through a sample test -
Objective: Measure the effectiveness of a spend incentive campaign at Store ABC on customers who are 'Loyalists'
Test Population: Loyalists defined as -
1) Past 12 month spend > $1000
2) Past 12 month visits > 10
Effectiveness will be measured by #customers who did high value transactions (>$1000) anytime in next 1 month
Now you can use a Z-test calculator (http://visualwebsiteoptimizer.com/ab-split-significance-calculator/) to determine what the minimum Test and Control sizes should be based on hypothesized event rate (%Customers who did High value transaction).
You can go with a very small Control and very big Test if your client wants to minimize opportunity cost of not offering a good incentive to all customers. Conversely, you can go with a small Test and big Control, if your client wants to just test this 'expensive' campaign
Incremental benefit will be %High Value transactors in Test - %High Value transactors in Control
How this translates into actual dollar value is actually a business question