Subscribe to DSC Newsletter

Predictive modeling is useless! Here's why.

If you're a modeler, you might say, “who the heck is this guy telling me that my precious thing is useless?” Wait a minute. I will explain later.

Let me tell you a story. Once upon a time, I was preaching in front of senior management on how we could get more money. Using predictive model, it was proven that business could get cleaner leads, the right customer with higher likelihood to take our product. On paper, we could increase the revenue. It turns out, the customer response went double and revenue went sky rocket. However, 3 months after the implementation, it went back to pre-model performance, even worse. What has happened, Is predictive modeling become useless?

We dig a little deeper to find out what went wrong. What really happened during the implementation? It turned out something we did not realize happened during the leads distribution. Somehow, the sales team leader were all gone.

The sales team has hundreds of sales troops which were led by team leaders/supervisors. These team leaders have the expertise and experience to distribute the leads to the right sales person.Should the “good” leads be given to a more senior sales person or junior/trainees?This methods were proven to provide good result most of the time. These team leaders have vast sales experience that gives them instinct on how to utilize the leads to the max. It worked beautifully until the perfect storm hit the sales room floor. These team leaders were suddenly away at the same time due to various reason for the whole month.

Suddenly, out of nowhere most of the team leaders were gone. The leads were distributed randomly to all the sales troops. Suddenly, the predictive model is useless.

This is what we call sales bias. If the model was implemented through different channel: direct mail, SMS or email campaign, there will be no bias and the predictive model will work perfectly. In our case, the model still has dependency on outside component: the sales person. The customer response will depend on how good the sales in pitching the product. What should we do? Can we replace “team leader experience” and distribute the leads “independently”?

Yes there is a way. We could actually provide logic to give “score” to each sales team member. The score will depend on sales’ past performance: how good they are in converting the sales. This score will then be matched to the leads’ score. The question now, should the high-score-leads matched to the high-score-salesperson? The best answer is to test it and make the mix and matched to give better result.

Do you have other challenges for your predictive analytics?


Views: 3370


You need to be a member of AnalyticBridge to add comments!

Join AnalyticBridge

Comment by Keith McCormick on June 7, 2015 at 7:03am

The outcome was unfortunate, but predictable. The managers of the team that is directly effected (the sales team in this case) must be present at the earliest stages of planning the predictive analytics project. They don't need to attend every project meeting, but they must be present at the first one. Potential deployment strategies must be discussed before modeling can begin otherwise this kind of outcome is assured. As @Thomas points out - ignoring the existing commission plan during project planning can also assure failure. It would be best to ask sales management as well as the sales team what assistance they desire. Did they even know that the modeling was taking place? Was this simply imposed on them from outside of their team? The sales team's desires and perspective are key factors in planning the modeling and planing the deployment. As this anecdote underscores, white board ROI does not exist - models must be deployed effectively to produce ROI. The outcome could be much worse than a single project failure - the earned lack of trust can undermine the entire predictive analytics program.

Comment by Thomas Lincoln on January 5, 2015 at 4:10pm

@wayne. . . poor title, I agree. . .and if you are debating the contents versus the title, i don't disagree.  However, i don't have enough time to critique people poor writing or sensational titles, but I do observe quite a bit of failures of management, which the article articulates very well, regardless of the title. .

Comment by Wayne G. Fischer, PhD on January 3, 2015 at 8:03am

@Thomas: The title of this blog implies that predictive analytics in some way is at fault..."useless."  When in reality the model did what it was meant to do, as the blog clearly relates.  All of your reasons in support of the blog's title are not due to the "uselessness" of predictive analytics, but rather the failures of leadership and management.  Quite a different thing.

If you're driving along a street and come up behind a car stopped at a red light, and don't apply the brakes and rear-end the other car, do you blame the brakes?  ...the car?  ...or the driver?

Comment by Thomas Lincoln on January 3, 2015 at 7:48am

@wayne, yes to point #1, but #2 you are projecting higher level sales mgmt skills, which in reality doesn't exist everywhere at your level of assumptions.  Direct selling, usually B2B, is way different than consumer sales, but has no where near the coverage of analytics than consumer sales, which lends itself to very different behaviors.  Direct sales hunters seldom take no for an answer, which means that they really don't listen to anything which isn't directly related to their commission plan or to mgmt organizational metrics to look good for the annual sales awards.  SAles mgmt or sales rep.  which don't always listen, which don't always believe in new  processes because they always see new ideas fail, customers not close, and sales results managed by exception.  I have seen very good predictive analytics be ignored not for good business reasons, but for organizational behavioral reasons, and personal behavioral biases.  I have seen willing customers told to go away due to mgmt constraints, which doesn't neatly fit into any business course theory, but for fear of getting on a corporate naughty list.  How does that fit into #1 or #2.  For most sales mgmt teams and members, behavioris all about personal motivational bias and greed, regardless what the analytics says about customer behavior, even to the point of legal and corporate increased liability! 


So nice in theory, but not always as clean cut and useful in reality, which is more the point of the article, and less the point about the actual analytics process, results or presentation.

Comment by Wayne G. Fischer, PhD on January 3, 2015 at 7:28am


1) The competent modeler will *always* communicate the assumptions upon which the model is built (e.g., process does not fundamentally change), and its limitations.

2) Even a sales mgmt team will recognize that something is fundamentally different if all team leads are gone at the same time.  [And I'm guessing the mgmt team would not let that happen in the first place.]

Comment by Thomas Lincoln on January 2, 2015 at 12:08pm

@Wayne, Yes, but every non modeling, non quantitative sales mgmt team doesn't realize that they have to remain more or less static if they want to use a customized predictive model.  Old direct-sell sales mgmt is all about the art of selling, not the science of selling.  I work at one :)

Comment by Wayne G. Fischer, PhD on August 31, 2013 at 11:57am

Ahhh, where to begin...?

1)  Sorry, you never did explain (in general) why "predictive modeling is useless."  One anecdote never constitutes proof.

2)  And, your story is a typical example of the *successful* use of predictive modeling, for two reasons:

a)  customer response significantly increased when the model's results were implemented; and

b)  when the process that was modeled changed (sales team leaders were gone), you knew from the difference between predicted and actual results that something fundamentally had changed and to investigate...which prompted further learning and better understanding.

3) Every competent modeler knows that if the process being modeled fundamentally changes, the current model will not predict the new conditions - since the model was developed using data only from the original process.

Comment by Lance Olson on August 28, 2013 at 8:37pm

Nice post Eka.

It always get really interesting to include the sales strategy and sales activities in a predictive model.

On Data Science Central

© 2019 is a subsidiary and dedicated channel of Data Science Central LLC   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service