I originally published this article in Analytics-Magazine.org.  The article relates to my book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie...

Every day’s a struggle. I’ve faced some tough challenges such as which surgery to get, how to invest for my business and even how to deal with identify theft. With so much stuff coming at me from all angles, daily prosperity relies on spam filters, Internet search engines, and personalized music and movie recommendations. My mailbox wonders why companies still don’t know me well enough to send less junk mail.

 

These predicaments matter. They can make or break your day, year or life. But what do they all have in common?

 

These challenges – and many others like them – are best addressed with prediction. Will the patient’s outcome from surgery be positive? Will the credit applicant turn out to be a fraudster? Will the investment fail? Will the customer respond if mailed a brochure?

 

There’s another angle. Beyond benefiting you and I as individuals, prediction bestows power upon an organization: Big business secures a competitive stronghold by predicting the future destiny and value of individual assets.

 

For example, in the mid-1990s, Chase Bank witnessed a windfall predicting mortgage outcome. By driving millions of transactional decisions with predictions about the future payment behavior of homeowners, Chase bolstered mortgage portfolio management, curtailing risk and boosting profit.

Introducing ... the Clairvoyant Computer.

 

Making such predictions poses a tough challenge. Each prediction depends on multiple factors: the various characteristics known about each patient, each homeowner and each e-mail that may be spam. How shall we attack the intricate problem of putting all these pieces together for each prediction?

The solution is machine learning; computers automatically discovering patterns and developing new knowledge by furiously feeding on modern society’s greatest and most potent unnatural resource: data.

Figure 1: The learning process.

Data can seem like such dry, uninteresting stuff. It’s a vast, endless regiment of recorded facts and figures. It’s the unsalted, flavorless residue deposited en masse as businesses churn away.

But the truth is that today’s big data embodies a priceless collection of experience from which to learn. Every medical procedure, credit application, Facebook post, movie recommendation, fraudulent act, spammy e-mail and purchase of any kind is encoded as data and warehoused. This veritable Big Bang delivers a plethora of examples so great in number only a computer could manage to learn from them.

This learning process discovers and builds on insightful gems such as:

• Early retirement decreases your life expectancy
• Online daters more consistently rated as attractive receive less interest
• Vegetarians miss fewer flights
• Local crime increases after public sporting events

 

Machine learning develops predictive capabilities with a form of number-crunching, a trial-and-error learning process that builds upon statistics and computer science. In commercial, industrial and government applications – in the real-world usage of machine learning to predict – it’s known as:

Predictive analytics — Technology that learns from experience (data) to predict the future behavior of individuals in order to drive better decisions.

 

Applied Prediction

“The powerhouse organizations of the Internet era, which include Google and Amazon ... have business models that hinge on predictive models based on machine learning.”

– Professor Vasant Dhar, Stern School of Business, NYU

 

Every important thing a person does is valuable to predict, including: consume, work, love, procreate, vote, mess up, commit a crime and die. Here are some examples:

 

• Prediction drives the coupons you get at the grocery cash register. U.K. grocery giant Tesco predicts which discounts will be redeemed in order to target more than 100 million personalized coupons annually at cash registers across 13 countries. This increased coupon redemption rates by a factor of 3.6 over previous methods.

• Predicting mouse clicks pays off massively: websites predict which ad you’ll click in order to instantly choose which ad to show, driving millions in new-found revenue.

• Netflix awarded $1 million to a team of scientists who best improved their recommendation system’s ability to predict which movies you will like.

• Obama was re-elected in 2012 with the help of voter prediction. The campaign predicted which voters would be positively persuaded by campaign contact, and which would be inadvertently influenced to vote adversely. Acting on these predictions was shown to successfully convince more voters to choose Obama than traditional campaign targeting.

• The leading career-focused social network, LinkedIn, predicts your job skills.

• Online dating leaders Match.com, OkCupid and eHarmony predict which prospect on your screen would be the best bet at your side.

• Target predicts customer pregnancy in order to market relevant products accordingly. Nothing foretells consumer need like predicting the birth of a new consumer.

• Student essay grade prediction has been developed for automatic grading. The system grades as accurately as human graders.

• Wireless carriers predict how likely it is you will cancel and defect to a competitor – possibly before you have even conceived a plan to do so – based on factors such as dropped calls, your phone usage, billing information and whether your contacts have already defected.

• Wikipedia predicts which of its editors, who work for free to keep this priceless asset alive, are going to discontinue their valuable service.

• Allstate Insurance tripled the accuracy of predicting bodily injury liability from car crashes based on the characteristics of the insured vehicle. This could be worth an estimated $40 million annually to the company.

• At Stanford University, a machine learned to diagnose breast cancer better than human doctors by discovering an innovative method that considers a greater number of factors in a tissue sample.

• Researchers predict your risk of death in surgery based on aspects of you and your condition in order to inform medical decisions.

• Crime-predicting computers help decide who belongs in prison. To assist with parole and sentencing decisions, officials in states such as Oregon and Pennsylvania consult prognostic machines that assess the risk a convict will offend again.

 

Organizations of all kinds benefit by applying predictive analytics, since there’s ample room for operational improvement; organizations are intrinsically inefficient and wasteful on a grand scale. Marketing casts a wide net; “junk mail” is marketing money wasted and trees felled to print unread brochures. An estimated 80 percent of all e-mail is spam. Risky debtors are given too much credit. Applications for government benefits are backlogged and delayed.

 

With predictive analytics, millions of decisions a day determine whom to call, mail, approve, test, diagnose, warn, investigate, incarcerate, set up on a date and medicate. By answering this mountain of smaller questions, predictive analytics combats financial risk, fortifies healthcare, conquers spam, toughens crime-fighting, boosts sales and may in fact answer the biggest question of all: How can we improve the effectiveness of all these massive functions across business, government, healthcare, non-profit and law enforcement work?