A Data Science Central Community
Is data mining more about fitting data well? - Exercise Results
Today, I am going to share results of an exercise that I carried out recently for a start-up. Intention of the study was to extract those major attributes that are generally driving less/in experienced (or) re-skilled data miners towards the given objective and to understand where they are failing back. Herein, twist is majority of them have given same conclusion or explanation for the given objective. Results highlight or comment on, those important aspects of the practice where most of them failed to cognize for the sake of quick answer/solution.
All members of the sample had experience both with R and data mining solutions; either through course projects (free/paid/part-of-curriculum) or through industry experience, however, industry experienced sample have been limited between minimum of 1 year to maximum of 3 years from whatever domain. Details of sample are as below:
a) 17 - Fresher’s from various engineering background (both Graduates and Post-Graduates)
b) 12 - Fresher’s from various quantitative background (Maths, Stats, MBAs, Econometrics, etc.)
c) 18 - Experienced from different industry background (data management related, programming, consulting, etc.)
d) All members of the sample belong to two major cities of India.
About Test Data:
Bank data of customers belonging to a particular city branch having around 17000 observations for a period of one month, which as information about customer’s age, few demographics, no of transactions they did in that month, whether they visited branch in that month, etc., total of 12 variables.
Computing machine with a pre-installed latest R (3.1.1) & RStudio that has 8GB RAM and Intel Core i7 Processor.
“Comment about the variables ‘visiting branch’ and ‘age’ relationship”.
A time limit of 20 minutes was given, which was almost two and half times more than average time of experienced people, took to give their comments.
Highlights from the Exercise:
What was wrong in the data?
When this data was originally received, I observed that due to a machine/man-made mistake, column ‘age of the customer’ in the data was having representation of an additive nature, for instance, if customer has visited the branch twice in the month and his original age is 25, it appeared as 50. Hence, positive relationship as age increased, however, it was not the case after the noise removal.
Data Mining is a process of many stages as depicted in CRISP-DM1 and data understanding is key of them, I always suggest process your data incrementally, if you want efficient analytical solution, ignoring it, and employing which fits the data well practice, may not work in all situations.
Author thank management of start-up for allowing to publish exercise highlights. He undertook several programs towards analytical talent development, views expressed here are from his industry experience. He can be reached at [email protected] for more details.