In this post, I've tried to capture some of the common aspects of working in the analytics industry. While we occasionally hear about India growing fast into this space, there are a lot of things happening in India that might transform this field further. While some of these aspects are specific to what I've observed in India, a lot of them are generic.
As in previous posts, I try to classify these aspects under different heads:
- The "cost" reduction trap - IT & General Technology has traditionally had an affinity for cost reduction through repeatable frameworks, off-shoring, removing redundancy etc. As a few people understand that analytics has more in common with the Consulting Industry, a lot of these projects bear the brunt of cost management and deliver poor results.
- Hiring Einstein to build a boat- This is pretty common today where companies try to club all skills and hire an expert to solve their problem which could be trivial given the kind of expert they are trying to bring in. An easier way could be to internally try understanding the objectives and then move on to defining the skills and hiring people who can do this. (E.g.: Rather than saying we need nlp, random forests, classification, SVD, hadoop etc... it would be overall easier if the problem was something like this - "We need to mine our email & ERP data to come up with useful insights to improve sales performance.")
- 'Let's make someone experienced like Beckham the captain of our Basketball team' - This is often a result of the hierarchical org structures designed to manage costs and timelines. While the most experienced guys might be excellent in managing repeatable projects, they often fail to understand the intricacies of analytics projects. As a result, the team gets bashed up to deliver some non-value-adding goal. In line with the basketball analogy, the team is under pressure to score at least one goal while they are experts at scoring baskets and the customer needs baskets rather than goals.
- It is of course technical, but where is the "Passion"? - This point is probably a bit more Indian. Everyone is an expert and every complex task is trivial in India. The challenge is that they are very passionate about techniques like SVD, Lasso, adaboost and so on. However, I don't seen enough people as passionate about things like - 70% forecast accuracy, 25% operational cost saving and the more "business" problems. This aspect is conveniently passed on to a person close to the client site often with poor understanding of building hypotheses. Or sometimes worse, it gets passed on to the experienced guy with experience in other fields.
- Focusing on getting it to work quickly sometimes kills the "amazing" insights that could help in the long run - This is a common challenge when the IT team owns an analytics project. They are so used to seeing results quickly that they sometimes challenge the smooth functioning of the project itself. A good practice here is to add in a visualization tool like Tableau and keep sending across beautiful looking graphs even though they might be of little use to solve the larger problem at hand. This ensures that a part of the team keeps it's focus on the key goal of the project.
Looking at the way things are going, I believe the future of this field lies on the shoulders of the Outliers. Fresh ideas would more likely come from people with little experience in this field, who come with a very different perspective of looking at problems. We do have enough veterans to build upon these ideas and validate them anyway.