Subscribe to DSC Newsletter

The screen shot below says it all: candidates for data science positions are plentiful, considerably more abundant than job openings: more than 600 LinkedIn users applied for this data science job. Even less prestigious companies routinely attract more than 50 applicants per job opening.

In view of this, you would thing that getting an expensive analytic degree is a waste of money (except a degree from MIT, Stanford, CMU, Northwestern and a few other select schools), and applying for a data science position is a waste of time (you would be competing with top candidates who apply to all the advertised positions). The problem is actually a bit more complicated. The root causes and solutions are as follows:

  • Companies want candidates with very deep rather than broad expertize, are not willing to accept telecommuting, and will not train a new employee, and in some cases only hire Ivy league candidates. In doing so, they drastically restrict the pool of of potential employees
  • Many university curricula are outdated, so despite the volume of applicants, hiring managers complain that very few have the right skill set. Few candidates are willing to acquire these new skills (e.g. Mapreduce), although it can be learned at no cost if you have an Internet connection and a browser.
  • Data Scientists are not properly used or hired. In the case of Facebook for instance, one might ask how - despite all the great scientists and great data that they collect about users - they generate so little revenue per page impression. They should generate 10 times more revenue if data science was fully and properly leveraged by top management, read comments here for details. In this case, the issue is probably poor communication between top management and data scientists: a solution to significantly increase ad revenue by optimizing ad relevancy is described in my free eBook, so there is no excuse for poor performance. The same can be said about many companies (Google under-utilizing its Internet real estate, Microsoft having very poor marketing campaigns and not hiring the right people) and many problems such as spam detection or fraud detection. 
  • There are many alternate options outside traditional employment for data scientists, and as a data scientist, you should consider these new options.

Finally, an unexpected consequence is a rise in sophisticated fraud, as an oversupply of unemployed math PhD's with great expertize end up working for rogue organizations (their only choice), while government and other organizations fail to hire the best people, for whatever reasons.

Views: 11194


You need to be a member of AnalyticBridge to add comments!

Join AnalyticBridge

Comment by Edmund Freeman on June 11, 2012 at 1:42pm

That's kind of a weird ad, actually. The skills & experience is purely about programming. There's nothing in there about analytic experience. Pure programming types can take some very unusual (in a bad sense) views of analytic issues.

Comment by Randy Bartlett on June 11, 2012 at 1:38pm

First, I take exception with the ubiquitous and unfounded idea that expensive college degrees are somehow worth more--the tuition bubble.  In the case of statistical training, I am observing the opposite.  I think other factors, which might offend some readers, are important.  The above marketing promotion is rammed down our throats all too often--call the dogs off.  Second, shortage/no shortage, what I am certain about is that the market is inefficient--as you pointed out.  Many hiring managers of analytics talent are unable to judge the necessary skills--one contributing cause is that they have not performed in that job.  Certification might help, yet this will take time.  Also, when statisticians are identified and hired, they are not always leveraged very well--as you point out.  Third, when my company places a job ad, we receive many resumes from people with EITHER the soft skills OR the Quant skills.  How should we more rigorously measure shortage/no shortage than counting applicants? 

Comment by David M. Fishman on June 11, 2012 at 12:56pm

Theo's comments are pretty much spot on; to this, I would add that educating hiring managers is vital. In my profession (marketing, where I differentiate with my analytic skills), I see a similar fundamental misunderstanding of what the job is about. As a result, it's no surprise that qualified candidates go overlooked owing to Type I and Type II error. 

Comment by Charlie Greenbacker on June 11, 2012 at 12:47pm

Thanks, Quyen. This is the point I was trying to make.

Comment by QUYEN KIET on June 11, 2012 at 12:45pm

The key word is "talent". Think 2003 when there was an overabundance of website developers after the internet bubble burst. Despite the large volume, the reality was that most of those "developers" had just enough skills to build crappy websites. Sadly, because of the large numbers (practically a commodity service), prices for web developers plummeted even though the truly talented remained a small group. So of the 600 applicants, how many of those applicants represent true "talent" qualified to do a "A" level job? "B" level? "C" level? 

No differently, try posting a job for a C++, Ruby or Java programmer, and you're going to get many responses, the volume of which dispute the prevailing belief in Silicon Valley that there is a shortage of back end programmers. 

Comment by Rebecca T Barber on June 11, 2012 at 12:42pm

I would bet that a good number of those 600+ applications had, at best, a subset of the skills.  Heck, I have a job like that already and have only a subset of those skills.  

Also, the places that CAN make good use of a data scientist tend to realize that the technical skills are just an exclusion criteria; what gets the job is those things combined with strong business acumen, industry knowledge and communication skills.  

Over time, I fear the field will evolve similarly to what has happened in IT; lots of technicians to do the coding working for managers who have the business/industry knowledge and speak both languages, who can bridge the gap to the executives who sponsor the projects.

Comment by Theodore Omtzigt on June 11, 2012 at 12:39pm

There clearly is a big difference by companies and industries that use analytics on the front-end of the business process versus the back-end.

The front-end is intimately connected to the sales and marketing  processes and for web-based businesses intersect with the actual product or service that the company makes. That is a more entrepreneurial activity at this time and thus more related to very tightly integrated teams. IMHO, this explains the heavy reliance on having degrees from the universities that are heavily represented in the start-up world.

The back-end analytics is centered around operational metrics, and the key here is risk management: don't tinker with a revenue stream that is paying the bills, and thus the HR department and the project managers chartered to managed these processes will be very narrowly focused.

The solution to this conundrum is to not focus on the 'job' but to create value with your skill. Build an analytic web service in the OpenData universe and demonstrate your wares. Have the hiring managers come to you based on the skills you are demonstrating.

Comment by Vincent Granville on June 11, 2012 at 12:29pm

Hi Charlie: if you are recruiting for jobs that require clearance and US citizenship, you probably have much fewer qualified applicants, and recruiting data scientists is likely to be challenging.

Comment by Charlie Greenbacker on June 11, 2012 at 12:23pm

Lots of applicants != lots of talent

Comment by Robert Rouse on June 11, 2012 at 6:16am

In my industry (utilities), your point that "Data Scientists are not properly used or hired" rings true the loudest.  The electric market is such a complex machine that we have come to rely on overly simplistic models to predict or optimize revenue.  We've been stuck in that paradigm a long time.  I've done my best to present some of the opportunities that can be found using advanced analytics, but it falls on deaf ears.  All they hear is that it's too costly to to engage in that kind of activity.  These days, it's costly NOT to!

On Data Science Central

© 2019 is a subsidiary and dedicated channel of Data Science Central LLC   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service