A Data Science Central Community
The author of the following discussion:
asks this quesiton: "So how can we become data-savvy, but not data-obsessed?"
I think this is (extremely) worth considering............ especially from my frame or research, where I find that "small datasets", maybe hidden wihtin big data, are what are really important .................
The comments on the article are better than the article....
The trails and tribulations of analytics have been documented a million times, big data simply adds a new and highly seductive way to hang yourself. Intuition, however, is definitely NOT the answer, as has been documented a billion times over. Bias, ignorance, and stubbornness are just too human.
Anyone that has stared a big data set in the eye, and has tried to make sense of the wisdom contained within, knows how difficult it is to discover and articulate a quality answer. Big data still has lots of opportunities, if not more, of partial information dark alleys that can yield a self-consistent answer that is gloriously wrong. The article conjectures that 'intuition' can avoid such failures, and I think that has been decisively disproved. Examples are abound in economics and financials.
I like the team approach with a decidedly Bayesian sensibility, in the sense that you always think of your current understanding as the best you know at this time, but that future information can change that. Then the litmus test to say whether or not you are data-obsessed or not is to understand how your current truths can change and associate probabilities to these possibilities. If you are refining aspects that don't change the Bayesian, you are polishing the proverbial turd.
If I understand Dr. Miner's point, it is that staying focused and hype-free (i.e., not obsessive about MORE and MORE data as an end unto itself) might allow an analyst to discover the patterns and values of small data sets within larger big data collections. This certainly makes sense.
And I agree with Theodore and Tim, who seem to espouse that a mix of intuition and data--or, perhaps more accurately, "data-fed intuition"--would be preferable to sole reliance on one or the other. However, I am unclear whether the term "intuition" has inadvertently been confused with "human decision-making" in general. Whether "intuition" is involved at all within a given decision-making process, nonetheless data should be a tool, not the final arbiter.
And then there's the POV espoused in a recent HBR post, whose author suggests an inverse relationship between data and judgment, where importance of human judgment should decrease as the amount of data goes up: http://blogs.hbr.org/2013/12/big-datas-biggest-challenge-convincing.... Taken to its logical conclusion, how would the data-intuition mix look then? Does assigning greater weight to data over human input mean that one is already "data-obsessed"?