A Data Science Central Community
The Big Data Analytics revolution is underway. This revolution is an historic and game-changing expansion of the role that information plays in business, government and consumer realms. To harness the power of this data revolution, a paradigm shift is required. Organizations must be able to do more than query their Big Data stores; search is no longer enough. We announced the release of our new Big Data white paper this week and wanted to share some of the highlights with you.
Nugget #1 – Know What to Ignore
Up until now in the history of data analysis, the objective of queries was to find the signal in the noise. And it worked because we had clear-cut business questions and the size of the data was smaller, the data set was more complete, and we usually knew what we were looking for. We were playing in the realm of known knowns and known unknowns. In the new world of Big Data, it is now more important to know what to ignore. Unless you know what to ignore, you’ll never get a chance to pay attention to what’s really important. Using algorithms to first ignore the noise and then find the insights is the way of the new world.
Nugget #2 – Change Your Relationship to Data
Humankind has always possessed a love for data. In the past, analysts could enjoy a very personal relationship with data. It was just the right size that we could touch it, feel it, mold it, and often make it say what we wanted it to say! But all that has changed, as data has evolved into Big Data – a big monster, no more cuddly and cute.
Nugget #3 – Any Method That Puts the Burden on the User is a Game-stopper
Now the size of Big Data demands a change in our relationship with the data. We need automation that can manage the data and let it speak to us. This will require smart machines. The overwhelming shortcoming of existing and emerging methods for obtaining insight is that they are query-based and labor-intensive. Humans need to do what humans do best and machines need to do what they do best. The machines, i.e., smart software, need to do the intense analysis while the humans need to evaluate the results. Big Data offers an infinite number of queries, causing all these methods to rely on analysts to produce questions. But any method that puts the burden on the user is a game-stopper.
Nugget #4 – Algorithms Are The Way of the Future
Analysis needs to be automated. This automation happens with algorithms, which are designed for calculation, data processing, and automated reasoning. Algorithms are designed for tasks that are beyond human comprehension and require the speed of machines. This is the realm of Big Data.