A Data Science Central Community
There have been 5 summers since former Wired Editor-in-Chief Chris Anderson published an article about Big Data that concluded with the following statement: “Correlation supersedes causation, and science can advance even without coherent models, unified theories, or really any mechanistic explanation at all.” Big Data is not only changing science–it’s changing the way businesses interact with and gain insight from their data.
Business intelligence and statistical analysis tools are used to derive insight from data collected about transactions, the populations they’re serving, and to figure out the next product or service that clients demand and deserve. These tools aren’t new–SAS, for instance, has been around since the 1970s. What is new, though, is the amount of data that companies have access to.
Traditional BI tools have some fantastic uses, but they were all built around classical statistical methods of querying, sampling, and modeling data. The classical models were fantastic when data wasn’t so unruly. Business data now comes in many shapes, sizes, and speeds, none of which kindly behave like the standard numeric data of yore. Chris Anderson’s 2008 statement about correlation and causation underlines the need for BI tools to be augmented by some other tool, particularly one that automatically detects patterns in data.
Standard BI and statistical analysis tools require all the trappings of classical statistical methods to find needed information in a database: query testing, sampling, and more. Moreover, classical methods require data to be in table form, and require those tables to be populated by numeric data. The fact about data today is that much of it is not numeric, nor is it structured in a neat database. These data types are best served in the analysis process by pattern detection.
When a pattern detection tool is paired with traditional BI and statistical analysis tools, many of the pains listed above are automatically eliminated. Automated pattern analysis is...