It is common to use R language to group and summarize data of files. Sometimes we may find ourselves processing comparatively big files which have smaller computed result and bigger source data. We cannot load them wholly to the memory when we need to compute them. The only solutions could be batch importing and computing as well as result merging. We’ll use an example in the following to illustrate the way of R language to group and summarize data from big text files.

Here is a file,…

A Tool That Grants Independence to Business Users

We all know the constant struggle between IT and business users when it comes to BI software: Business users want to access data in order to make fast decisions independently, without having to use IT as a middle man every time a new requirement arises, query is run, or data is added. Yet, IT is overwhelmed by constantly changing requests and requirements, and struggle to deliver data in an actionable time-frame. The…

Global Energy Forecasting Competition 2014 by IEEE - Power Energy Society launches on CrowdANALYTIX platform. Check out the 4 tracks of your choice: Electric Load, Electric Price, Wind Power and Solar Power.

IEEE, the world’s largest professional organization dedicated to advancing technology for humanity, today announced the start of the Global Energy Forecasting Competition 2014 (GEFCom2014)—sponsored by the IEEE Power & Energy Society (PES) and organized by the IEEE Working…

Original blog post on sctr7.com, by Scott Mongeau.

Executive summary

Network analysis offers a new set of techniques to tackle the persistent and growing problem of complex fraud. Network…

Original blog posted on sctr7.com.

The adage ‘garbage-in-garbage-out’ is an analytics mantra so ingrained it has its own shorthand: GIGO. Yet, in the mad, blind rush toward all things ‘big data’, there is the danger of sidelining the crucial-but-dreary topic of data quality, to which GIGO refers.

While data quality is not as ‘sexy’ as big data, anyone who wants…

Originally posted on sctr7.com. 

Network analysis is a rapidly growing analytics domain propelled by the explosion of interest in social networking. The methods rest upon much older foundations in the realms of statistics and social science. Euler’s graph theory was proposed in the early 18^th century and Moreno established the foundations for…

In this post, I've tried to capture some of the common aspects of working in the analytics industry. While we occasionally hear about India growing fast into this space, there are a lot of things happening in India that might transform this field further. While some of these aspects are specific to what I've observed in India, a lot of them are generic.

As in previous posts, I try to classify these aspects under different heads:

1. The "cost" reduction trap…