Replies to This Discussion

Permalink Reply by Philip (flip) Kromer on February 1, 2009 at 2:43pm

Have you looked well at Hadoop? With Hadoop streaming, you can use anything to analyze your data, as long as you can recenter the problem in a map-reduce paradigm. I write all my code in Ruby, and there are some good Python frameworks (eg Dumbo), but you could absolutely use R where it's a good match to the problem. Hadoop lets you scale out rather than up, and scale arbitrarily. (The parallelization of map-reduce is astonishingly near-linear).

I was able to turn the idle time of a computer lab at my university into a 70-machine cluster with ease, and amazon EC2 lets you pull down as many computers as you care to request for very little. (CPU time for a 20-machine cluster over a 2000-hour working year costs $4,000). Using one-off ruby scripts and Hadoop Pig I regularly query and model datasets with 80M row-cardinality in several dimensions. Up in this several-hundred-GB range it's clearly not interactive, but you can extract an appropriately reduced dataset to qualify your analysis and then run it at scale on the full mama pajama.

• ▶ Reply

Permalink Reply by Scott Terry on February 2, 2009 at 6:17am

R and open source software are great tools. But they're limited with respect to volume.

I've found that SPSS gives you the best cost/benefit. Since V14, it's truly become industrial strength. And its limits are more of a function of the hardware than that of SPSS's limits.

For example, I've SPSS to process Experian's compiled file ... 120 million rows of data, with each record being fairly wide. No crashing at all.

Coding-wise, SPSS (and, of course, SAS) has a syntax language that's pretty powerful. You can also get the functions to write code for you that you can either use as is or modify.

While SPSS interacts with other languages, like Python, it'll do just about anything I need. As another example, over time, I've also written a lot of data quality, xform, and matching programs in SPSS that I can incorporate as well.

• ▶ Reply

Permalink Reply by John Bauer on February 9, 2009 at 11:22am

It is possible and sometimes very helpful to sample differentially from unbalanced data. At the extreme, take all of the least frequently occurring and a fraction of the others. Taking an equal number of each is the idea used in case-control studies, and is also the core of the idea which makes boosting work, so it may be worth experimenting with.

• ▶ Reply

Permalink Reply by Ralf Klinkenberg on February 11, 2009 at 4:47pm

The open source data mining software RapidMiner can handle very large data sets and lets you freely choose between fast in-memory data mining and extremely scalable on-database data mining.

Among the users of RapidMiner in more than 40 countries are some of the world's largest companies, e.g.
* Lufthansa, the leading European airline,
* mobilkom austria, leading Austrian mobile phone service provider,
* Bank of America, leading US bank,
* BNP Paribas, leading European bank,
* Sanofi-Aventis, leading European pharma company,
* HP, Nokia, Philips, Miele, and many more.

Their data sets often include millions of transactions or records or text documents.

In addition to the RapidMiner Community Edition, which can be downloaded free of charge, there also is the RapidMiner Enterprise Edition with 64bit and multi-core-processor parallelization support as well as professional technical support with guaranteed response times.

For more information please visit: www.rapid-i.com

• ▶ Reply

Permalink Reply by Bill Cassill on April 26, 2009 at 1:29pm

Vincent,

Save yourself the headache and get a desktop copy of SAS. I've comfortably processed 200+GB files on my desktop for clients without issues, or the need for extra workstations, or advanced hardware, etc. And the good thing is that SAS will have a very small memory footprint on your machine, something on the order of 200MB or so. So even if you have a couple of GB of RAM, you should be just fine.

Bill

• ▶ Reply

Permalink Reply by Jens Godsk Katz-Kolberg on April 26, 2009 at 2:26pm

Still, SAS (SAS/Base and SAS/Stat is minimum) is much more expensive than a high end PC, with lots of RAM. I would definitely go with a great PC with lots of RAM, since that is nice in many other situations, and then use one of the open source alternatives.

• ▶ Reply

Permalink Reply by Larry on April 29, 2010 at 6:33am

Agreed. With the cost of a SAS license you could setup a nice PC cluster farm with lots of RAM in each node.

• ▶ Reply

Permalink Reply by Jaap Vink on May 5, 2009 at 12:38am

PASW Modeler (formerly known as SPSS Clementine) does not store data in RAM (except for the data needed at any step in an algorithm). I'm not sure about the other products. PASW Modeler uses SQL optimization to have maximum benefits from the database containing the data (Oracle, SQL Server, DB2, Netezza, Terradata,....). It also has a transparent integration with in-database mining algorithms (SQL Server, Oracle, DB2) and scoring engines. Some of the appliactions of PASW modeler involve many millions of records (10m+).

• ▶ Reply

Permalink Reply by Ashok B on January 4, 2011 at 4:18am

Hi jaap Vink,

            Thanks for your good information. Iam making use of this PASW Modeler for predicting fraudulent customers. I have generated a model using around 3000 records of customers along with their fraud history and used this obtained model in a new stream for new single customer data. It is going to give predicted result(i..e T/F) and probability value. Am i doing in correct way in predicting new customer based on previous customers? if not please suggest  me a solution. Any suggestion may help us.

 

Thanks in Advance,

Ashok B.

• ▶ Reply

Permalink Reply by Karl Rexer on January 4, 2011 at 7:26am

Ashok --

     It sounds like what you're doing is correct.  Fraud can sometimes be tricky due to the low frequency of fraud (hopefully your business has only a handful of fraudulent customers).  

     Am I correct in assuming that for the 3000 customer records you're using to build this model, you have BOTH some customers who you know committed fraud and some customers who did not?  If this is the case, then you would have a binary (Yes/No) variable that indicates whether the customers committed fraud -- this is your "target" variable.  How many fraud and non-fraud customers are in your 3000 customer sample?  What types of variables are you using in the predictive model (e.g., is it transaction history, demographics, etc.)?  How you code your predictor variables can sometimes be important for both model building and model scoring.

     If you don't have a binary fraud target variable, there are still some good things you can do.  But I would need to know more about your data and business situation to help you.  You can contact me directly at [email protected] if you want to discuss this further.  We're a small analytic consulting firm, and we've conducted analytics to identify fraud for clients in several industries.

     Good luck!  --  Karl Rexer, PhD

• ▶ Reply

Permalink Reply by Ashok B on January 6, 2011 at 4:52am

Hi Karl,

          Thanks for your quick reply. Iam going to contact you through my personal-id, please respond for that one.

 

Thanks in Advance,

Ashok B.

• ▶ Reply

Permalink Reply by Jim Falgout on May 20, 2009 at 4:08pm

Take a look at DataRush, a new product from Pervasive Software (http://www.pervasivedatarush.com). As a disclosure, I work for Pervasive on the DataRush product. It is a platform for building scalable applications that we are using to build out data mining operators/applications. It is built on dataflow concepts and so allows the pipelining of data through the system. As such, it can work on large amounts (millions, billions of records) without having to have the whole dataset in memory at one time.

We'll be at KDD in Paris later next month presenting a paper on our experience using DataRush to process the Netflix data.

• ▶ Reply

• ‹ Previous
• 1
• 2
• 3
• Next ›
• Page