AnalyticBridge

A Data Science Central Community

Regression Trees - What is the best reference?

I'm doing some work with classification and regression trees, and I was wondering who the thought leaders are on this topic, and where I can find the most current research.

I have found some sources
The R documentation mentions Classification and Regression Trees by Breiman, Friedman, Olshen, and Stone. However the publication date is 1984, and I'm wondering if that's the best reference.

The examples in "Modern Applied Statistics with S" by Venables and Ripley has been useful, but not detailed enough.

I'm wondering who the real experts are in this area.

Perhaps this research has morphed into Random Forests?  From the little I know about that theory, it seems like an extension of the recursive partitioning / tree idea.

Thanks in advance for any insight

Tags: R, forests, partitioning, random, trees

Views: 4261

Replies to This Discussion

http://bit.ly/bOzPuq -> will have to hunt for this for free. random forests of multinomial logits. 2008 good for multiclass classification and regression and overcoming curse of dim.

http://bit.ly/asrmLW -> BART. rigor of bayes, coolness of bagging. 2008 you can fire this up in R as well.

now I'm curious what the state of the art is in regression trees. Throw back some feedback if you find something else out there.
I am aware that the original documentation written by Brieman et al can be a little hard to get hold of (I mean, it is more than 20 years old), but there are a few papers and books out there that may be of interest to you:

1) Lee, M., Chrysostomou, K.A., Chen, S.Y., and Liu, X., (2008). Applications of Decision Trees for Data Modelling. Encyclopedia of Artificial Intelligence, 1, 437-442 – Gives an overview of the different types of decision trees including CART, and also the popular applications of such decision trees. [Only Abstract at http://www.igi-global.com/bookstore/Chapter.aspx?TitleId=10284. If you’re interested in the full article then please let me know and I might be able to send you the full version]
2) Kohavi, R. and Quinlan, R., (1999). Decision Tree Discovery. – Detailed and profound comparison of C4.5 and CART decision tree classifiers (I would highly recommend this as base reading for CART) [see the following link for full article: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.4.5353]
3) You may also want to read the book: Hand, D.J., Mannila, H., and Smyth, P., (2001). Principles of data mining, MIT Press, which gives a detailed description of all types of decision trees (including CART)

In your quest to learn about decision trees, in particular the CART classifier, please remember that all types of decision tree classifiers that you read about will more or less follow the same process: (1) splitting data using a so-called splitting criterion (2) forming the final decision tree, and (3) pruning the final tree to reduce its size and increase its classification abilities. So, if you understand how one decision tree classifier works, you will probably understand how all of them work.

However, the decision tree classifiers differ mainly in how they do Step 1 and Step 2. In terms of Step 1, decision tree classifiers may use different splitting criterion, for example the CART classifier uses a gini index to make the splits in the data (which only results in binary splits) as opposed to the information gain measure (which can result in two or more splits) like other tree classifiers use. Another major difference between decision tree classifiers is the type of data they can handle/process: CART can process both categorical and numerical data, while others can only handle categorical data.

Hope this helps you in your quest to find some useful and detailed information on CART.

Dr Kyriacos Chrysostomou
Dear Dr Kyriacos Chrysostomou,

I am working on decsion trees produced using CHAId. Can you please send the first article to me also, my email id is [email protected]

Regards
Amit

There is a great book called "Elements of Statistical Learning" by Hastie, Tibshirani, and Friedman which describes in detail classification and regression trees and many other data mining methods.  You can buy the book or download it in pdf. The link is:

Lester Wollman

﻿

That book is amazing.  I was looking for something more specialized, but I'm not working on that problem anymore anyway.  Thanks for the input.

Isn't it funny that CART is one of the best algorithms 27 years after it was introduced? :-)

Logistic regression has similar story - very old and very useful for today's problems.

If you want to improve accuracy learn more about ensembles - boosting, bagging, adaboost, random forests. But don't expect miracles, they all improve CART's performance by relatively small margin. And all got some pros&cons.

http://www.salford-systems.com/doc/newhybridmethods.pdf - here's nice presentation about CART, only 13 years old, but really worth reading.