Validation of Cluster - AnalyticBridge2020-09-28T03:34:51Zhttps://www.analyticbridge.datasciencecentral.com/forum/topics/validation-of-cluster?commentId=2004291%3AComment%3A114377&feed=yes&xn_auth=noThank you for the information…tag:www.analyticbridge.datasciencecentral.com,2011-06-30:2004291:Comment:1201792011-06-30T15:57:32.357ZPanchanana M.R.L.N.https://www.analyticbridge.datasciencecentral.com/profile/PanchananaMRLN
Thank you for the information. I will get back if I have any queries.
Thank you for the information. I will get back if I have any queries. Hi,
1) there are several node…tag:www.analyticbridge.datasciencecentral.com,2011-06-14:2004291:Comment:1152412011-06-14T14:27:34.849ZFranz Hofnerhttps://www.analyticbridge.datasciencecentral.com/profile/FranzHofner
<p>Hi,</p>
<p>1) there are several nodes in EM available: variable selection tests for collinearity, variable clustering is an alternative; you might as well start with a factor analysis using sas stat and cluster for the factors as variables</p>
<p>2) EM offers in the cluster node several methods to automatically bin class-variables into numeric variables (problematic as you loose control on the variable weight). In stat-environment you might need to do it manually.</p>
<p>3) First step is to…</p>
<p>Hi,</p>
<p>1) there are several nodes in EM available: variable selection tests for collinearity, variable clustering is an alternative; you might as well start with a factor analysis using sas stat and cluster for the factors as variables</p>
<p>2) EM offers in the cluster node several methods to automatically bin class-variables into numeric variables (problematic as you loose control on the variable weight). In stat-environment you might need to do it manually.</p>
<p>3) First step is to find business sense in the clusters by generating profiles or descriptive trees for each cluster. For the technical validation: the EM cluster node generates score code. So you can split your data in two parts, cluster both parts separatly and apply the score code from the model set to the validation set. The grouping should overlap for a large portion of the data (the numbering of the clusters will vary, though, so you need to find the permutation which maximizes the overlap - that's called Munkres Algorithm if I recall it right and can be achieved via SAS OR or any other linear optimizer)</p>
<p>Regards</p>
<p>Franz</p> Hi Panchanana,
For your first…tag:www.analyticbridge.datasciencecentral.com,2011-06-11:2004291:Comment:1143772011-06-11T09:42:06.981ZSandeep Sunkarahttps://www.analyticbridge.datasciencecentral.com/profile/SandeepSunkara
<p>Hi Panchanana,</p>
<p><br></br>For your first question,</p>
<p><strong>Variable Reduction for Numerical Variables can be done in two ways:</strong></p>
<p>1. Before the Cluster Analysis(which again can be done in two ways):</p>
<p> (a) Use top N(choose as per the variation explained by the components up to your mark) principal components.</p>
<p> (b) Run a variable cluster analysis and select one or two variables from each cluster (You may go for PCA on these selected variables and choose…</p>
<p>Hi Panchanana,</p>
<p><br/>For your first question,</p>
<p><strong>Variable Reduction for Numerical Variables can be done in two ways:</strong></p>
<p>1. Before the Cluster Analysis(which again can be done in two ways):</p>
<p> (a) Use top N(choose as per the variation explained by the components up to your mark) principal components.</p>
<p> (b) Run a variable cluster analysis and select one or two variables from each cluster (You may go for PCA on these selected variables and choose top components from here).</p>
<p>2. After the Cluster Analysis: Run Cluster Analysis on all the variables. For every variable check if the average(Mean or Median) value is significantly different across the clusters(You may go for F-Test). If it is not significantly differing you can remove those variables and re-run the cluster procedure.</p>
<p> </p>
<p><strong>Variable Reduction for Categorical Variables:</strong> You may get measure of association between all pairs of categorical variables and choose one or two variables from each set of highly associated variables.</p>
<p><br/>For your Second Question,</p>
<p>You can do cluster analysis with categorical variables also, but choose appropriate distance measure while using PROC DISTANCE (Obviously Euclidean distance does not work, you may choose DGower's dissimilarity). After you select set of numerical and categorical variables to use for cluster analysis, numerical variables need to be standardized before calculating distance matrix(there is option in PROC DISTANCE), otherwise variables with higher values will dominate the measure of distance.</p>
<p><br/>Once you create the distance matrix dataset, you can use it as input to cluster procedure to run cluster analysis. As and when you decide to remove one variable, distance has to be recalculated removing that variable.</p>
<p> </p>
<p>I am still checking out for 'Validation for Cluster Analysis'.</p>
<p> </p>
<p>This is just my view, appreciate more inputs.</p>
<p> </p>
<p>Thanks,</p>
<p>Sandeep</p>