What is the best correlation coefficient R(X, Y) to measure non-linear dependencies between two variables X and Y? Let's say that you want to assess weather there is a linear or quadratic relationship between X and Y. One way to do it is to perform a polynomial regression such as Y = a + bX + cX^2, and then measure the standard coefficient of correlation between the predicted and observed values. How good is this approach? 

Note that the proposed correlation coefficient R(X, Y) is not symmetric. One way to get a symmetric version, is to use the maximum between | R(X, Y) | and | R(Y, X) |. It will be equal to 1 if and only if there is an exact polynomial or inverse polynomial relationship between X and Y. 

Note: If one checks the model Y = a + bX + cX^2, the "inverse polynomial" model would be X = a' + b'Y + c'Y^2. So, R(X, Y) is computed on the first regression, while R(Y, X) is computed on the second (reversed, also called dual) regression. 

Discussion

An issue with my approach is the risk of over-fitting. If you have n observations and n coefficients in the regression, my correlation will always be 1.

There are various ways to avoid this problem, for instance:

• Use a polynomial of degree 2 maximum, regardless of the number of observations.
• Use much smoother functions than polnomials, for instance functions that have one extremum (maximum or minimum) at most, and growing not faster than a linear function. Even in that case, use a small number of coefficients in the regression, maybe log(log(n))) where n is the number of observations.

The correlation coefficient in question can also be used for model selection: The best model would provide the correlation closest to 1.