I am trying to solve the regression Y=AX
is the response, X
the input, and A
the regression coefficients. I came up with the following iterative algorithm:
Ak+1 = cYU + Ak (I-cXU),
- c is an arbitrary constant
- U is an arbitrary matrix such that YU has same dimension as A. For instance U = transposed(X) works.
- A0 is the initial estimate for A. For instance A0 is the correlation vector between the independent variables and the response.
- What are the conditions for convergence? Do I have convergence if and only if the largest eigenvalue (in absolute value) of the matrix I-cXU is strictly less than 1?
- In case of convergence, will it converge to the solution of the regression problem? For instance, if c=0, the algorithm converges, but not to the solution. In that case, it converges to A0.
- n: number of independent variables
- m: number of observations
Why using an iterative algorithm instead of the traditional solution?
- A: (1,n) (one row, n columns)
- I: (n,n)
- X: (n,m)
- U: (m,n)
- Y: (1,m)
- We are dealing with an ill-conditioned problem; most independent variables are highly correlated.
- Many solutions (as long as the regression coefficients are positive) provide a very good fit, and the global optimum is not that much better than a solution where all regression coefficients are equal to 1.
- The plan is to use an iterative algorithm to start at iteration #1 with an approximate solution that has interesting properties, then move to iteration #2 to improve a bit, then stop.
Note: this question is not related to the ridge regression algorithm described here
- From Ray Koopman
No need to apologize for not using "proper" weights. See
Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571-582.