Subscribe to DSC Newsletter

Iterative Algorithm for Linear Regression

I am trying to solve the regression Y=AX where Y is the response, X the input, and A the regression coefficients. I came up with the following iterative algorithm:

Ak+1 = cYU + Ak (I-cXU),

where:

  • c is an arbitrary constant
  • U is an arbitrary matrix such that YU has same dimension as A. For instance U = transposed(X) works.
  • A0 is the initial estimate for A. For instance A0 is the correlation vector between the independent variables and the response.

Questions:

  • What are the conditions for convergence? Do I have convergence if and only if the largest eigenvalue (in absolute value) of the matrix I-cXU is strictly less than 1?
  • In case of convergence, will it converge to the solution of the regression problem? For instance, if c=0, the algorithm converges, but not to the solution. In that case, it converges to A0.

Parameters:

  • n: number of independent variables
  • m: number of observations

Matrix dimensions:

  • A: (1,n) (one row, n columns)
  • I: (n,n)
  • X: (n,m)
  • U: (m,n)
  • Y: (1,m)

Why using an iterative algorithm instead of the traditional solution?

  • We are dealing with an ill-conditioned problem; most independent variables are highly correlated.
  • Many solutions (as long as the regression coefficients are positive) provide a very good fit, and the global optimum is not that much better than a solution where all regression coefficients are equal to 1.
  • The plan is to use an iterative algorithm to start at iteration #1 with an approximate solution that has interesting properties, then move to iteration #2 to improve a bit, then stop.

Note: this question is not related to the ridge regression algorithm described here.

Contributions:


  • From Ray Koopman

    No need to apologize for not using "proper" weights. See

    Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571-582.

Views: 3952

Comment

You need to be a member of AnalyticBridge to add comments!

Join AnalyticBridge

Follow Us

On Data Science Central

On DataViz

On Hadoop

© 2017   AnalyticBridge.com is a subsidiary and dedicated channel of Data Science Central LLC   Powered by

Badges  |  Report an Issue  |  Terms of Service