A Data Science Central Community

I am trying to solve the regression *Y=AX* where *Y* is the response, *X* the input, and *A* the regression coefficients. I came up with the following iterative algorithm:

*A*_{k+1} = cYU + A_{k} (I-cXU),

**where:**

**Questions:**

**Parameters:**

**Matrix dimensions:**

**Why using an iterative algorithm instead of the traditional solution?**

Note: this question is not related to the ridge regression algorithm described here.

*c*is an arbitrary constant*U*is an arbitrary matrix such that*YU*has same dimension as*A*. For instance*U*= transposed(*X*) works.*A*_{0}is the initial estimate for*A*. For instance*A*_{0}is the correlation vector between the independent variables and the response.

- What are the conditions for convergence? Do I have convergence if and only if the largest eigenvalue (in absolute value) of the matrix
*I-cXU*is strictly less than 1? - In case of convergence, will it converge to the solution of the regression problem? For instance, if
*c=0*, the algorithm converges, but not to the solution. In that case, it converges to*A*_{0}.

*n*: number of independent variables*m*: number of observations

*A*:*(1,n)*(one row,*n*columns)*I*:*(n,n)**X*:*(n,m)**U*:*(m,n)**Y*:*(1,m)*

- We are dealing with an ill-conditioned problem; most independent variables are highly correlated.
- Many solutions (as long as the regression coefficients are positive) provide a very good fit, and the global optimum is not that much better than a solution where all regression coefficients are equal to 1.
- The plan is to use an iterative algorithm to start at iteration #1 with an approximate solution that has interesting properties, then move to iteration #2 to improve a bit, then stop.

Note: this question is not related to the ridge regression algorithm described here.

**Contributions:**

- From Ray Koopman
No need to apologize for not using "proper" weights. See

Dawes, Robyn M. (1979).

*The robust beauty of improper linear models in decision making*. American Psychologist, 34, 571-582.

© 2021 TechTarget, Inc. Powered by

Badges | Report an Issue | Privacy Policy | Terms of Service

**Most Popular Content on DSC**

To not miss this type of content in the future, subscribe to our newsletter.

- Book: Applied Stochastic Processes
- Long-range Correlations in Time Series: Modeling, Testing, Case Study
- How to Automatically Determine the Number of Clusters in your Data
- New Machine Learning Cheat Sheet | Old one
- Confidence Intervals Without Pain - With Resampling
- Advanced Machine Learning with Basic Excel
- New Perspectives on Statistical Distributions and Deep Learning
- Fascinating New Results in the Theory of Randomness
- Fast Combinatorial Feature Selection

**Other popular resources**

- Comprehensive Repository of Data Science and ML Resources
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- 100 Data Science Interview Questions and Answers
- Cheat Sheets | Curated Articles | Search | Jobs | Courses
- Post a Blog | Forum Questions | Books | Salaries | News

**Archives:** 2008-2014 |
2015-2016 |
2017-2019 |
Book 1 |
Book 2 |
More

**Most popular articles**

- Free Book and Resources for DSC Members
- New Perspectives on Statistical Distributions and Deep Learning
- Time series, Growth Modeling and Data Science Wizardy
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- Comprehensive Repository of Data Science and ML Resources
- Advanced Machine Learning with Basic Excel
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- Selected Business Analytics, Data Science and ML articles
- How to Automatically Determine the Number of Clusters in your Data
- Fascinating New Results in the Theory of Randomness
- Hire a Data Scientist | Search DSC | Find a Job
- Post a Blog | Forum Questions

## You need to be a member of AnalyticBridge to add comments!

Join AnalyticBridge