ROBUST STATISTICAL ESTIMATION FOR MULTICOLLINEAR DATA UNDER RESPONSE CONTAMINATION: A COMPARATIVE ANALYSIS
DOI:
https://doi.org/10.69980/evcf4246Keywords:
Multicollinearity, Robust regression, Ridge regression, Elastic Net, Huber-loss SGD regressionAbstract
The reliability of the traditional estimators may be diminished due to regression data with highly correlated predictors and influential observations. This study employed and benchmarked ordinary least squares, Ridge, LASSO, Elastic Net, and Huber-loss SGD regression with a community-level dataset with high predictor dependence. The observations and predictors after pre-processing were 1,994 and 40, respectively. To check the multicollinearity, the pairwise correlations and the condition number were calculated. To evaluate the model performance, they used the 70:30 training-test split and the RMSE, MAE, R² metrics. Robustness was subsequently assessed with simulated responses contaminated at 0%, 5%, 10% and 20%. The predictor matrix was highly collinear (maximum absolute correlation 0.986, condition number 232.95). In the un-contaminated cases Huber-loss SGD regression and Ridge had the lowest RMSE values. However, as the contamination increased, the Huber loss SGD regression performed relatively well, while OLS, LASSO, Ridge and Elastic Net experienced significant loss in performance. The Huber-loss SGD regression was able to retain the same RMSE (366.50) and R² (0.6042) as the same contamination level as the OLS regression (479.08, 0.3232). With the estimators evaluated, Huber-loss SGD regression exhibited the strongest stability with respect to the investigated cases of multicollinearity and response contamination.
References
1. Adomaityte, U., Defilippis, L., Loureiro, B., & Sicuro, G. (2024). High-dimensional robust regression under heavy-tailed data: Asymptotics and universality. Journal of Statistical Mechanics: Theory and Experiment, 2024(11), 114002.
2. Affindi, A. N., & Ahmad, S. (2022). Robust ridge regression approach for combined multicollinearity-outlier problem. Applied Mathematics and Computational Intelligence, 11(1), 123–132.
3. Altelbany, S. I. (2021). Evaluation of Ridge, Elastic Net and Lasso Regression Methods in Precedence of Multicollinearity Problem: A Simulation Study. Journal of Applied Economics & Business Studies (JAEBS), 5(1), 131.
4. Arum, K. C., & Ugwuowo, F. I. (2022). Combining principal component and robust ridge estimators in linear regression model with multicollinearity and outlier. Concurrency and Computation: Practice and Experience, 34(10), e6803.
5. Chamlal, H., Benzmane, A., & Ouaderhman, T. (2024). Elastic net-based high dimensional data selection for regression. Expert Systems with Applications, 244, 122958.
6. De Menezes, D. Q., Prata, D. M., Secchi, A. R., & Pinto, J. C. (2021). A review on robust M-estimators for regression analysis. Computers & Chemical Engineering, 147, 107254.
7. Filzmoser, P., & Nordhausen, K. (2021). Robust linear regression for high‐dimensional data: An overview. Wiley Interdisciplinary Reviews: Computational Statistics, 13(4), e1524.
8. Göktaş, A., Akkuş, Ö., & Kuvat, A. (2021). A new robust ridge parameter estimator based on search method for linear regression model. Journal of Applied Statistics, 48(13-15), 2457-2472.
9. Ghosh, A., & Thoresen, M. (2021). A robust variable screening procedure for ultra-high dimensional data. Statistical Methods in Medical Research, 30(8), 1816-1832.
10. Herawati, N., Wijayanti, A., Sutrisno, A., Nusyirwan, & Misgiyati. (2024). The performance of ridge regression, LASSO, and elastic-net in controlling multicollinearity: A simulation and application. Journal of Modern Applied Statistical Methods, 23(2), 1–13.
11. Khan, M. S., Ali, A., Suhail, M., Alotaibi, E. S., & Alsubaie, N. E. (2024). On the estimation of ridge penalty in linear regression: Simulation and application. Kuwait Journal of Science, 51(4), 100273.
12. Kim, H., & Lee, T. H. (2021). A robust elastic net via bootstrap method under sampling uncertainty for significance analysis of high-dimensional design problems. Knowledge-Based Systems, 225, 107117.
13. Liu, L., Shen, Y., Li, T., & Caramanis, C. (2020, June). High dimensional robust sparse regression. In International Conference on Artificial Intelligence and Statistics (pp. 411-421). PMLR.
14. Loh, P. L. (2025). A theoretical review of modern robust statistics. Annual Review of Statistics and Its Application, 12(1), 477-496.
15. Lukman, A. F., Farghali, R. A., Kibria, B. G., & Oluyemi, O. A. (2023). Robust-stein estimator for overcoming outliers and multicollinearity. Scientific Reports, 13(1), 9066.
16. Majid, A., Amin, M., Aslam, M., & Ahmad, S. (2023). New robust ridge estimators for the linear regression model with outliers. Communications in Statistics-Simulation and Computation, 52(10), 4717-4738.
17. Nayem, H. M., Aziz, S., & Kibria, B. G. (2024). Comparison among ordinary least squares, ridge, lasso, and elastic net estimators in the presence of outliers: Simulation and application. International Journal of Statistical Sciences, 24(2), 25–48.
18. Redmond, M. (2009). Communities and crime dataset (unnormalized data) [Data set]. Kaggle. https://www.kaggle.com/datasets/johnp47/communities-and-crime-dataset
19. Roozbeh, M., Maanavi, M., & Mohamed, N. A. (2024). A robust counterpart approach for the ridge estimator to tackle outlier effect in restricted multicollinear regression models. Journal of Statistical Computation and Simulation, 94(2), 279-296.
20. Su, M., & Wang, W. (2021). Elastic net penalized quantile regression model. Journal of Computational and Applied Mathematics, 392, 113462.
21. Suhail, M., Chand, S., & Kibria, B. G. (2021). Quantile-based robust ridge M-estimator for linear regression model in presence of multicollinearity and outliers. Communications in Statistics-Simulation and Computation, 50(11), 3194-3206.
22. Sun, Q., Zhou, W. X., & Fan, J. (2020). Adaptive huber regression. Journal of the American Statistical Association, 115(529), 254-265.
23. Sztepanacz, J. L., & Houle, D. (2024). Regularized regression can improve estimates of multivariate selection in the face of multicollinearity and limited data. Evolution Letters, 8(3), 361-373.
24. Wang, Y., & Karunamuni, R. J. (2022). High-dimensional robust regression with Lq-loss functions. Computational Statistics & Data Analysis, 176, 107567.
25. Wang, L., Peng, B., Bradic, J., Li, R., & Wu, Y. (2020). A tuning-free robust and efficient approach to high-dimensional regression. Journal of the American Statistical Association, 115(532), 1700-1714.
26. Wasim, D., Suhail, M., Albalawi, O., & Shabbir, M. (2024). Weighted penalized m-estimators in robust ridge regression: an application to gasoline consumption data. Journal of Statistical Computation and Simulation, 94(15), 3427-3456.
27. Wasim, D., Suhail, M., Khan, S. A., Shabbir, M., Awwad, F. A., Ismail, E. A. A., Ahmad, H., & Ali, A. (2025a). Quantile-based robust Kibria–Lukman estimator for linear regression model to combat multicollinearity and outliers: Real life applications using T20 cricket sports and anthropometric data. Kuwait Journal of Science, 52(1), 100336.
28. Wasim, D., Zaman, Q., Ahmad, M., & Kibria, B. G. (2025b). Mitigating multicollinearity and outliers in regression: comparison of some new and old robust ridge M-estimators. Journal of Statistical Computation and Simulation, 95(16), 3526-3547.
29. Yang, Z., Fu, L., Wang, Y. G., Dong, Z., & Jiang, Y. (2022). A robust and efficient variable selection method for linear regression. Journal of Applied Statistics, 49(14), 3677-3692.
30. Yasin, S., Salem, S., Ayed, H., Kamal, S., Suhail, M., & Khan, Y. A. (2021). Modified robust ridge M-estimators in two-parameter ridge regression model. Mathematical Problems in Engineering, 2021, 1845914.
