COMPARATIVE PERFORMANCE OF CLASSICAL AND REGULARIZED REGRESSION MODELS FOR SMALL-SAMPLE CONTINUOUS DATA
DOI:
https://doi.org/10.69980/8mghew57Keywords:
Small-sample regression, Regularization, Ridge regression, Lasso regression, Elastic NetAbstract
This study compared the predictive performance and estimation stability of ordinary least squares (OLS), Ridge, Lasso, and Elastic Net regression under systematically varied small-sample conditions. An empirical dataset containing 21 numerical predictors was evaluated using repeated random samples of 30, 50, 75, 100, 150, and 200 observations, with an independent 300-observation test set retained for evaluation. Model performance was assessed using root mean squared error (RMSE), mean absolute error, coefficient of determination, coefficient variability, and variable-selection stability. Strong multicollinearity was identified among several predictors, with variance inflation factors approaching 20. At n = 30, OLS produced an RMSE of 71.02, compared with 31.81 for Ridge, 27.31 for Lasso, and 26.14 for Elastic Net. Prediction error and coefficient variability generally decreased as sample size increased, while performance differences between classical and regularized estimation progressively narrowed. Friedman and Holm-adjusted Wilcoxon tests confirmed significantly lower RMSE for the regularized estimators relative to OLS across all sample-size conditions. Lasso produced the sparsest solutions, whereas Elastic Net retained broader groups of correlated predictors. Overall, regularization substantially improved predictive and coefficient stability when observations were limited and predictors were strongly correlated, with its relative advantage diminishing as sample availability increased.
References
1. Archer, L., Snell, K. I., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021). Minimum sample size for external validation of a clinical prediction model with a continuous outcome. Statistics in Medicine, 40(1), 133-146.
2. Chan, J. Y. L., Leow, S. M. H., Bea, K. T., Cheng, W. K., Phoong, S. W., Hong, Z. W., & Chen, Y. L. (2022). Mitigating the multicollinearity problem and its machine learning approach: a review. Mathematics, 10(8), 1283.
3. Dar, I. S., Chand, S., Shabbir, M., & Kibria, B. G. (2023). Condition-index based new ridge regression estimator for linear regression model with multicollinearity. Kuwait Journal of Science, 50(2), 91-96.
4. Dustin, D., Clarke, B., & Clarke, J. (2024). Predictive stability criteria for penalty selection in linear models: D. Dustin et al. Computational Statistics, 39(3), 1241-1280.
5. Fernández-Delgado, M., Sirsat, M. S., Cernadas, E., Alawadi, S., Barro, S., & Febrero-Bande, M. (2019). An extensive experimental survey of regression methods. Neural Networks, 111, 11-34.
6. Gana, R. (2022). Ridge regression and the elastic net: How do they do as finders of true regressors and their coefficients?. Mathematics, 10(17), 3057.
7. Genç, M. (2022). Bir Simülasyon Çalışması ile Cezalı Regresyon Yöntemlerinin Karşılaştırılması. Bilecik Şeyh Edebali Üniversitesi Fen Bilimleri Dergisi, 9(1), 80-91.
8. Greenwood, C. J., Youssef, G. J., Letcher, P., Macdonald, J. A., Hagg, L. J., Sanson, A., ... & Olsson, C. A. (2020). A comparison of penalised regression methods for informing the selection of predictive markers. PloS one, 15(11), e0242730.
9. Heinze, G., Wallisch, C., & Dunkler, D. (2018). Variable selection–a review and recommendations for the practicing statistician. Biometrical journal, 60(3), 431-449.
10. Herawati, N., Sutrisno, A., Nusyirwan, N., & Misgiyati, M. (2024). The performance of ridge regression, LASSO, and elastic-net in controlling multicollinearity: A simulation and application. Journal of Modern Applied Statistical Methods,, 23(2), 1-13.
11. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning: with applications in R (Vol. 2). New York: springer.
12. Kipruto, E., & Sauerbrei, W. (2024). Post‐Estimation Shrinkage in Full and Selected Linear Regression Models in Low‐Dimensional Data Revisited. Biometrical Journal, 66(7), e202300368.
13. Nayem, H. M., Aziz, S., & Kibria, B. G. (2024). Comparison among ordinary least squares, ridge, lasso, and elastic net estimators in the presence of outliers: Simulation and application. International Journal of Statistical Sciences, 25-48.
14. PMLB regression datasets. (2018). Regression datasets. https://doi.org/10.5281/zenodo.1475816
15. Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019). Minimum sample size for developing a multivariable prediction model: Part I–Continuous outcomes. Statistics in medicine, 38(7), 1262-1275.
16. Riley, R. D., Snell, K. I., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021). Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. Journal of Clinical Epidemiology, 132, 88-96.
17. Schreiber-Gregory, D. N. (2018). Ridge Regression and multicollinearity: An in-depth review. Model Assisted Statistics and Applications, 13(4), 359-365.
18. Sztepanacz, J. L., & Houle, D. (2024). Regularized regression can improve estimates of multivariate selection in the face of multicollinearity and limited data. Evolution Letters, 8(3), 361-373.
19. Tian, Y., & Zhang, Y. (2022). A comprehensive survey on regularization strategies in machine learning. Information Fusion, 80, 146-166.
20. Van Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020). Regression shrinkage methods for clinical prediction models do not guarantee improved performance: simulation study. Statistical methods in medical research, 29(11), 3166-3178.
21. Wang, F., Mukherjee, S., Richardson, S., & Hill, S. M. (2020). High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking: Wang et al. Statistics and computing, 30(3), 697-719.
