COMPARATIVE STATISTICAL EVALUATION OF REGRESSION MODELS FOR IMBALANCED CONTINUOUS DATA USING ROBUST PERFORMANCE MEASURES
DOI:
https://doi.org/10.69980/edvdh578Keywords:
Imbalanced regression, Continuous-target imbalance, Robust performance measures, Support Vector Regression, Machine learning, Rare-value predictionAbstract
A great difficulty with regression modelling of imbalanced continuous outcomes is that traditional assessment methods may overestimate the accuracy of prediction in well-explored response zones and underestimate the error in poorly-explored rare extremes. The present study compared the performance of seven statistical and machine-learning regression models using powerful and specific performance measures. The models experimented with were Multiple Linear Regression, Ridge Regression, Lasso Regression, Support Vector Regression, Random Forest Regression, Gradient Boosting Regression and XGBoost Regression. 10-fold cross-validation was used, and errors were measured in terms of mean absolute error, root mean square error, median absolute error, R2, and errors specific to the normal and rare parts of the response space. The distribution of responses was negatively skewed and heavy-tailed, which is indicative of meaningful continuous-target imbalance. Support Vector Regression performed best in general with the minimum MAE, RMSE, median absolute error and maximum R² values and in addition had the smallest rare-region error values. Linear models had significantly larger errors especially extreme observations compared to the performance of Random Forest and XGBoost. Friedman tests showed that there were significant differences between models on all the evaluation measures. The results show the importance of using strong and region-specific assessment for practical applications to compare regression models when the target is continuous and the imbalance is high.
References
1. Arteaga, M., Gacto, M. J., Galende, M., Alcalá-Fdez, J., & Alcalá, R. (2023). Enhancing soft computing techniques to actively address imbalanced regression problems. Expert Systems with Applications, 234, 121011.
2. Avelino, J. G., Cavalcanti, G. D., & Cruz, R. M. (2024). Resampling strategies for imbalanced regression: a survey and empirical analysis: JG Avelino et al. Artificial Intelligence Review, 57(4), 82.
3. Belhaouari, S. B., Islam, A., Kassoul, K., Al-Fuqaha, A., & Bouzerdoum, A. (2024). Oversampling techniques for imbalanced data in regression. Expert systems with applications, 252, 124118.
4. Björklund, A., Henelius, A., Oikarinen, E., Kallonen, K., & Puolamäki, K. (2022). Robust regression via error tolerance. Data Mining and Knowledge Discovery, 36(2), 781-810.
5. Bottmer, L., Croux, C., & Wilms, I. (2022). Sparse regression for large data sets with outliers. European Journal of Operational Research, 297(2), 782-794.
6. Camacho, L., & Bacao, F. (2024). WSMOTER: a novel approach for imbalanced regression: L. Camacho and F. Bacao. Applied intelligence, 54(19), 8789-8799.
7. Dolar, T., Chen, J., & Chen, W. (2025). Uncertainty quantification driven machine learning for improving model accuracy in imbalanced regression tasks. Expert Systems with Applications, 261, 125526.
8. Fawagreh, K., & Gaber, M. M. (2020). Resource-efficient fast prediction in healthcare data analytics: A pruned Random Forest regression approach: K. Fawagreh, MM Gaber. Computing, 102(5), 1187-1198.
9. Guan, Y., & Fu, G. H. (2024). Sparse feature selection and rare value prediction in imbalanced regression. Information Sciences, 680, 121145.
10. Hicks, S. A., Strümke, I., Thambawita, V., Hammou, M., Riegler, M. A., Halvorsen, P., & Parasa, S. (2022). On evaluation metrics for medical applications of artificial intelligence. Scientific reports, 12(1), 5979.
11. Li, Z. Z., Huang, N., Yi, L. Z., & Fu, G. H. (2024). Affine combination‐based over‐sampling for imbalanced regression. Journal of Chemometrics, 38(3), e3537.
12. Miller, C., Portlock, T., Nyaga, D. M., & O’Sullivan, J. M. (2024). A review of model evaluation metrics for machine learning in genetics and genomics. Frontiers in Bioinformatics, 4, 1457619.
13. Mishra, A., & Müller, C. L. (2022). Robust regression with compositional covariates. Computational Statistics & Data Analysis, 165, 107315.
14. Moguerza, J. M., Muñoz, A., & Venturini, G. M. (2020). Support Vector Regression.
15. Orhobor, O. I., Grinberg, N. F., Soldatova, L. N., & King, R. D. (2023). Imbalanced regression using regressor-classifier ensembles. Machine Learning, 112(4), 1365-1387.
16. Rainio, O., Teuho, J., & Klén, R. (2024). Evaluation metrics and statistical tests for machine learning. Scientific reports, 14(1), 6086.
17. Ren, J., Zhang, M., Yu, C., & Liu, Z. (2022, June). Balanced mse for imbalanced visual regression. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 7916-7925). IEEE.
18. Ribeiro, R. P., & Moniz, N. (2020). Imbalanced regression and extreme value prediction. Machine Learning, 109(9), 1803-1835.
19. Steininger, M., Kobs, K., Davidson, P., Krause, A., & Hotho, A. (2021). Density-based weighting for imbalanced regression. Machine Learning, 110(8), 2187-2211.
20. Tabacu, S. (2025). ReEntrant_ML_DataSets_ (Version 2) [Data set]. Mendeley Data. https://doi.org/10.17632/z4m93zsb2s.2
21. Tian, H., Tian, C., Li, K., & Jia, W. (2023). Unbalanced regression sample generation algorithm based on confrontation. Information Sciences, 642, 119157.
22. Wilson, A., Kasy, M., & Mackey, L. (2020, June). Approximate cross-validation: Guarantees for model assessment and selection. In International conference on artificial intelligence and statistics (pp. 4530-4540). PMLR.
23. Xiong, H., & Yao, A. (2024, June). Deep imbalanced regression via hierarchical classification adjustment. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 23721-23730). IEEE.
24. Yang, Y., Zha, K., Chen, Y., Wang, H., & Katabi, D. (2021, July). Delving into deep imbalanced regression. In International conference on machine learning (pp. 11842-11851). PMLR.
25. Zhang, W., Shi, P., Jia, P., & Zhou, X. (2024). A novel gradient boosting approach for imbalanced regression. Neurocomputing, 601, 128091.
26. Zhang, X., Yan, C., Gao, C., Malin, B. A., & Chen, Y. (2020). Predicting missing values in medical data via XGBoost regression. Journal of healthcare informatics research, 4(4), 383-394.
27. Zhu, W., Wu, O., & Yang, N. (2024). IRDA: Implicit data augmentation for deep imbalanced regression. Information Sciences, 677, 120873.
