STATISTICAL MODELLING OF HIGH-DIMENSIONAL DATA UNDER SEVERE MULTICOLLINEARITY USING REGRESSION AND PRINCIPAL COMPONENT METHODS
DOI:
https://doi.org/10.69980/3aj86879Keywords:
High-dimensional data, Multicollinearity, Principal component analysis, Principal component regression, Ordinary least squaresAbstract
Many high-dimensional data sets have highly correlated and redundant predictors that may affect the stability and interpretability of traditional regression models. In this study, a statistical model of high-dimensional data with severe multicollinearity was evaluated using ordinary least squares (OLS) regression, principal component analysis (PCA) and principal component regression (PCR) was evaluated. The number of observations was 500, the number of continuous predictors was 150, and the number of continuous target variables was 1. The underlying structure of the data was explored through descriptive statistics, Pearson correlations, tolerance, variance inflation factors and condition diagnostics. The original predictor space was highly multicollinear, and they could not estimate the total set of predictors using OLS. PCA then projected the 150 predictors onto two orthogonal components, which explained nearly all the variance in the predictors. These components were used in the development of a PCR, which was statistically significant and accounted for 4.2% of the target variance. The first principal component was significant, but the second was not. However, PCR did not compensate for the lack of goodness of fit over the estimable baseline OLS model but was able to completely remove the multicollinearity and offer a substantially more parsimonious and numerically stable representation of the predictor space. The results show that dimensionality reduction is a useful statistical tool to overcome high-dimensional, highly collinear predictor redundancy without loss of predictive value and validate PCR as a valuable tool in such cases.
References
[1]. Auddy, A., Xia, D., & Yuan, M. (2025). Tensors in high-dimensional data analysis: Methodological opportunities and theoretical challenges. Annual Review of Statistics and Its Application, 12, 527–551.
[2]. Bühlmann, P., Kalisch, M., & Meier, L. (2014). High-dimensional statistics with a view toward applications in biology. Annual Review of Statistics and Its Application, 1, 255–278.
[3]. Cai, T. T., Guo, Z., & Xia, Y. (2023). Statistical inference and large-scale multiple testing for high-dimensional regression models. TEST, 32, 1135–1171.
[4]. Cook, R. D. (2018). Principal components, sufficient dimension reduction, and envelopes. Annual Review of Statistics and Its Application, 5, 533–559.
[5]. Davino, C., Romano, R., & Vistocco, D. (2022). Handling multicollinearity in quantile regression through the use of principal component regression. Metron, 80, 153–174.
[6]. Dormann, C. F., Elith, J., Bacher, S., Buchmann, C., Carl, G., Carré, G., García Márquez, J. R., Gruber, B., Lafourcade, B., Leitão, P. J., Münkemüller, T., McClean, C., Osborne, P. E., Reineking, B., Schröder, B., Skidmore, A. K., Zurell, D., & Lautenbach, S. (2013). Collinearity: A review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36(1), 27–46.
[7]. Fan, J., & Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(5), 849–911.
[8]. Filzmoser, P., & Nordhausen, K. (2021). Robust linear regression for high-dimensional data: An overview. WIREs Computational Statistics, 13(4), e1524.
[9]. Fleming, S. W., & Garen, D. C. (2022). Simplified cross-validation in principal component regression (PCR) and PCR-like machine learning for water supply forecasting. Journal of the American Water Resources Association, 58(4), 517–524.
[10]. Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22.
[11]. Greenacre, M., Groenen, P. J. F., Hastie, T., Iodice D’Enza, A., Markos, A., & Tuzhilina, E. (2022). Principal component analysis. Nature Reviews Methods Primers, 2, Article 100.
[12]. Gu, F., & Cheung, M. W.-L. (2023). A model-based approach to multivariate principal component regression: Selecting principal components and estimating standard errors for unstandardized regression coefficients. British Journal of Mathematical and Statistical Psychology, 76(3), 605–622.
[13]. Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.
[14]. Jolliffe, I. T., & Cadima, J. (2016). Principal component analysis: A review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065), 20150202.
[15]. Lee, E. R., Cho, J., & Yu, K. (2019). A systematic review on model selection in high-dimensional regression. Journal of the Korean Statistical Society, 48(1), 1–12.
[16]. Lee, S., Seo, M. H., & Shin, Y. (2016). The Lasso for high dimensional regression with a possible change point. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(1), 193–210.
[17]. Liu, R. X., Kuang, J., Gong, Q., & Hou, X. L. (2003). Principal component regression analysis with SPSS. Computer Methods and Programs in Biomedicine, 71(2), 141–147.
[18]. Meinshausen, N., & Bühlmann, P. (2010). Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(4), 417–473.
[19]. O’Brien, R. M. (2007). A caution regarding rules of thumb for variance inflation factors. Quality & Quantity, 41(5), 673–690.
[20]. Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288.
[21]. Wu, Y., & Wang, L. (2020). A survey of tuning parameter selection for high-dimensional regression. Annual Review of Statistics and Its Application, 7, 209–226.
[22]. Zou, H. (2006). The adaptive Lasso and its oracle properties. Journal of the American Statistical Association, 101(476), 1418–1429.
[23]. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301–320.
